References & Citations
Computer Science > Information Retrieval
Title: Climbing the WOL: Training for Cheaper Inference
(Submitted on 2 Jul 2020 (v1), last revised 3 Jul 2020 (this version, v2))
Abstract: Efficient inference for wide output layers (WOLs) is an essential yet challenging task in large scale machine learning. Most approaches reduce this problem to approximate maximum inner product search (MIPS), which relies heavily on the observation that for a given model, ground truth labels correspond to logits of highest value during full model inference. However, such an assumption is restrictive in practice. In this paper, we argue that approximate MIPS subroutines, despite having sub-linear computation time, are sub-optimal because they are tailored for retrieving large inner products with high recall instead of retrieving the correct labels. With WOL, the labels often have moderate inner products, which makes approximate MIPS more challenging. We propose an alternative problem formulation, called Label Superior Sampling (LSS), where the objective is to tailor the system to ensure retrieval of the correct label. Accordingly, we propose a novel learned hash approach, which is significantly more efficient and sufficient for high inference accuracy than MIPS baselines. Our extensive evaluation indicates that LSS can match or even outperform full inference accuracy with around 5x speed up and 87% energy reduction.
Submission history
From: Zichang Liu [view email][v1] Thu, 2 Jul 2020 16:26:26 GMT (605kb)
[v2] Fri, 3 Jul 2020 03:11:54 GMT (605kb)
Link back to: arXiv, form interface, contact.