References & Citations
Computer Science > Computation and Language
Title: Distilling Knowledge from Reader to Retriever for Question Answering
(Submitted on 8 Dec 2020 (v1), last revised 4 Aug 2022 (this version, v2))
Abstract: The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used to solve the task based on retrieved documents, to obtain synthetic labels for the retriever. We evaluate our method on question answering, obtaining state-of-the-art results.
Submission history
From: Gautier Izacard [view email][v1] Tue, 8 Dec 2020 17:36:34 GMT (76kb)
[v2] Thu, 4 Aug 2022 17:36:08 GMT (55kb)
Link back to: arXiv, form interface, contact.