Current browse context:
cs.CL
Change to browse by:
References & Citations
Computer Science > Computation and Language
Title: The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus
(Submitted on 18 Dec 2021 (v1), last revised 24 May 2022 (this version, v2))
Abstract: In order to address increasing demands of real-world applications, the research for knowledge-intensive NLP (KI-NLP) should advance by capturing the challenges of a truly open-domain environment: web-scale knowledge, lack of structure, inconsistent quality and noise. To this end, we propose a new setup for evaluating existing knowledge intensive tasks in which we generalize the background corpus to a universal web snapshot. We investigate a slate of NLP tasks which rely on knowledge - either factual or common sense, and ask systems to use a subset of CCNet - the Sphere corpus - as a knowledge source. In contrast to Wikipedia, otherwise a common background corpus in KI-NLP, Sphere is orders of magnitude larger and better reflects the full diversity of knowledge on the web. Despite potential gaps in coverage, challenges of scale, lack of structure and lower quality, we find that retrieval from Sphere enables a state of the art system to match and even outperform Wikipedia-based models on several tasks. We also observe that while a dense index can outperform a sparse BM25 baseline on Wikipedia, on Sphere this is not yet possible. To facilitate further research and minimise the community's reliance on proprietary, black-box search engines, we share our indices, evaluation metrics and infrastructure.
Submission history
From: Aleksandra Piktus [view email][v1] Sat, 18 Dec 2021 13:15:34 GMT (5426kb,D)
[v2] Tue, 24 May 2022 18:16:24 GMT (5680kb,D)
Link back to: arXiv, form interface, contact.