We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.IR

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Information Retrieval

Title: Relevance Judgment Convergence Degree -- A Measure of Inconsistency among Assessors for Information Retrieval

Abstract: Relevance judgment of human assessors is inherently subjective and dynamic when evaluation datasets are created for Information Retrieval (IR) systems. However, a small group of experts' relevance judgment results are usually taken as ground truth to "objectively" evaluate the performance of the IR systems. Recent trends intend to employ a group of judges, such as outsourcing, to alleviate the potentially biased judgment results stemmed from using only a single expert's judgment. Nevertheless, different judges may have different opinions and may not agree with each other, and the inconsistency in human relevance judgment may affect the IR system evaluation results. In this research, we introduce a Relevance Judgment Convergence Degree (RJCD) to measure the quality of queries in the evaluation datasets. Experimental results reveal a strong correlation coefficient between the proposed RJCD score and the performance differences between the two IR systems.
Comments: To appear on 30th International Conference on Information Systems Development (ISD2022)
Subjects: Information Retrieval (cs.IR)
Cite as: arXiv:2208.04057 [cs.IR]
  (or arXiv:2208.04057v1 [cs.IR] for this version)

Submission history

From: Dengya Zhu [view email]
[v1] Mon, 8 Aug 2022 11:09:26 GMT (513kb)

Link back to: arXiv, form interface, contact.