We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.IR

Change to browse by:

cs

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Information Retrieval

Title: On Ranking Consistency of Pre-ranking Stage

Authors: Siyu Gu
Abstract: Industrial ranking systems, such as advertising systems, rank items by aggregating multiple objectives into one final objective to satisfy user demand and commercial intent. Cascade architecture, composed of retrieval, pre-ranking, and ranking stages, is usually adopted to reduce the computational cost. Each stage may employ various models for different objectives and calculate the final objective by aggregating these models' outputs. The multi-stage ranking strategy causes a new problem - the ranked lists of the ranking stage and previous stages may be inconsistent. For example, items that should be ranked at the top of the ranking stage may be ranked at the bottom of previous stages. In this paper, we focus on the \textbf{ranking consistency} between the pre-ranking and ranking stages. Specifically, we formally define the problem of ranking consistency and propose the Ranking Consistency Score (RCS) metric for evaluation. We demonstrate that ranking consistency has a direct impact on online performance. Compared with the traditional evaluation manner that mainly focuses on the individual ranking quality of every objective, RCS considers the ranking consistency of the fused final objective, which is more proper for evaluation. Finally, to improve the ranking consistency, we propose several methods from the perspective of sample selection and learning algorithms. Experimental results on one of the biggest industrial E-commerce platforms in China validate the efficacy of the proposed metrics and methods.
Comments: 9 pagees, 5 figures
Subjects: Information Retrieval (cs.IR)
Cite as: arXiv:2205.01289 [cs.IR]
  (or arXiv:2205.01289v3 [cs.IR] for this version)

Submission history

From: Siyu Gu [view email]
[v1] Tue, 3 May 2022 03:47:42 GMT (2599kb,D)
[v2] Wed, 11 May 2022 08:54:27 GMT (2599kb,D)
[v3] Sat, 9 Jul 2022 16:01:23 GMT (2599kb,D)
[v4] Tue, 12 Jul 2022 02:26:56 GMT (2599kb,D)
[v5] Thu, 3 Nov 2022 09:05:58 GMT (2599kb,D)

Link back to: arXiv, form interface, contact.