arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Information Systems

R.: Minimal test collections for retrieval evaluation

Ben Carterette, James Allan, Ben Carterette, James Allan, Ramesh Sitaraman

See on the map

Abstract

Accurate estimation of information retrieval evaluation met-rics such as average precision require large sets of relevance judgments. Building sets large enough for evaluation of real-world implementations is at best inefficient, at worst infea-sible. In this work we link evaluation with test collection construction to gain an understanding of the minimal judg-ing effort that must be done to have high confidence in the outcome of an evaluation. A new way of looking at av-erage precision leads to a natural algorithm for selecting documents to judge and allows us to estimate the degree of confidence by defining a distribution over possible docu-ment judgments. A study with annotators shows that this method can be used by a small group of researchers to rank a set of systems in under three hours with 95 % confidence.

More like this