arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Information Systems

Minimal test collections for retrieval evaluation

Ben Carterette, James Allan, Ramesh K. Sitaraman

· ScholarWorks@UMassAmherst (University of Massachusetts Amherst)

See on the map

Abstract

Accurate estimation of information retrieval evaluation metrics such as average precision require large sets of relevance judgments. Building sets large enough for evaluation of real-world implementations is at best inefficient, at worst infeasible. In this work we link evaluation with test collection construction to gain an understanding of the minimal judging effort that must be done to have high confidence in the outcome of an evaluation. A new way of looking at average precision leads to a natural algorithm for selecting documents to judge and allows us to estimate the degree of confidence by defining a distribution over possible document judgments. A study with annotators shows that this method can be used by a small group of researchers to rank a set of systems in under three hours with 95% confidence.

Cited by 3

3 papers in the index cite this.

The cited-by list is still filling in while reference lists are collected.

More like this