Computer Science · Information Systems
R.: Minimal test collections for retrieval evaluation
Ben Carterette, James Allan, Ben Carterette, James Allan, Ramesh Sitaraman
See on the mapAbstract
Accurate estimation of information retrieval evaluation met-rics such as average precision require large sets of relevance judgments. Building sets large enough for evaluation of real-world implementations is at best inefficient, at worst infea-sible. In this work we link evaluation with test collection construction to gain an understanding of the minimal judg-ing effort that must be done to have high confidence in the outcome of an evaluation. A new way of looking at av-erage precision leads to a natural algorithm for selecting documents to judge and allows us to estimate the degree of confidence by defining a distribution over possible docu-ment judgments. A study with annotators shows that this method can be used by a small group of researchers to rank a set of systems in under three hours with 95 % confidence.