arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · Information Systems

chrisb @ sabir.com

Chris Buckley, Ellen M. Voorhees

See on the map

Abstract

This paper presents a novel way of examining the accuracy of the evaluation measures commonly used in information re-trieval experiments. It validates everal of the rules-of-thumb experimenters u e, such as the number of queries needed for a good experiment is at least 25 and 50 is better, while chal-lenging other beliefs, such as the common evaluation measures are equally reliable. As an example, we show that Precision at 30 documents has about twice the average rror rate as Aver-age Precision has. These results can help information retrieval researchers design experiments that provide a desired level of confidence in their results. In particular, we suggest researchers using Web measures such as Precision at 10 documents will need to use many more than 50 queries or will have to require two methods to have a very large difference in evaluation scores be-fore concluding that the two methods are actually different. 1

More like this