An infrastructure for the evaluation and comparison of information retrieval systems

ABSTRACT

Even though information retrieval systems have been successfully deployed for over 45 years, the field continues to evolve in performance, functionality, and accuracy. There are hundreds of different products available with different indexing and retrieval characteristics. How does one choose the appropriate system for a given application? The first step in that choice is the creation of a framework for comparison of IR products and an infrastructure that supports automated execution and analysis of testing results. The next step is providing an environment for subjective measurement using human evaluators. In this paper we briefly introduce the concepts used in IR system evaluation and report on our initial implementation of a framework for evaluating indexing performance. We also report a test case, which provides a comparative analysis of the indexing characteristics for three IR system implementations using a common collection of documents.

References

1. Baeza-Yates, R. and Ribeiro-Neto, B. Modern Information Retrieval. Addison-Wesley Longman, Reading, MA, 1999.
2. Brown, A. B. and Seltzer, M. I. Operating system benchmarking in the wake of lmbench: A case study of the performance of NetBSD on the Intel x86 architecture. In Proceedings of the 1997 ACM SIGMETRICS international conference on Measurement and modeling of computer systems (Seattle, Washington, United States, June 15-18, 1997) ACM Press, New York NY, 1997, 214--224.
3. Croft, W. B. and Turtle, H. R. Text Retrieval and Inference. In P. S. Jacobs (ed.) Text-Based Intelligent Systems: Current Research and Practice in Information Extraction and Retrieval. Lawrence Erlbaum Associates, Hillsdale, New Jersey, 1992, 127--155.
4. Fox, C. Lexical analysis and stoplists. In W. B. Frakes and R. Baeza-Yates (eds.) Information Retrieval: Data Structures and Algorithms. Prentice-Hall, Englewood Cliffs, NJ, 1992, 102--130.
5. Gao, X., Murugesan, S., and Lo, B. Multi-dimensional evaluation of information retrieval results. In Proceedings of the IEEE/WIC/ACM International Conference (WI '04) (September 20-24, 2004). IEEE Computer Society, Washington, DC, 2004, 192--198.
6. Hart, M. S. History and Philosophy of Project Gutenberg, 1992. Retrieved August 15, 2006, from http://www.gutenberg.org/about/history
7. Salton, G., Wong, A., and Yang, C. S. A vector space model for automatic indexing, Communications of the ACM, 18, 11 (Nov. 1975). ACM Press, New York, NY, 1975, 613--620.
8. Van Rijsbergen, C. J., Information Retrieval, 2nd Ed. Butterworth-Heinemann, Newton, MA, 1979.
9. The Gutenberg Project. Retrieved March 5, 2006, from http://www.gutenberg.org

Index Terms

• Information systems

  • Information retrieval

    • Retrieval models and ranking