Falk Scholer
Andrew Turpin
ABSTRACT
We introduce and explore the concept of an individual's relevance threshold as a way of reconciling differences in outcomes between batch and user experiments.
- A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between IR effectiveness measures and user satisfaction. SIGIR'07, p773--774. Google Scholar
Digital Library
- J. Allan, B. Carterette, and J. Lewis. When will information retrieval be "good enough"? SIGIR'05, p433--440. Google ScholarDigital Library
- C. Clarke, N. Craswell, and I. Soboroff. Overview of the TREC 2004 terabyte track. Gaithersburg, MD, 2005.Google Scholar
- G.A Gescheider. Psychophysics: method, theory and application. Lawrence Erlbaum Ass., New Jersey, 1985.Google Scholar
- W. Hersh, A. Turpin, S. Price, B. Chan, D. Kraemer, L. Sacherek, and D. Olson. Do batch and user evaluations give the same results? SIGIR'00, p17--24. Google ScholarDigital Library
- D. Kelly, X. Fu, and C. Shah. Effects of rank and precision of search results on users' evaluations of system performance. TR-2007-02, U. of North Carolina, 2007.Google Scholar
- A. Turpin and W. Hersh. Why batch and user evaluations do not give the same results. SIGIR'01, p225--231. Google ScholarDigital Library
- A. Turpin and F. Scholer. User performance versus precision measures.. SIGIR'06, p11--18. Google ScholarDigital Library
- E. M. Voorhees and D. K. Harman. TREC : experiment and evaluation in informationretrieval. MIT Press, 2005. Google ScholarDigital Library
Index Terms
- Relevance thresholds in system evaluations
Recommendations
User performance versus precision measures for simple search tasks
SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrievalSeveral recent studies have demonstrated that the type of improvements in information retrieval system effectiveness reported in forums such as SIGIR and TREC do not translate into a benefit for users. Two of the studies used an instance recall task, ...
TREC-Style evaluations
PROMISE'12: Proceedings of the 2012 international conference on Information Retrieval Meets Information VisualizationTREC-style evaluation is generally considered to be the use of test collections, an evaluation methodology referred to as the Cranfield paradigm. This paper starts with a short description of the original Cranfield experiment, with the emphasis on the ...
Understanding Relevance Judgments in Legal Case Retrieval
Legal case retrieval, which aims to retrieve relevant cases given a query case, has drawn increasing research attention in recent years. While much research has worked on developing automatic retrieval models, how to characterize relevance in this ...
Comments