Scott B. Huffman
ABSTRACT
Per-query relevance measures provide standardized, repeatable measurements of search result quality, but they ignore much of what users actually experience in a full search session. This paper examines how well we can approximate a user's ultimate session-level satisfaction using a simple relevance metric. We find that thisrelationship is surprisingly strong. By incorporating additional properties of the query itself, we construct a model which predicts user satisfaction even more accurately than relevance alone.
- J. Allan, B. Carterette, and J. Lewis. When will information retrieval be \good enough"? In SIGIR'05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 433--440, New York, NY, USA, 2005. ACM Press. Google Scholar
Digital Library
- D. Bilal. Children's use of the Yahooligans! Web search engine: Cognitive, physical, and affective behaviors on fact-based search tasks. J. Am. Soc. Inf. Sci., 51(7):646--665, 2000. Google ScholarDigital Library
- P. Borlund. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research, 8(3), April 2003.Google Scholar
- A. Broder. A taxonomy of web search. SIGIR Forum, 36(2):3--10, 2002. Google ScholarDigital Library
- C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 33--40, New York, NY, USA, 2000. ACM Press. Google ScholarDigital Library
- C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 25--32, New York, NY, USA, 2004. ACM Press. Google ScholarDigital Library
- C. W. Cleverdon. The signifcance of the Cranfeld tests on index languages. In SIGIR '91: Proceedings of the 14th annual international ACM SIGIR conference on Research and development in information retrieval, pages 3--12, New York, NY, USA, 1991. ACM Press. Google ScholarDigital Library
- B. J. Jansen and U. Pooch. A review of web searching studies and a framework for future research. J. Am. Soc. Inf. Sci. Technol., 52(3):235--246, 2001. Google ScholarDigital Library
- K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst., 20(4):422--446, 2002. Google ScholarDigital Library
- D. Kahneman, P. P. Wakker, and R. Sarin. Back to Bentham? Explorations of experienced utility. The Quarterly Journal of Economics, 112(2):375--405, May 1997.Google ScholarCross Ref
- J. Reid. A task-oriented non-interactive evaluation methodology for information retrieval systems. Information Retrieval, 2(1):115--129, 2000. Google ScholarDigital Library
- D. E. Rose and D. Levinson. Understanding user goals in web search. In WWW '04: Proceedings of the 13th international conference on World Wide Web, pages 13--19, New York, NY, USA, 2004. ACM Press. Google ScholarDigital Library
- D. M. Russell and C. Grimes. Assigned and self-chosen tasks are not the same in web search. In HICSS '07: Proceedings of the 40th Annual International Conference on Systems and Software, 2007.Google ScholarDigital Library
- M. Sanderson and J. Zobel. Information retrieval system evaluation: effort, sensitivity, and reliability. In SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 162--169, New York, NY, USA, 2005. ACM Press. Google ScholarDigital Library
- A. Spink. A user-centered approach to evaluating human interaction with web search engines: an exploratory study. Inf. Process. Manage.,38(3):401--426, 2002. Google ScholarDigital Library
- J. Teevan, C. Alvarado, M. S. Ackerman, and D. R. Karger. The perfect search engine is not enough: A study of orienteering behavior in directed search. In CHI '04: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 415--422, New York, NY, USA, 2004. ACM Press. Google ScholarDigital Library
- A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 11--18, New York, NY, USA, 2006. ACM Press. Google ScholarDigital Library
- A. H. Turpin and W. Hersh. Why batch and user evaluations do not give the same results. In SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 225--231, New York, NY, USA, 2001. ACM Press. Google ScholarDigital Library
- E. M. Voorhees. Evaluation by highly relevant documents. In SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74--82, New York, NY, USA, 2001. ACM Press. Google ScholarDigital Library
Index Terms
- How well does result relevance predict session satisfaction?
Recommendations
When does Relevance Mean Usefulness and User Satisfaction in Web Search?
SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information RetrievalRelevance is a fundamental concept in information retrieval (IR) studies. It is however often observed that relevance as annotated by secondary assessors may not necessarily mean usefulness and satisfaction perceived by users. In this study, we confirm ...
Investigating Cognitive Effects in Session-level Search User Satisfaction
KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data MiningUser satisfaction is an important variable in Web search evaluation studies and has received more and more attention in recent years. Many studies regard user satisfaction as the ground truth for designing better evaluation metrics. However, most of the ...
Modeling user variance in time-biased gain
HCIR '12: Proceedings of the Symposium on Human-Computer Interaction and Information RetrievalCranfield-style information retrieval evaluation considers variance in user information needs by evaluating retrieval systems over a set of search topics. For each search topic, traditional metrics model all users searching ranked lists in exactly the ...
Comments