Learning user interaction models for predicting web search result preferences

ABSTRACT

Evaluating user preferences of web search results is crucial for search engine development, deployment, and maintenance. We present a real-world study of modeling the behavior of web search users to predict web search result preferences. Accurate modeling and interpretation of user behavior has important applications to ranking, click spam detection, web search personalization, and other tasks. Our key insight to improving robustness of interpreting implicit feedback is to model query-dependent deviations from the expected "noisy" user behavior. We show that our model of clickthrough interpretation improves prediction accuracy over state-of-the-art clickthrough methods. We generalize our approach to model user behavior beyond clickthrough, which results in higher preference prediction accuracy than models based on clickthrough information alone. We report results of a large-scale experimental evaluation that show substantial improvements over published implicit feedback interpretation methods.

References

1. E. Agichtein, E. Brill, and S. Dumais, Improving Web Search Ranking by Incorporating User Behavior, in Proceedings of the ACM Conference on Research and Development on Information Retrieval (SIGIR), 2006
2. J. Allan. HARD Track Overview in TREC 2003: High Accuracy Retrieval from Documents. In Proceedings of TREC 2003, 24--37, 2004.
3. S. Brin and L. Page, The Anatomy of a Large-scale Hypertextual Web Search Engine,. In Proceedings of WWW7, 107--117, 1998.
4. C.J.C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, Learning to Rank using Gradient Descent, in Proceedings of the International Conference on Machine Learning (ICML), 2005
5. D.M. Chickering, The WinMine Toolkit, Microsoft Technical Report MSR-TR-2002-103, 2002
6. M. Claypool, D. Brown, P. Lee and M. Waseda. Inferring user interest, in IEEE Internet Computing. 2001
7. S. Fox, K. Karnawat, M. Mydland, S. T. Dumais and T. White. Evaluating implicit measures to improve the search experience. In ACM Transactions on Information Systems, 2005
8. J. Goecks and J. Shavlick. Learning users' interests by unobtrusively observing their normal behavior. In Proceedings of the IJCAI Workshop on Machine Learning for Information Filtering. 1999.
9. T. Joachims, Optimizing Search Engines Using Clickthrough Data, in Proceedings of the ACM Conference on Knowledge Discovery and Datamining (SIGKDD), 2002
10. T. Joachims, L. Granka, B. Pang, H. Hembrooke and G. Gay, Accurately Interpreting Clickthrough Data as Implicit Feedback, in Proceedings of the ACM Conference on Research and Development on Information Retrieval (SIGIR), 2005
11. T. Joachims, Making Large-Scale SVM Learning Practical. Advances in Kernel Methods, in Support Vector Learning, MIT Press, 1999
12. D. Kelly and J. Teevan, Implicit feedback for inferring user preference: A bibliography. In SIGIR Forum, 2003
13. J. Konstan, B. Miller, D. Maltz, J. Herlocker, L. Gordon and J. Riedl. GroupLens: Applying collaborative filtering to usenet news. In Communications of ACM, 1997.
14. M. Morita, and Y. Shinoda, Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the ACM Conference on Research and Development on Information Retrieval (SIGIR), 1994
15. D. Oard and J. Kim. Implicit feedback for recommender systems. in Proceedings of AAAI Workshop on Recommender Systems. 1998
16. D. Oard and J. Kim. Modeling information content using observable behavior. In Proceedings of the 64th Annual Meeting of the American Society for Information Science and Technology. 2001
17. P. Pirolli, The Use of Proximal Information Scent to Forage for Distal Content on the World Wide Web. In Working with Technology in Mind: Brunswikian. Resources for Cognitive Science and Engineering, Oxford University Press, 2004
18. F. Radlinski and T. Joachims, Query Chains: Learning to Rank from Implicit Feedback, in Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD), ACM, 2005
19. F. Radlinski and T. Joachims, Evaluating the Robustness of Learning from Implicit Feedback, in the ICML Workshop on Learning in Web Search, 2005
20. G. Salton and M. McGill. Introduction to modern information retrieval. McGraw-Hill, 1983
21. E.M. Voorhees, D. Harman, Overview of TREC, 2001

Index Terms

• Information systems

  • Information retrieval

    • Evaluation of retrieval results

      • Relevance assessment

    • Information retrieval query processing