ABSTRACT
Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or human-built. Multilingual and multimedia information access systems, such as search engines, are increasingly complex: they need to satisfy diverse user needs and support challenging tasks. Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. In this context, failure analysis is crucial to understand the behaviour of complex systems. Unfortunately, this is an especially challenging activity, requiring vast amounts of human effort to inspect query-by-query the output of a system in order to understand what went well or bad. It is therefore fundamental to provide automated tools to examine system behaviour, both visually and analytically. Moreover, once you understand the reason behind a failure, you still need to conduct a "what-if" analysis to understand what among the different possible solutions is most promising and effective before actually starting to modify your system. This paper provides an analytical model for examining performances of IR systems, based on the discounted cumulative gain family of metrics, and visualization for interacting and exploring the performances of the system under examination. Moreover, we propose machine learning approach to learn the ranking model of the examined system in order to be able to conduct a "what-if" analysis and visually explore what can happen if you adopt a given solution before having to actually implement it.
- {Banks et al., 1999} Banks, D., Over, P., and Zhang, N.-F. (1999). Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval, 1: 7--34. Google ScholarDigital Library
- {Berkhin, 2006} Berkhin, P. (2006). A Survey of Clustering Data Mining Techniques. In Kogan, J., Nicholas, C., and Teboulle, M., editors, Grouping Multidimensional Data, pages 25--71. Springer-Verlag, Heidelberg, Germany.Google ScholarCross Ref
- {Derthick et al., 2003a} Derthick, M., Christel, M. G., Hauptmann, A. G., and Wactlar, H. D. (2003a). Constant density displays using diversity sampling. In Proceedings of InfoVis'03, pages 137--144, Washington, DC, USA. IEEE Computer Society. Google ScholarDigital Library
- {Derthick et al., 2003b} Derthick, M., Christel, M. G., Hauptmann, A. G., and Wactlar, H. D. (2003b). Constant density displays using diversity sampling. In Proceedings of the IEEE Information Visualization, pages 137--144. Google ScholarDigital Library
- {Freund et al., 2003} Freund, Y., Iyer, R., Schapire, R. E., and Singer, Y. (2003). An Efficient Boosting Algorithm for Combining Preferences. Journal of Machine Learning Research, 4(Nov): 933--969. Google ScholarDigital Library
- {Geng et al., 2007} Geng, X., Liu, T.-Y., Qin, T., and Li, H. (2007). Feature Selection for Ranking. In Kraaij, W., de Vries, A. P., Clarke, C. L. A., Fuhr, N., and Kando, N., editors, Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), pages 407--414. ACM Press, New York, USA. Google ScholarDigital Library
- {Harman and Buckley, 2009} Harman, D. and Buckley, C. (2009). Overview of the reliable information access workshop. Information Retrieval, 12(6): 615--641. Google ScholarDigital Library
- {Järvelin and Kekäläinen, 2002} Järvelin, K. and Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information System (TOIS), 20(4): 422--446. Google ScholarDigital Library
- {Keskustalo et al., 2008} Keskustalo, H., Järvelin, K., Pirkola, A., and Kekäläinen, J. (2008). Intuition-Supporting Visualization of User's Performance Based on Explicit Negative Higher-Order Relevance. In Chua, T.-S., Leong, M.-K., Oard, D. W., and Sebastiani, F., editors, Proc. 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008), pages 675--682. ACM Press, New York, USA. Google ScholarDigital Library
- {Korfhage, 1997} Korfhage, R. R. (1997). Information Storage and Retrieval. Wiley Computer Publishing, John Wiley & Sons, Inc., USA. Google ScholarDigital Library
- {Liu, 2009} Liu, T.-Y. (2009). Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval, 3(3): 225--331. Google ScholarDigital Library
- {Liu et al., 2007} Liu, T.-Y. Y., Xu, J., Qin, T., Xiong, W., and Li, H. (2007). LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval. In Joachims, T., Li, H., Liu, T.-Y., and Zhai, C., editors, SIGIR 2007 Workshop on Learning to Rank for Information Retrieval.Google ScholarCross Ref
- {Seo and Shneiderman, 2004} Seo, J. and Shneiderman, B. (2004). A rank-by-feature framework for interactive exploration of multidimensional data. In Proceedings of the IEEE Information Visualization, pages 65--72. Google ScholarDigital Library
- {Seo and Shneiderman, 2005} Seo, J. and Shneiderman, B. (2005). A rank-by-feature framework for interactive exploration of multidimensional data. Information Visualization, 4: 96--113. Google ScholarDigital Library
- {Sormunen et al., 2002} Sormunen, E., Hokkanen, S., Kangaslampi, P., Pyy, P., and Sepponen, B. (2002). Query Performance Analyser -- a Web-based tool for IR research and instruction. In Järvelin, K., Beaulieu, M., Baeza-Yates, R., and Hyon Myaeng, S., editors, Proc. 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), page 450. ACM Press, New York, USA. Google ScholarDigital Library
- {Teevan et al., 2010} Teevan, J., Dumais, S. T., and Horvitz, E. (2010). Potential for personalization. ACM Transactions on Computer-Human Interaction (TOCHI), 17(1): 1--31. Google ScholarDigital Library
- {van Rijsbergen, 1979} van Rijsbergen, C. J. (1979). Information Retrieval. Butterworths, London, England, 2nd edition. Google ScholarDigital Library
- {Voorhees and Harman, 1999} Voorhees, E. and Harman, D. (1999). Overview of the Seventh Text REtrieval Conference (TREC-7). In NIST Special Publication 500--242: The Seventh Text REtrieval Conference (TREC 7). Springer-Verlag, Heidelberg, Germany.Google ScholarCross Ref
Index Terms
- Visual interactive failure analysis: supporting users in information retrieval evaluation
Recommendations
DESIRE 2011: first international workshop on data infrastructures for supporting information retrieval evaluation
CIKM '11: Proceedings of the 20th ACM international conference on Information and knowledge managementThe workshop focuses on the three areas of interest to CIKM to discuss how to envisage and design evaluation infrastructures able to store, manage, and make accessible the scientific data and knowledge of interest for advancing the evaluation of ...
VIRTUE: A visual tool for information retrieval performance evaluation and failure analysis
Objective: Information Retrieval (IR) is strongly rooted in experimentation where new and better ways to measure and interpret the behavior of a system are key to scientific advancement. This paper presents an innovative visualization environment: ...
VizDeck: a card game metaphor for fast visual data exploration
CHI EA '12: CHI '12 Extended Abstracts on Human Factors in Computing SystemsScientists in all fields are acquiring data at a rate that is challenging the limits of human cognitive capacity. At the same time, researchers' attention is increasingly claimed by ever more diverse demands on their time. Visual perception is the ...
Comments