Visual interactive failure analysis: supporting users in information retrieval evaluation

ABSTRACT

Measuring is a key to scientific progress. This is particularly true for research concerning complex systems, whether natural or human-built. Multilingual and multimedia information access systems, such as search engines, are increasingly complex: they need to satisfy diverse user needs and support challenging tasks. Their development calls for proper evaluation methodologies to ensure that they meet the expected user requirements and provide the desired effectiveness. In this context, failure analysis is crucial to understand the behaviour of complex systems. Unfortunately, this is an especially challenging activity, requiring vast amounts of human effort to inspect query-by-query the output of a system in order to understand what went well or bad. It is therefore fundamental to provide automated tools to examine system behaviour, both visually and analytically. Moreover, once you understand the reason behind a failure, you still need to conduct a "what-if" analysis to understand what among the different possible solutions is most promising and effective before actually starting to modify your system. This paper provides an analytical model for examining performances of IR systems, based on the discounted cumulative gain family of metrics, and visualization for interacting and exploring the performances of the system under examination. Moreover, we propose machine learning approach to learn the ranking model of the examined system in order to be able to conduct a "what-if" analysis and visually explore what can happen if you adopt a given solution before having to actually implement it.

References

1. {Banks et al., 1999} Banks, D., Over, P., and Zhang, N.-F. (1999). Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval, 1: 7--34.
2. {Berkhin, 2006} Berkhin, P. (2006). A Survey of Clustering Data Mining Techniques. In Kogan, J., Nicholas, C., and Teboulle, M., editors, Grouping Multidimensional Data, pages 25--71. Springer-Verlag, Heidelberg, Germany.
3. {Derthick et al., 2003a} Derthick, M., Christel, M. G., Hauptmann, A. G., and Wactlar, H. D. (2003a). Constant density displays using diversity sampling. In Proceedings of InfoVis'03, pages 137--144, Washington, DC, USA. IEEE Computer Society.
4. {Derthick et al., 2003b} Derthick, M., Christel, M. G., Hauptmann, A. G., and Wactlar, H. D. (2003b). Constant density displays using diversity sampling. In Proceedings of the IEEE Information Visualization, pages 137--144.
5. {Freund et al., 2003} Freund, Y., Iyer, R., Schapire, R. E., and Singer, Y. (2003). An Efficient Boosting Algorithm for Combining Preferences. Journal of Machine Learning Research, 4(Nov): 933--969.
6. {Geng et al., 2007} Geng, X., Liu, T.-Y., Qin, T., and Li, H. (2007). Feature Selection for Ranking. In Kraaij, W., de Vries, A. P., Clarke, C. L. A., Fuhr, N., and Kando, N., editors, Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007), pages 407--414. ACM Press, New York, USA.
7. {Harman and Buckley, 2009} Harman, D. and Buckley, C. (2009). Overview of the reliable information access workshop. Information Retrieval, 12(6): 615--641.
8. {Järvelin and Kekäläinen, 2002} Järvelin, K. and Kekäläinen, J. (2002). Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information System (TOIS), 20(4): 422--446.
9. {Keskustalo et al., 2008} Keskustalo, H., Järvelin, K., Pirkola, A., and Kekäläinen, J. (2008). Intuition-Supporting Visualization of User's Performance Based on Explicit Negative Higher-Order Relevance. In Chua, T.-S., Leong, M.-K., Oard, D. W., and Sebastiani, F., editors, Proc. 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2008), pages 675--682. ACM Press, New York, USA.
10. {Korfhage, 1997} Korfhage, R. R. (1997). Information Storage and Retrieval. Wiley Computer Publishing, John Wiley & Sons, Inc., USA.
11. {Liu, 2009} Liu, T.-Y. (2009). Learning to Rank for Information Retrieval. Foundations and Trends in Information Retrieval, 3(3): 225--331.
12. {Liu et al., 2007} Liu, T.-Y. Y., Xu, J., Qin, T., Xiong, W., and Li, H. (2007). LETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval. In Joachims, T., Li, H., Liu, T.-Y., and Zhai, C., editors, SIGIR 2007 Workshop on Learning to Rank for Information Retrieval.
13. {Seo and Shneiderman, 2004} Seo, J. and Shneiderman, B. (2004). A rank-by-feature framework for interactive exploration of multidimensional data. In Proceedings of the IEEE Information Visualization, pages 65--72.
14. {Seo and Shneiderman, 2005} Seo, J. and Shneiderman, B. (2005). A rank-by-feature framework for interactive exploration of multidimensional data. Information Visualization, 4: 96--113.
15. {Sormunen et al., 2002} Sormunen, E., Hokkanen, S., Kangaslampi, P., Pyy, P., and Sepponen, B. (2002). Query Performance Analyser -- a Web-based tool for IR research and instruction. In Järvelin, K., Beaulieu, M., Baeza-Yates, R., and Hyon Myaeng, S., editors, Proc. 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), page 450. ACM Press, New York, USA.
16. {Teevan et al., 2010} Teevan, J., Dumais, S. T., and Horvitz, E. (2010). Potential for personalization. ACM Transactions on Computer-Human Interaction (TOCHI), 17(1): 1--31.
17. {van Rijsbergen, 1979} van Rijsbergen, C. J. (1979). Information Retrieval. Butterworths, London, England, 2nd edition.
18. {Voorhees and Harman, 1999} Voorhees, E. and Harman, D. (1999). Overview of the Seventh Text REtrieval Conference (TREC-7). In NIST Special Publication 500--242: The Seventh Text REtrieval Conference (TREC 7). Springer-Verlag, Heidelberg, Germany.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

    • Interaction devices

      • Graphics input devices

• Information systems

  • Information retrieval

    • Evaluation of retrieval results

    • Information retrieval query processing