An IR-based evaluation framework for web search query segmentation

ABSTRACT

This paper presents the first evaluation framework for Web search query segmentation based directly on IR performance. In the past, segmentation strategies were mainly validated against manual annotations. Our work shows that the goodness of a segmentation algorithm as judged through evaluation against a handful of human annotated segmentations hardly reflects its effectiveness in an IR-based setup. In fact, state-of the-art algorithms are shown to perform as good as, and sometimes even better than human annotations a fact masked by previous validations. The proposed framework also provides us an objective understanding of the gap between the present best and the best possible segmentation algorithm. We draw these conclusions based on an extensive evaluation of six segmentation strategies, including three most recent algorithms, vis-a-vis segmentations from three human annotators. The evaluation framework also gives insights about which segments should be necessarily detected by an algorithm for achieving the best retrieval results. The meticulously constructed dataset used in our experiments has been made public for use by the research community.

References

1. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In SIGIR '08, pages 667--674. ACM, 2008.
2. M. Bendersky, W. B. Croft, and D. A. Smith. Two-stage query segmentation for information retrieval. In SIGIR '09, pages 810--811. ACM, 2009.
3. S. Bergsma and Q. I. Wang. Learning noun phrase query segmentation. In EMNLP-CoNLL'07, pages 819--826, 2007.
4. D. J. Brenes, D. Gayo-Avello, and R. Garcia. On the fly query segmentation using snippets. In CERI '10, pages 259--266, 2010.
5. A. L. da Costa Carvalho, E. S. de Moura, and P. Calado. Using statistical features to find phrasal terms in text collections. JIDM, 1(3):583--597, 2010.
6. M. Hagen, M. Potthast, B. Stein, and C. Brautigam. The power of naive query segmentation. In SIGIR '10, pages 797--798. ACM, 2010.
7. M. Hagen, M. Potthast, B. Stein, and C. Brautigam. Query segmentation revisited. In WWW '11, pages 97--106, 2011.
8. M. Lease, J. Allan, and W. B. Croft. Regression rank: Learning to meet the opportunity of descriptive queries. In Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval}, ECIR '09, pages 90--101, Berlin, Heidelberg, 2009. Springer-Verlag.
9. Y. Li, B.-J. P. Hsu, C. Zhai, and K. Wang. Unsupervised query segmentation using clickthrough for information retrieval. In SIGIR '11, pages 285--294. ACM, 2011.
10. D. Metzler and W. B. Croft. A markov random field model for term dependencies. In SIGIR'05, pages 472--479, 2005.
11. N. Mishra, R. Saha Roy, N. Ganguly, S. Laxman, and M. Choudhury. Unsupervised query segmentation using only query logs. In WWW '11, pages 91--92. ACM, 2011.
12. K. M. Risvik, T. Mikolajewski, and P. Boros. Query segmentation for web search. In WWW (Posters), 2003.
13. B. Tan and F. Peng. Unsupervised query segmentation using generative language models and wikipedia. In WWW '08, pages 347--356. ACM, 2008.
14. E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Inf. Process. Manage.}, 36:697--716, September 2000.
15. C. Zhang, N. Sun, X. Hu, T. Huang, and T.-S. Chua. Query segmentation based on eigenspace similarity. In ACL/AFNLP (Short Papers)'09, pages 185--188, 2009.
16. W. Zhang, S. Liu, C. Yu, C. Sun, F. Liu, and W. Meng. Recognition and classification of noun phrases in queries for effective retrieval. In CIKM '07, pages 711--720. ACM, 2007.

Index Terms

• Information systems

  • Information retrieval

    • Information retrieval query processing

    • Retrieval models and ranking