TF-IDF uncovered: a study of theories and probabilities

ABSTRACT

Interpretations of TF-IDF are based on binary independence retrieval, Poisson, information theory, and language modelling. This paper contributes a review of existing interpretations, and then, TF-IDF is systematically related to the probabilities P(q|d) and P(d|q). Two approaches are explored: a space of independent, and a space of disjoint terms. For independent terms, an "extreme" query/non-query term assumption uncovers TF-IDF, and an analogy of P(d|q) and the probabilistic odds O(r|d, q) mirrors relevance feedback. For disjoint terms, a relationship between probability theory and TF-IDF is established through the integral + 1/x dx = log x. This study uncovers components such as divergence from randomness and pivoted document length to be inherent parts of a document-query independence (DQI) measure, and interestingly, an integral of the DQI over the term occurrence probability leads to TF-IDF.

References

1. Akiko Aizawa. An information-theoretic perspective of tf-idf measures. Information Processing and Management, 39:45--65, January 2003.
2. Gianni Amati and C. J. van Rijsbergen. Probabilistic models of information retrieval based on measuring the divergence from randomness. ACM TOIS, 20(4):357--389, October 2002.
3. K. Church and W Gale. Inverse document frequency (idf): A measure of deviation from poisson. In Third Workshop on Very Large Corpora, pages 121--130, 1995.
4. W.B. Croft and D.J. Harper. Using probabilistic models of document retrieval without relevance information. Journal of Documentation, 35:285--295, 1979.
5. Arjen de Vries and Thomas Roelleke. Relevance information: A loss of entropy but a gain for idf? In ACM SIGIR, Salvador, Brazil, 2005.
6. David A. Grossman and Ophir Frieder. Information Retrieval. Algorithms and Heuristics, 2nd ed., volume 15 of The Information Retrieval Series. Springer, 2004.
7. Djoerd Hiemstra. A probabilistic justification for using tf.idf term weighting in information retrieval. International Journal on Digital Libraries, 3(2):131--139, 2000.
8. John Lafferty and ChengXiang Zhai. Probabilistic Relevance Models Based on Document and Query Generation, chapter 1. Kluwer, 2003.
9. Qiaozhu Mei, Hui Fang, and ChengXiang Zhai. A study of Poisson query generation model for information retrieval. In ACM SIGIR, pages 319--326, New York, 2007.
10. J.M. Ponte and W.B. Croft. A language modeling approach to information retrieval. ACM SIGIR, pages 275--281, 1998.
11. S. E. Robertson and S. Walker. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. ACM SIGIR, pages 232--241, 1994.
12. S.E. Robertson. Understanding inverse document frequency: On theoretical arguments for idf. Journal of Documentation, 60:503--520, 2004.
13. S.E. Robertson and K. Sparck Jones. Relevance weighting of search terms. Journal of the American Society for Information Science, 27:129--146, 1976.
14. Thomas Roelleke. A frequency-based and a Poisson-based probability of being informative. In ACM SIGIR, pages 227--234, Toronto, Canada, 2003.
15. Thomas Roelleke and Jun Wang. A parallel derivation of probabilistic information retrieval models. In ACM SIGIR, pages 107--114, Seattle, USA, 2006.
16. S.K.M. Wong and Y.Y. Yao. On modeling information retrieval with probabilistic inference. ACM TOIS, 13(1):38--68, 1995.
17. Hugo Zaragoza, Djoerd Hiemstra, and Michael E. Tipping. Bayesian extension to the language model for ad hoc information retrieval. In ACM SIGIR, pages 4--9, Toronto, Canada, 2003.

Index Terms

• Information systems

  • Information retrieval

    • Retrieval models and ranking