David Newman
Sarvnaz Karimi
ABSTRACT
Topic models could have a huge impact on improving the ways users find and discover content in digital libraries and search interfaces through their ability to automatically learn and apply subject tags to each and every item in a collection, and their ability to dynamically create virtual collections on the fly. However, much remains to be done to tap this potential, and empirically evaluate the true value of a given topic model to humans. In this work, we sketch out some sub-tasks that we suggest pave the way towards this goal, and present methods for assessing the coherence and interpretability of topics learned by topic models. Our large-scale user study includes over 70 human subjects evaluating and scoring almost 500 topics learned from collections from a wide range of genres and domains. We show how scoring model -- based on pointwise mutual information of word-pair using Wikipedia, Google and MEDLINE as external data sources - performs well at predicting human scores. This automated scoring of topics is an important first step to integrating topic modeling into digital libraries
- L. AlSumait, D. Barbará, J. Gentle, and C. Domeniconi. Topic significance ranking of LDA generative models. In ECML/PKDD (1), pages 67--82, 2009.Google Scholar
- D. Andrzejewski, X. Zhu, and M. Craven. Incorporating domain knowledge into topic modeling via Dirichlet forest priors. In ICML, page 4, 2009. Google ScholarDigital Library
- T. Armstrong, A. Moffat, W. Webber, and J. Zobel. Improvements that don't add up: ad-hoc retrieval results since 1998. In CIKM, pages 601--610, 2009. Google ScholarDigital Library
- D. Blei and J. Lafferty. Dynamic topic models. In ICML, pages 113--120, 2006. Google ScholarDigital Library
- D. Blei, A. Ng, and M. Jordan. Latent Dirichlet allocation. JMLR, 3:993--1022, 2003. Google ScholarDigital Library
- W. Buntine and A. Jakulin. Applying discrete PCA in data analysis. In UAI, pages 59--66, Banff, Canada, 2004. Google ScholarDigital Library
- J. Chang, J. Boyd-Graber, S. Gerrish, C. Wang, and D. Blei. Reading tea leaves: How humans interpret topic models. In NIPS, pages 288--296, 2009.Google ScholarDigital Library
- T. Griffiths and M. Steyvers. Finding scientific topics. In PNAS, volume 101, pages 5228--5235, 2004.Google ScholarCross Ref
- Q. Mei, X. Shen, and C. Zhai. Automatic labeling of multinomial topic models. In SIGKDD, pages 490--499, 2007. Google ScholarDigital Library
- D. Mimno and A. McCallum. Organizing the OCA: learning faceted subjects from a library of digital books. In JCDL, pages 376--385, 2007. Google ScholarDigital Library
- D. Newman, T. Baldwin, L. Cavedon, S. Karimi, D. Martinez, and J. Zobel. Visualizing document collections and search results using topic mapping. Journal of Web Semantics, to appear.Google Scholar
- D. Newman, K. Hagedorn, C. Chemudugunta, and P. Smyth. Subject metadata enrichment using statistical topic models. In JCDL, pages 366--375, 2007. Google ScholarDigital Library
- D. Newman, S. Karimi, and L. Cavedon. External evaluation of topic models. In ADCS, pages 11--18, 2009.Google Scholar
- D. Newman, J. Lau, K. Grieser, and T. Baldwin. Automatic evaluation of topic coherence. In NAACL HLT 2010, Los Angeles, USA, to appear. Google ScholarDigital Library
- Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical Dirichlet processes. JASA, 101(476):1566--1581, 2006.Google Scholar
- H. Wallach, D. Mimno, and A. McCallum. Rethinking LDA: Why priors matter. In NIPS, pages 1973--1981, 2009.Google ScholarDigital Library
Index Terms
- Evaluating topic models for digital libraries
Recommendations
Topic modelling for qualitative studies
Qualitative studies, such as sociological research, opinion analysis and media studies, can benefit greatly from automated topic mining provided by topic models such as latent Dirichlet allocation LDA. However, examples of qualitative studies that ...
Topic sentiment mixture: modeling facets and opinions in weblogs
WWW '07: Proceedings of the 16th international conference on World Wide WebIn this paper, we define the problem of topic-sentiment analysis on Weblogs and propose a novel probabilistic model to capture the mixture of topics and sentiments simultaneously. The proposed Topic-Sentiment Mixture (TSM) model can reveal the latent ...
Investigating task performance of probabilistic topic models: an empirical study of PLSA and LDA
AbstractProbabilistic topic models have recently attracted much attention because of their successful applications in many text mining tasks such as retrieval, summarization, categorization, and clustering. Although many existing studies have reported ...
Comments