This dissertation investigates the role of contextual information in the automated retrieval and display of full-text documents, using robust natural language processing algorithms to automatically detect structure in and assign topic labels to texts. Many long texts are comprised of complex topic and subtopic structure, a fact ignored by existing information access methods. I present two algorithms which detect such structure, and two visual display paradigms which use the results of these algorithms to show the interactions of multiple main topics, multiple subtopics, and the relations between main topics and subtopics. The first algorithm, called {\it TextTiling}, recognizes the subtopic structure of texts as dictated by their content. It uses domain-independent lexical frequency and distribution information to partition texts into multi-paragraph passages. The results are found to correspond well to reader judgments of major subtopic boundaries. The second algorithm assigns multiple main topic labels to each text, where the labels are chosen from pre-defined, intuitive category sets; the algorithm is trained on unlabeled text. A new iconic representation, called {\it TileBars} uses TextTiles to simultaneously and compactly display query term frequency, query term distribution and relative document length. This representation provides an informative alternative to ranking long texts according to their overall similarity to a query. For example, a user can choose to view those documents that have an extended discussion of one set of terms and a brief but overlapping discussion of a second set of terms. This representation also allows for relevance feedback on patterns of term distribution. TileBars display documents only in terms of words supplied in the user query. For a given retrieved text, if the query words do not correspond to its main topics, the user cannot discern in what context the query terms were used. For example, a query on {\sl contaminants} may retrieve documents whose main topics relate to nuclear power, food, or oil spills. To address this issue, I describe a graphical interface, called {\it Cougar}, that displays retrieved documents in terms of interactions among their automatically-assigned main topics, thus allowing users to familiarize themselves with the topics and terminology of a text collection.
Cited By
- Hearst M (1997). TextTiling, Computational Linguistics, 23:1, (33-64), Online publication date: 1-Mar-1997.
- Phelps T and Wilensky R Toward active, extensible, networked documents Proceedings of the first ACM international conference on Digital libraries, (100-108)
- Hirschberg J and Nakatani C A prosodic analysis of discourse segments in direction-giving monologues Proceedings of the 34th annual meeting on Association for Computational Linguistics, (286-293)
- Hearst M Multi-paragraph segmentation of expository text Proceedings of the 32nd annual meeting on Association for Computational Linguistics, (9-16)
- Hearst M Using categories to provide context for full-text retrieval results Intelligent Multimedia Information Retrieval Systems and Management - Volume 1, (115-130)
Recommendations
Finding structure in noisy text: topic classification and unsupervised clustering
This paper addresses two types of classification of noisy, unstructured text such as newsgroup messages: (1) spotting messages containing topics of interest, and (2) automatic conceptual organization of messages without prior knowledge of topics of ...