New ranking algorithms for parsing and tagging: kernels over discrete structures, and the voted perceptron

ABSTRACT

This paper introduces new learning algorithms for natural language processing based on the perceptron algorithm. We show how the algorithms can be efficiently applied to exponential sized representations of parse trees, such as the "all subtrees" (DOP) representation described by (Bod 1998), or a representation tracking all sub-fragments of a tagged sentence. We give experimental results showing significant improvements on two tasks: parsing Wall Street Journal text, and named-entity extraction from web data.

References

1. Aizerman, M., Braverman, E., & Rozonoer, L. (1964). Theoretical Foundations of the Potential Function Method in Pattern Recognition Learning. In Automation and Remote Control, 25:821--837.
2. Bod, R. (1998). Beyond Grammar: An Experience-Based Theory of Language. CSLI Publications/Cambridge University Press.
3. Bod, R. (2001). What is the Minimal Set of Fragments that Achieves Maximal Parse Accuracy? In Proceedings of ACL 2001.
4. Borthwick, A., Sterling, J., Agichtein, E., and Grishman, R. (1998). Exploiting Diverse Knowledge Sources via Maximum Entropy in Named Entity Recognition. Proc. of the Sixth Workshop on Very Large Corpora.
5. Charniak, E. (2000). A maximum-entropy-inspired parser. In Proceedings of NAACL 2000.
6. Collins, M. 1999. Head-Driven Statistical Models for Natural Language Parsing. PhD Dissertation, University of Pennsylvania.
7. Collins, M. (2000). Discriminative Reranking for Natural Language Parsing. Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000).
8. Collins, M., and Duffy, N. (2001). Convolution Kernels for Natural Language. In Proceedings of Neural Information Processing Systems (NIPS 14).
9. Collins, M. (2002a). Ranking Algorithms for Named--Entity Extraction: Boosting and the Voted Perceptron. In Proceedings of ACL 2002.
10. Collins, M. (2002b). Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with the Perceptron Algorithm. In Proceedings of EMNLP 2002.
11. Cristianini, N., and Shawe-Tayor, J. (2000). An introduction to Support Vector Machines and other kernel-based learning methods. Cambridge University Press.
12. Freund, Y. & Schapire, R. (1999). Large Margin Classification using the Perceptron Algorithm. In Machine Learning, 37(3):277--296.
13. Freund, Y., Iyer, R., Schapire, R. E., & Singer, Y. (1998). An efficient boosting algorithm for combining preferences. In Machine Learning: Proceedings of the Fifteenth International Conference. San Francisco: Morgan Kaufmann.
14. Goodman, J. (1996). Efficient algorithms for parsing the DOP model. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 143--152.
15. Haussler, D. (1999). Convolution Kernels on Discrete Structures. Technical report, University of Santa Cruz.
16. Johnson, M., Geman, S., Canon, S., Chi, S., & Riezler, S. (1999). Estimators for stochastic 'unification-based" grammars. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics.
17. Johnson, M. (2002). The DOP estimation method is biased and inconsistent. Computational Linguistics, 28, 71--76.
18. Lodhi, H., Christianini, N., Shawe-Taylor, J., & Watkins, C. (2001). Text Classification using String Kernels. In Advances in Neural Information Processing Systems 13, MIT Press.
19. Marcus, M., Santorini, B., & Marcinkiewicz, M. (1993). Building a large annotated corpus of english: The Penn treebank. Computational Linguistics, 19, 313--330.
20. Ratnaparkhi, A. (1996). A maximum entropy part-of-speech tagger. In Proceedings of the empirical methods in natural language processing conference.
21. Rosenblatt, F. 1958. The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain. Psychological Review, 65, 386--408. (Reprinted in Neurocomputing (MIT Press, 1998).)

• Computing methodologies

  • Artificial intelligence

• Hardware

  • Power and energy

    • Power estimation and optimization