Mining for proposal reviewers: lessons learned at the national science foundation

ABSTRACT

In this paper, we discuss a prototype application deployed at the U.S. National Science Foundation for assisting program directors in identifying reviewers for proposals. The application helps program directors sort proposals into panels and find reviewers for proposals. To accomplish these tasks, it extracts information from the full text of proposals both to learn about the topics of proposals and the expertise of reviewers. We discuss a variety of alternatives that were explored, the solution that was implemented, and the experience in using the solution within the workflow of NSF.

References

1. Willett, P. (1998). Recent Trends in Hierarchic Document Clustering: A Critical Review, Information Processing and Management, 24(5), 577--597.
2. Larsen, B. and Chinatsu A. (1999). Fast and effective text mining using linear-time document clustering, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 16--22.
3. Hopcroft, J., Khan, O., Kulis, B. & Selman, B. Tracking evolving communities in large linked networks. Proc. Natl Acad. Sci. USA 101(Suppl.1), 5249--5253
4. Bradley, P., Bennett, P and. Demiriz., A. (2000) Constrained k-means clustering. Technical report, MSR-TR-2000-6 5 Microsoft Research.
5. Banerjee, A. & Ghosh, J. (2002). Frequency Sensitive Competitive Learning for Clustering on High-dimensional Hypersphere, International Joint Conference on Neural Networks (IJCNN), pp. 1590--95.
6. Furnas, G. W., Landauer, T. K., Gomez, L. M., Dumais, S. T. (1987): The vocabulary problem in human-system communication. Commun. ACM 30. 964--971
7. Ding, W., and Marchionini, G. A Study on Video Browsing Strategies. Technical Report UMIACS-TR-97-40, University of Maryland, College Park, MD, 1997.
8. Furnas, G. W., Landauer, T. K., Gomez, L. M., Dumais, S. T. (1987): The vocabulary problem in human-system communication. Communications of the ACM 30. 964--971
9. van de Stadt, R. (2000). CyberChair, an Online Submission and Reviewing System or: A Program Chair's Best Friend, WWW9.
10. Salton, G., & McGill, MJ (1983). Introduction to modern information retrieval. NY: McGraw-Hill
11. Porter, M. F., (1980), An algorithm for suffix stripping, Program, 14(3) :130--137
12. Giles, C. Bollacker, K., Lawrence, S. (1998). CiteSeer: An Automatic Citation Indexing System. Third ACM Conference on Digital Libraries, pp. 89--98, 1998.
13. Rocchio, J. (1971) Relevance feedback in information retrieval, in. The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice-Hall Inc., pg 313--323.
14. Segal, R and Kephart, J (1999). MailCat: An Intelligent Assistant for Organizing E-Mail. In Proceedings of the Third International Conference on Autonomous Agents.
15. Basu, C., Hirsh, H., Cohen, W., and Nevill-Manning, C., (1999). Recommending Papers by Mining the Web, Proc. IJCAI Workshops on Learning About Users and Machine Learning for Information Filtering, IJCAI 99, Stockholm, Sweden.
16. Geller, J. and Scherl, R., 1997 Challenge: Technology for Automated Reviewer Selection, IJCAI 1997 55--61
17. Dumais, S., Nielsen, J. (1992, Automating the Assignment of Submitted Manuscripts to Reviewers Proceedings of the 15th annual international ACM SIGIR conference on Research and development in information retrieval: 233--244
18. Steyvers, M., Smyth, P., Griffiths, T. (2004) Probabilistic Author-Topic Models for Information Discovery KDD'04, Seattle, Washington USA.
19. Mann, G., Mimno, D. and McCallum, A (in press). Bibliometric Impact Measures Leveraging Topic Analysis. Joint Conference on Digital Libraries (JCDL).
20. Carbonell, J. and Goldstein, J (1998). The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries, SIGIR'98, Melbourne Australia

Index Terms

• Information systems

  • Information systems applications

    • Data mining