Computational support for academic peer review: a perspective from artificial intelligence

Abstract

New tools tackle an age-old practice.

References

1. André, P., Zhang, H., Kim, J., Chilton, L.B., Dow, S.P. and Miller, R.C. Community clustering: Leveraging an academic crowd to form coherent conference sessions. In Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing (Palm Springs, CA, Nov. 7--9, 2013). B. Hartman and E. Horvitz, ed. AAAI, Palo Alto, CA.
2. Balog, K., Azzopardi, L. and de Rijke, M. Formal models for expert finding in enterprise corpora. In Proceedings of the 29^th Annual International ACM Conference on Research and Development in Information Retrieval (2006). ACM, New York, NY, 43--50.
3. Benferhat, S. and Lang, J. Conference paper assignment. International Journal of Intelligent Systems 16, 10 (2001), 1183--1192.
4. Blei, D.M, Ng, A.Y. and Jordan, M.I. Latent dirichlet allocation. J. Mach. Learn. Res. (Mar. 2003), 993--1022.
5. Bornmann, L., Bowman, B., Bauer, J., Marx, W., Schier, H. and Palzenberger, M. Standards for using bibliometrics in the evaluation of research institutes. Next Generation Metrics, 2013.
6. Boxwala, A.A., Dierks, M., Keenan, M., Jackson, S., Hanscom, R., Bates, D.W. and Sato, L. Review paper: Organization and representation of patient safety data: Current status and issues around generalizability and scalability. J. American Medical Informatics Association 11, 6 (2004), 468--478.
7. Brixey, J., Johnson, T. and Zhang, J. Evaluating a medical error taxonomy. In Proceedings of the American Medical Informatics Association Symposium, 2002.
8. Charlin, L. and Zemel, R. The Toronto paper matching system: An automated paper-reviewer assignment system. In Proceedings of ICML Workshop on Peer Reviewing and Publishing Models, 2013.
9. Charlin, L., Zemel, R. and Boutilier, C. A framework for optimizing paper matching. In Proceedings of the 27^th Annual Conference on Uncertainty in Artificial Intelligence (Corvallis, OR, 2011). AUAI Press, 86--95.
10. De Roure, D. Towards computational research objects. In Proceedings of the 1^st International Workshop on Digital Preservation of Research Methods and Artefacts (2013). ACM, New York, NY, 16--19.
11. Deng, H., King, I. and Lyu, M.R. Formal models for expert finding on DBLP bibliography data. In Proceedings of the 8^th IEEE International Conference on Data Mining (2008). IEEE Computer Society, Washington, D.C., 163--172.
12. Devedzić, V. Understanding ontological engineering. Commun. ACM 45, 4 (Apr. 2002), 136--144.
13. Di Mauro, N., Basile, T. and Ferilli, S. Grape: An expert review assignment component for scientific conference management systems. Innovations in Applied Artificial Intelligence. LNCS 3533 (2005). M. Ali and F. Esposito, eds. Springer, Berlin Heidelberg, 789--798.
14. Dumais S.T. and Nielsen, J. Automating the assignment of submitted manuscripts to reviewers. In Proceedings of the 15^th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (1992). ACM, New York, NY, 233--244.
15. Fang, H. and Zhai, C. Probabilistic models for expert finding. In Proceedings of the 29^th European Conference on IR Research (2007). Springer-Verlag, Berlin, Heidelberg, 418--430.
16. Ferilli, S., Di Mauro, N., Basile, T., Esposito, F. and Biba, M. Automatic topics identification for reviewer assignment. Advances in Applied Artificial Intelligence. LNCS 4031 (2006). M. Ali and R. Dapoigny, eds. Springer, Berlin Heidelberg, 721--730.
17. Flach, P. Machine Learning: The Art and Science of Algorithms That Make Sense of Data. Cambridge University Press, 2012.
18. Flach, P.A., Spiegler, S., Golénia, B., Price, S., Herbrich, J.G.R., Graepel, T. and Zaki, M. J. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. SIGKDD Explorations 11, 2 (Dec. 2009), 63--67.
19. Garg, N., Kavitha, T., Kumar, A., Mehlhorn, K., and Mestre, J. Assigning papers to referees. Algorithmica 58, 1 (Sept. 2010), 119--136.
20. Goldsmith, J. and Sloan, R.H. The AI conference paper assignment problem. In Proceedings of the 22^nd AAAI Conference on Artificial Intelligence (2007).
21. Harnad, S. Open access scientometrics and the U.K. research assessment exercise. Scientometrics 79, 1 (Apr. 2009), 147--156.
22. Hettich, S. and Pazzani, M.J. Mining for proposal reviewers: Lessons learned at the National Science Foundation. In Proceedings of the 12^th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2006). ACM, New York, NY, 862--871.
23. Jennings, C. Quality and value: The true purpose of peer review. Nature, 2006.
24. Karimzadehgan, M. and Zhai, C. Integer linear programming for constrained multi-aspect committee review assignment. Inf. Process. Manage. 48, 4 (July 2012), 725--740.
25. Karimzadehgan, M., Zhai, C. and Belford, G. Multi-aspect expertise matching for review assignment. In Proceedings of the 17^th ACM Conference on Information and Knowledge Management (2008). ACM, New York, NY 1113--1122.
26. Kou, N.M., U, L.H. Mamoulis, N. and Gong, Z. Weighted coverage based reviewer assignment. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, New York, NY, 2031--2046.
27. Langford, J. and Guzdial, M. The arbitrariness of reviews, and advice for school administrators. Commun. ACM 58, 4 (Apr. 2015), 12--13.
28. Lawrence, P.A. The politics of publication. Nature 422 (Mar. 2003), 259--261.
29. Ley, M. The DBLP computer science bibliography: Evolution, research issues, perspectives. In Proceedings of the 9^th International Symposium on String Processing and Information Retrieval (London, U.K., 2002). Springer-Verlag, 1--10.
30. Liu, X., Suel, T. and Memon, N. A robust model for paper reviewer assignment. In Proceedings of the 8^th ACM Conference on Recommender Systems (2014). ACM, New York, NY, 25--32.
31. Long, C., Wong, R.C., Peng, Y. and Ye, L. On good and fair paper-reviewer assignment. In Proceedings of the 2013 IEEE 13^th International Conference on Data Mining (Dallas, TX, Dec. 7--10, 2013), 1145--1150.
32. Mehlhorn, K., Vardi, M.Y. and Herbstritt, M. Publication culture in computing research (Dagstuhl Perspectives Workshop 12452). Dagstuhl Reports 2, 11 (2013).
33. Meyer, B., Choppy, C., Staunstrup, J. and van Leeuwen, J. Viewpoint: Research evaluation for computer science. Commun. ACM 52, 4 (Apr. 2009), 31--34.
34. Mimno, D. and McCallum, A. Expertise modeling for matching papers with reviewers. In Proceedings of the 13^th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 2007, 500--509.
35. Minka, T. Expectation propagation for approximate Bayesian inference. In Proceedings of the 17^th Conference in Uncertainty in Artificial Intelligence. J.S. Breese and D. Koller, Eds. Morgan Kaufmann, 2001, 362--369.
36. Price, S. and Flach, P.A. Mining and mapping the research landscape. In Proceedings of the Digital Research Conference. University of Oxford, Sept. 2013.
37. Price, S., Flach, P.A., Spiegler, S., Bailey, C. and Rogers, N. SubSift Web services and workflows for profiling and comparing scientists and their published works. Future Generation Comp. Syst. 29, 2 (2013), 569--581.
38. Pritchard, A. et al. Statistical bibliography or bibliometrics. J. Documentation 25, 4 (1969), 348--349.
39. Rodriguez, M.A and Bollen, J. An algorithm to determine peer-reviewers. In Proceedings of the 17^th ACM Conference on Information and Knowledge Management. ACM, New York, NY, 319--328.
40. Sidiropoulos, N.D. and Tsakonas, E. Signal processing and optimization tools for conference review and session assignment. IEEE Signal Process. Mag. 32, 3 (2015), 141--155.
41. Surpatean, A., Smirnov, E.N. and Manie, N. Master orientation tool. ECAI 242, Frontiers in Artificial Intelligence and Applications. L.De Raedt, C. Bessière, D. Dubois, P. Doherty, P. Frasconi, F. Heintz, and P.J.F. Lucas, Eds. IOS Press, 2012, 995--996.
42. Tang, W. Tang, J., Lei, T., Tan, C., Gao, B. and Li, T. On optimization of expertise matching with various constraints. Neurocomputing 76, 1 (Jan. 2012), 71--83.
43. Tang, W., Tang, J. and Tan, C. Expertise matching via constraint-based optimization. In Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (Vol 1). IEEE Computer Society, Washington, DC, 2010, 34--41.
44. Taylor, C.J. On the optimal assignment of conference papers to reviewers. Technical Report MS-CIS-08-30, Computer and Information Science Department, University of Pennsylvania, 2008.
45. Terry, D. Publish now, judge later. Commun. ACM 57, 1 (Jan. 2014), 44--46.
46. Vardi, M.Y. Scalable conferences. Commun. ACM 57, 1 (Jan. 2014), 5.
47. Yimam-Seid, D. and Kobsa, A. Expert finding systems for organizations: Problem and domain analysis and the DEMOIR approach. J. Organizational Computing and Electronic Commerce 13 (2003).

Index Terms

• Computing methodologies

  • Artificial intelligence

  • Machine learning

    • Machine learning approaches