ModelDB: a system for machine learning model management

ABSTRACT

Building a machine learning model is an iterative process. A data scientist will build many tens to hundreds of models before arriving at one that meets some acceptance criteria (e.g. AUC cutoff, accuracy threshold). However, the current style of model building is ad-hoc and there is no practical way for a data scientist to manage models that are built over time. As a result, the data scientist must attempt to "remember" previously constructed models and insights obtained from them. This task is challenging for more than a handful of models and can hamper the process of sensemaking. Without a means to manage models, there is no easy way for a data scientist to answer questions such as "Which models were built using an incorrect feature?", "Which model performed best on American customers?" or "How did the two top models compare?" In this paper, we describe our ongoing work on ModelDB, a novel end-to-end system for the management of machine learning models. ModelDB clients automatically track machine learning models in their native environments (e.g. scikit-learn, spark.ml), the ModelDB backend introduces a common layer of abstractions to represent models and pipelines, and the ModelDB frontend allows visual exploration and analyses of models via a web-based interface.

References

1. L. Bavoil, S. P. Callahan, P. J. Crossno, J. Freire, C. E. Scheidegger, C. T. Silva, and H. T. Vo. Vistrails: Enabling interactive multiple-view visualizations. In Visualization, 2005. VIS 05. IEEE, pages 135--142. IEEE, 2005.
2. S. P. Callahan, J. Freire, E. Santos, C. E. Scheidegger, C. T. Silva, and H. T. Vo. Managing the evolution of dataflows with vistrails. In Data Engineering Workshops, 2006. Proceedings. 22nd International Conference on, pages 71--71. IEEE, 2006.
3. S. P. Callahan, J. Freire, E. Santos, C. E. Scheidegger, C. T. Silva, and H. T. Vo. Vistrails: visualization meets data management. In Proceedings of the 2006 ACM SIGMOD international conference on Management of data, pages 745--747. ACM, 2006.
4. M. Derthick and S. F. Roth. Enhancing data exploration with a branching history of user operations, 2001.
5. J. Heer, J. D. Mackinlay, C. Stolte, and M. Agrawala. Graphical histories for visualization: Supporting analysis, communication, and evaluation. Visualization and Computer Graphics, IEEE Transactions on, 14(6):1189--1196, 2008.
6. J. Heer and B. Shneiderman. Interactive dynamics for visual analysis. Commun. ACM, 55(4):45--54, Apr. 2012.
7. M. Kreuseler, T. Nocke, and H. Schumann. A history mechanism for visual data mining. In Proceedings of the IEEE Symposium on Information Visualization, INFOVIS '04, pages 49--56, Washington, DC, USA, 2004. IEEE Computer Society.
8. B. Ludäscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. A. Lee, J. Tao, and Y. Zhao. Scientific workflow management and the kepler system. Concurrency and Computation: Practice and Experience, 18(10):1039--1065, 2006.
9. P. Pirolli and S. Card. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, volume 5, pages 2--4, 2005.
10. A. C. Robinson and C. Weaver. Re-visualization: Interactive visualization of the process of visual analysis. In Proceedings of the GIScience Workshop on Visual Analytics and Spatial Decision Support, 2006.
11. B. Shneiderman. The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings of the 1996 IEEE Symposium on Visual Languages, VL '96, pages 336--, Washington, DC, USA, 1996. IEEE Computer Society.
12. K. Wolstencroft, R. Haines, D. Fellows, A. Williams, D. Withers, S. Owen, S. Soiland-Reyes, I. Dunlop, A. Nenadic, P. Fisher, et al. The taverna workflow suite: designing and executing workflows of web services on the desktop, web or in the cloud. Nucleic acids research, page gkt328, 2013.