Martín Abadi
ABSTRACT
TensorFlow is a powerful, programmable system for machine learning. This paper aims to provide the basics of a conceptual framework for understanding the behavior of TensorFlow models during training and inference: it describes an operational semantics, of the kind common in the literature on programming languages. More broadly, the paper suggests that a programming-language perspective is fruitful in designing and in explaining systems such as TensorFlow.
- M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. J´ozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. G. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. B. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. TensorFlow: Large-scale machine learning on heterogeneous distributed systems. CoRR, abs/1603.04467, 2016.Google Scholar
- M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. Tensorflow: A system for largescale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation, Proceedings, pages 265–283, 2016. Google ScholarDigital Library
- M. Abadi and M. Isard. Timely dataflow: A model. In Formal Techniques for Distributed Objects, Components, and Systems - 35th IFIP WG 6.1 International Conference, FORTE 2015, Proceedings, pages 131–145, 2015.Google Scholar
- M. Abadi and M. Isard. Timely rollback: Specification and verification. In K. Havelund, G. Holzmann, and R. Joshi, editors, NASA Formal Methods – 7th International Symposium, Proceedings, pages 19–34. Springer, 2015.Google Scholar
- Arvind and D. E. Culler. Dataflow architectures. In J. F. Traub, B. J. Grosz, B. W. Lampson, and N. J. Nilsson, editors, Annual Review of Computer Science Vol. 1, 1986, pages 225–253. Annual Reviews Inc., Palo Alto, CA, USA, 1986. Google ScholarDigital Library
- Arvind and R. S. Nikhil. Executing a program on the MIT taggedtoken dataflow architecture. IEEE Trans. Comput., 39(3):300–318, Mar. 1990. Google ScholarDigital Library
- F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. J. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, and Y. Bengio. Theano: new features and speed improvements. CoRR, abs/1211.5590, 2012.Google Scholar
- M. Giraud, D. G. Murray, and P. Tucker. control dependencies and assign new shape not working (using validate shape=false). Discussion at https://github.com/tensorflow/tensorflow/ issues/7782, 2017.Google Scholar
- T. T. Hildebrandt, P. Panangaden, and G. Winskel. A relational model of non-deterministic dataflow. Mathematical Structures in Computer Science, 14(5):613–649, 2004. Google ScholarDigital Library
- B. Jonsson. A fully abstract trace model for dataflow and asynchronous networks. Distributed Computing, 7(4):197–212, 1994. Google ScholarDigital Library
- G. Kahn. The semantics of a simple language for parallel programming. In J. L. Rosenfeld, editor, Information processing, pages 471– 475, Stockholm, Sweden, Aug 1974. North Holland, Amsterdam.Google Scholar
- M. Kudlur. Imperative programming in TensorFlow. Code repository at https://github.com/tensorflow/tensorflow/blob/ master/tensorflow/contrib/imperative, 2017.Google Scholar
- L. Lamport. Specifying Systems, The TLA+ Language and Tools for Hardware and Software Engineers. Addison-Wesley, 2002. Google ScholarDigital Library
- M. Looks, M. Herreshoff, D. Hutchins, and P. Norvig. Deep learning with dynamic computation graphs. CoRR, abs/1702.02181, 2017.Google Scholar
- D. G. Murray, F. McSherry, M. Isard, R. Isaacs, P. Barham, and M. Abadi. Incremental, iterative data processing with timely dataflow. Communications of the ACM, 59(10):75–83, 2016. Google ScholarDigital Library
- B. Recht, C. Ré, S. J. Wright, and F. Niu. HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent. In 25th Annual Conference on Neural Information Processing Systems, Proceedings, pages 693–701, 2011. Google ScholarDigital Library
- The XLA Team. XLA – TensorFlow compiled. Post in the Google Developers Blog, at https://developers.googleblog.com/2017/ 03/xla-tensorflow-compiled.html, 2017.Google Scholar
Index Terms
- A computational model for TensorFlow: an introduction
Recommendations
Machine learning for intelligent mobile user interfaces using TensorFlow
MobileHCI '17: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and ServicesOne key feature of TensorFlow includes the possibility to compile the trained model to run efficiently on mobile phones. This enables a wide range of opportunities for researchers and developers. In this tutorial, we teach attendees two basic steps to ...
Machine learning with tensorflow for mobile and ubiquitous interaction
MUM '17: Proceedings of the 16th International Conference on Mobile and Ubiquitous MultimediaDue to the increasing amount of sensors integrated into the environment and worn by the user, a sheer amount of context-sensitive data become available. While interpreting them with traditional methods (e.g., formulas and simple heuristics) is ...
Accelerated Machine Learning Using TensorFlow and SYCL on OpenCL Devices
IWOCL '17: Proceedings of the 5th International Workshop on OpenCLMachine learning is being used in more and more artificial intelligence applications. While existing machine learning frameworks mostly support NVIDIA CUDA GPUs, there has been little research dedicated to targeting other devices through open standards ...
Comments