Bingzhe Li
M. Hassan Najafi
David J. Lilja
Skip Abstract Section
Abstract
With increased interests of neural networks, hardware implementations of neural networks have been investigated. Researchers pursue low hardware cost by using different technologies such as stochastic computing (SC) and quantization. More specifically, the quantization is able to reduce total number of trained weights and results in low hardware cost. SC aims to lower hardware costs substantially by using simple gates instead of complex arithmetic operations. However, the advantages of both quantization and SC in neural networks are not well investigated. In this article, we propose a new stochastic multiplier with simple CMOS transistors called the stochastic hybrid multiplier for quantized neural networks. The new design uses the characteristic of quantized weights and tremendously reduces the hardware cost of neural networks. Experimental results indicate that our stochastic design achieves about 7.7x energy reduction compared to its counterpart binary implementation while maintaining slightly higher recognition error rates than the binary implementation. Compared to previous stochastic neural network implementations, our work derives at least 4x, 9x, and 10x reduction in terms of area, power, and energy, respectively.
- Armin Alaghi and John P. Hayes. 2013. Exploiting correlation in stochastic circuit design. In Proceedings of the IEEE 31st International Conference on Computer Design (ICCD’13). IEEE, Los Alamitos, CA, 39--46.Google Scholar
- Armin Alaghi and John P. Hayes. 2013. Survey of stochastic computing. ACM Transactions on Embedded Computing Systems 12, 2s (2013), Article 92. Google ScholarDigital Library
- Bradley D. Brown and Howard C. Card. 2001. Stochastic neural computation. I. Computational elements. IEEE Transactions on Computers 50, 9 (2001), 891--905. Google ScholarDigital Library
- J. A. Dickson, R. D. McLeod, and H. C. Card. 1993. Stochastic arithmetic implementations of neural networks with in situ learning. In Proceedings of the IEEE International Conference on Neural Networks. IEEE, Los Alamitos, CA, 711--716.Google Scholar
- Brian R. Gaines. 1969. Stochastic computing systems. Advances in Information Systems Science 2, 2 (1969), 37--172.Google ScholarCross Ref
- Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv:1510.00149.Google Scholar
- Geoffrey E. Hinton. 2012. A practical guide to training restricted Boltzmann machines. In Neural Networks: Tricks of the Trade. Springer, 599--619.Google Scholar
- Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2017. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research 18, 1 (2017), 6869--6898. Google ScholarDigital Library
- Kyuyeon Hwang and Wonyong Sung. 2014. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Proceedings of the IEEE Workshop on Signal Processing Systems (SiPS’14). IEEE, Los Alamitos, CA, 1--6.Google ScholarCross Ref
- Devon Jenson and Marc Riedel. 2016. A deterministic approach to stochastic computation. In Proceedings of the IEEE/ACM International Conference on Computer-Aided Design (ICCAD’16). IEEE, Los Alamitos, CA, 1--8. Google ScholarDigital Library
- Seul Jung and Sung Su Kim. 2007. Hardware implementation of a real-time neural network controller with a DSP and an FPGA for nonlinear systems. IEEE Transactions on Industrial Electronics 54, 1 (2007), 265--271.Google ScholarCross Ref
- Kyounghoon Kim, Jungki Kim, Joonsang Yu, Jungwoo Seo, Jongeun Lee, and Kiyoung Choi. 2016. Dynamic energy-accuracy trade-off using stochastic computing in deep neural networks. In Proceedings of the 53rd Annual Design Automation Conference. ACM, New York, NY, 124. Google ScholarDigital Library
- Kyounghoon Kim, Jongeun Lee, and Kiyoung Choi. 2016. An energy-efficient random number generator for stochastic circuits. In Proceedings of the 2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC’16). IEEE, Los Alamitos, CA, 256--261.Google ScholarDigital Library
- Yann LeCun and Corinna Cortes. 2010. Database of Handwritten Digits. Retrieved March 11, 2019 from http://yann.lecun.com/exdb/mnist.Google Scholar
- Vincent T. Lee, Armin Alaghi, John P. Hayes, Visvesh Sathe, and Luis Ceze. 2017. Energy-efficient hybrid stochastic-binary neural networks for near-sensor computing. In Proceedings of the 2017 Design, Automation, and Test in Europe Conference and Exhibition (DATE’17). IEEE, Los Alamitos, CA, 13--18. Google ScholarDigital Library
- Bingzhe Li, M. Hassan Najafi, and David J. Lilja. 2015. An FPGA implementation of a restricted Boltzmann machine classifier using stochastic bit streams. In Proceedings of the 2015 IEEE 26th International Conference on Application-Specific Systems, Architectures, and Processors (ASAP’15). IEEE, Los Alamitos, CA, 68--69.Google Scholar
- Bingzhe Li, M. Hassan Najafi, and David J. Lilja. 2016. Using stochastic computing to reduce the hardware requirements for a restricted Boltzmann machine classifier. In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. ACM, New York, NY, 36--41. Google ScholarDigital Library
- B. Li, M. H. Najafi, B. Yuan, and D. J. Lilja. 2018. Quantized neural networks with new stochastic multipliers. In Proceedings of the 2018 19th International Symposium on Quality Electronic Design (ISQED’18). 376--382.Google Scholar
- Bingzhe Li, Yaobin Qin, Bo Yuan, and David J. Lilja. 2017. Neural network classifiers using stochastic computing with a hardware-oriented approximate activation function. In Proceedings of the 2017 IEEE 35th International Conference on Computer Design (ICCD’17). IEEE, Los Alamitos, CA, 97--104.Google Scholar
- Ji Li, Zihao Yuan, Zhe Li, Caiwen Ding, Ao Ren, Qinru Qiu, Jeffrey Draper, et al. 2017. Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks. arXiv:1703.04135.Google Scholar
- Peng Li, David J. Lilja, Weikang Qian, Kia Bazargan, and Marc D. Riedel. 2014. Computation on stochastic bit streams digital image processing case studies. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 22, 3 (2014), 449--462. Google ScholarDigital Library
- Peng Li, David J. Lilja, Weikang Qian, Marc D. Riedel, and Kia Bazargan. 2012. Logical computation on stochastic bit streams with linear finite-state machines. IEEE Transactions on Computers 63, 6(2012), 1474--1486. Google ScholarDigital Library
- Zhe Li, Ao Ren, Ji Li, Qinru Qiu, Yanzhi Wang, and Bo Yuan. 2016. DSCNN: Hardware-oriented optimization for stochastic computing based deep convolutional neural networks. In Proceedings of the IEEE 34th International Conference on Computer Design (ICCD’16). IEEE, Los Alamitos, CA, 678--681.Google ScholarCross Ref
- Siting Liu and Jie Han. 2017. Energy efficient stochastic computing with Sobol sequences. In Proceedings of the 2017 Design, Automation, and Test in Europe Conference and Exhibition (DATE’17). IEEE, Los Alamitos, CA, 650--653. Google ScholarDigital Library
- Ankit Mondal and Ankur Srivastava. 2017. Power optimizations in MTJ-based neural networks through stochastic computing. In Proceedings of the IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED’17). IEEE, Los Alamitos, CA, 1--6.Google ScholarCross Ref
- M. Hassan Najafi, Shiva Jamali-Zavareh, David J. Lilja, Marc D. Riedel, Kia Bazargan, and Ramesh Harjani. 2017. Time-encoded values for highly efficient stochastic circuits. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 25, 5 (2017), 1644--1657. Google ScholarDigital Library
- Amos R. Omondi and Jagath Chandana Rajapakse. 2006. FPGA Implementations of Neural Networks. Vol. 365. Springer. Google ScholarDigital Library
- Weikang Qian, Xin Li, Marc D. Riedel, Kia Bazargan, and David J. Lilja. 2011. An architecture for fault-tolerant computation with stochastic logic. IEEE Transactions on Computers 60, 1 (2011), 93--105. Google ScholarDigital Library
- Weikang Qian and Marc D. Riedel. 2010. Synthesizing logical computation on stochastic bit streams. http://www.mriedel.ece.umn.edu/wiki/images/6/64/Qian_Riedel_Synthesizing_Logical_Computation_on_Stochastic_Bit_Streams.pdf.Google Scholar
- Ruslan Salakhutdinov and Geoffrey Hinton. 2009. Deep Boltzmann machines. In Proceedings of the 12th Interantional Conference on Artificial Intelligence and Statistics. 448--455.Google Scholar
- James E. Stine, Ivan Castellanos, Michael Wood, Jeff Henson, Fred Love, W. Rhett Davis, Paul D. Franzon, et al. 2007. FreePDK: An open-source variation-aware design kit. In Proceedings of the IEEE International Conference on Microelectronic Systems Education (MSE’07). IEEE, Los Alamitos, CA, 173--174. Google ScholarDigital Library
- Rangharajan Venkatesan, Swagath Venkataramani, Xuanyao Fong, Kaushik Roy, and Anand Raghunathan. 2015. Spintastic: Spin-based stochastic logic for energy-efficient computing. In Proceedings of the Design, Automation, and Test in Europe Conference and Exhibition (DATE’15). IEEE, Los Alamitos, CA, 1575--1578. Google ScholarDigital Library
- Meng Yang, Bingzhe Li, David J. Lilja, Bo Yuan, and Weikang Qian. 2018. Towards theoretical cost limit of stochastic number generators for stochastic computing. In Proceedings of the 2018 IEEE Computer Society Annual Symposium on VLSI (ISVLSI’18). IEEE, Los Alamitos, CA, 154--159.Google ScholarCross Ref
- Dong Yu, Frank Seide, Gang Li, and Li Deng. 2012. Exploiting sparseness in deep neural networks for large vocabulary speech recognition. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP’12). IEEE, Los Alamitos, CA, 4409--4412.Google ScholarCross Ref
- Reto Zimmermann and Wolfgang Fichtner. 1997. Low-power logic styles: CMOS versus pass-transistor logic. IEEE Journal of Solid-State Circuits 32, 7 (1997), 1079--1090.Google ScholarCross Ref
Index Terms
- Low-Cost Stochastic Hybrid Multiplier for Quantized Neural Networks
Recommendations
Neural Network Classifiers Using a Hardware-Based Approximate Activation Function with a Hybrid Stochastic Multiplier
Special Issue on Emerging Networks-on-Chip and Regular PapersNeural networks are becoming prevalent in many areas, such as pattern recognition and medical diagnosis. Stochastic computing is one potential solution for neural networks implemented in low-power back-end devices such as solar-powered devices and ...
Bayesian asymmetric quantized neural networks
Highlights- M-ary quantized neural network is proposed with adjustable M to balance between end performance and implementation cost.
AbstractThis paper develops a robust model compression for neural networks via parameter quantization. Traditionally, quantized neural networks (QNN) were constructed by binary or ternary weights where the weights were deterministic. This ...
Reconfigurable hardware for neural networks: binary versus stochastic
This paper is focused on hardware implementation of neural networks. We propose a reconfigurable, low-cost and readily available hardware architecture for an artificial neuron. For this purpose, we use field-programmable gate arrays i.e. FPGAs. As the ...
Comments