Kangjun Bai
Skip Abstract Section
Abstract
Neuromorphic computing, which is built on a brain-inspired silicon chip, is uniquely applied to keep pace with the explosive escalation of algorithms and data density on machine learning. Reservoir computing, an emerging computing paradigm based on the recurrent neural network with proven benefits across multifaceted applications, offers an alternative training mechanism only at the readout stage. In this work, we successfully design and fabricate an energy-efficient analog delayed feedback reservoir (DFR) computing system, which is built upon a temporal encoding scheme, a nonlinear transfer function, and a dynamic delayed feedback loop. Measurement results demonstrate its high energy efficiency with rich dynamic behaviors, making the designed system a candidate for low power embedded applications. The system performance, as well as the robustness, are studied and analyzed through the Monte Carlo simulation. The chaotic time series prediction benchmark, NARMA10, is examined through the proposed DFR computing system, and exhibits a 36%−85% reduction on the error rate compared to state-of-the-art DFR computing system designs. To the best of our knowledge, our work represents the first analog integrated circuit (IC) implementation of the DFR computing system.
- Abdulrahman Alalshekmubarak and Leslie S. Smith. 2014. On improving the classification capability of reservoir computing for arabic speech recognition. In Proceedings of the International Conference on Artificial Neural Networks. Springer, 225--232.Google Scholar
- Miquel L. Alomar, Vincent Canals, Nicolas Perez-Mora, Víctor Martínez-Moll, and Josep L. Rosselló. 2016. FPGA-based stochastic echo state networks for time-series forecasting. Comput. Intell. Neurosci. 2016 (2016), 1--14. Google ScholarDigital Library
- Pablo Amil, Cecilia Cabeza, and Arturo C. Marti. 2015. Exact discrete-time implementation of the Mackey--glass delayed model. IEEE Trans. Circ. Syst. 62, 681--685.Google Scholar
- Lennert Appeltant, Miguel Cornelles Soriano, Guy Van der Sande, Jan Danckaert, Serge Massar, Joni Dambre, Benjamin Schrauwen, Claudio R. Mirasso, and Ingo Fischer. 2011. Information processing using a single dynamical node as complex system. Nature Commun. 2, 468.Google ScholarCross Ref
- Kangjun Bai, Jialing Li, Kian Hamedani, and Yang Yi. 2018. Enabling a new era of brain-inspired computing: Energy-efficient spiking neural network with ring topology. In Proceedings of the 55th Annual Design Automation Conference. ACM, 166. Google ScholarDigital Library
- Kangjun Bai and Yang Yi. 2018. A path to energy-efficient spiking delayed feedback reservoir computing system for brain-inspired neuromorphic processors. In Proceedings of 19th International Symposium in Quality Electronic Design (ISQED’18).Google ScholarCross Ref
- Rezaul Begg and Joarder Kamruzzaman. 2005. A machine learning approach for automated recognition of movement patterns using basic, kinetic and kinematic gait data. J. Biomech. 38 401--408.Google Scholar
- Anne Beuter, Jacques Bélair, and Christiane Labrie. 1993. Feedback and delays in neurological diseases: A modeling study using dynamical systems. Bull. Math. Biol. 55, 525--541.Google Scholar
- Kristine E. Callan, Lucas Illing, Zheng Gao, Daniel J. Gauthier, and Eckehard Schöll. 2010. Broadband chaos generated by an optoelectronic oscillator. Phys. Rev. Lett. 104, 113901.Google ScholarCross Ref
- Qiuwen Chen, Qinru Qiu, Hai Li, and Qing Wu. 2013. A neuromorphic architecture for anomaly detection in autonomous large-area traffic monitoring. In Proceedings of the International Conference on Computer-Aided Design. IEEE Press, 202--205. Google ScholarDigital Library
- Myonglae Chu, Byoungho Kim, Sangsu Park, Hyunsang Hwang, Moongu Jeon, Byoung Hun Lee, and Byung-Geun Lee. 2015. Neuromorphic hardware system for visual pattern recognition with memristor array and CMOS neuron. IEEE Trans. Industr. Electron. 62, 2410--2419.Google ScholarCross Ref
- Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, and Shweta Jain. 2018. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 38, 82--99.Google ScholarCross Ref
- Miyuru Dayarathna, Yonggang Wen, and Rui Fan. 2016. Data center energy consumption modeling: A survey. IEEE Commun. Surveys Tutor. 18, 732--794.Google ScholarDigital Library
- Sukru Burc Eryilmaz, Siddharth Joshi, Emre Neftci, Weier Wan, Gert Cauwenberghs, and H.-S. Philip Wong. 2016. Neuromorphic architectures with electronic synapses. In Proceedings of the 17th International Symposium on Quality Electronic Design (ISQED’16). IEEE, 118--123.Google ScholarCross Ref
- Steve K. Esser, Rathinakumar Appuswamy, Paul Merolla, John V. Arthur, and Dharmendra S. Modha. 2015. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems. NIPS, 1117--1125. Google ScholarDigital Library
- Arfan Ghani. 2010. Neuro-inspired speech recognition based on reservoir computing. In Advances in Speech Recognition, InTech.Google Scholar
- C. Lee Giles and Tom Maxwell. 1987. Learning, invariance, and generalization in high-order neural networks. Appl. Optics 26, 4972--4978.Google ScholarCross Ref
- Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. 2016. Deep Learning, MIT Press, Cambridge. Google ScholarDigital Library
- Alireza Goudarzi, Peter Banda, Matthew R. Lakin, Christof Teuscher, and Darko Stefanovic. 2014. A comparative study of reservoir computing for temporal signal processing. Arxiv preprint Arxiv:1401.2224.Google Scholar
- Alireza Goudarzi, Matthew R. Lakin, and Darko Stefanovic. 2014. Reservoir computing approach to robust computation using unreliable nanoscale networks. In Proceedings of the International Conference on Unconventional Computation and Natural Computation. Springer, 164--176.Google ScholarCross Ref
- Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning. ACM, 369--376. Google ScholarDigital Library
- Lyudmila Grigoryeva, Julie Henriques, Laurent Larger, and Juan-Pablo Ortega. 2015. Optimal nonlinear information processing capacity in delay-based reservoir computers, Sci. Rep. 5, 12858.Google Scholar
- Hermann Haken. 2002. Brain dynamics (synchronisation and activity patterns in pulse-coupled neural nets with delays and noise). Springer Series in Synergetics. Google ScholarDigital Library
- Simon S. Haykin. 2001. Neural Networks: A Comprehensive Foundation, Tsinghua University Press. Google ScholarDigital Library
- Rainer Hegger, Martin J. Bünner, Holger Kantz, and Antonino Giaquinta. 1998. Identifying and modeling delay feedback systems. Phys. Rev. Lett. 81, 558.Google ScholarCross Ref
- Xavier Hinaut and Peter F. Dominey. 2012. On-line processing of grammatical structure using reservoir computing. In Proceedings of the International Conference on Artificial Neural Networks. Springer, 596--603. Google ScholarDigital Library
- Miao Hu, John Paul Strachan, Zhiyong Li, Emmanuelle M. Grafals, Noraica Davila, Catherine Graves, Sity Lam, Ning Ge, Jianhua Joshua Yang, and R. Stanley Williams. 2016. Dot-product engine for neuromorphic computing: Programming 1T1M crossbar to accelerate matrix-vector multiplication. In Proceedings of the 53rd Annual Design Automation Conference. ACM, 19. Google ScholarDigital Library
- Kensuke Ikeda and Kenji Matsumoto. 1987. High-dimensional chaotic behavior in systems with time-delayed feedback. Physica D: Nonlin. Phenom. 29, 223--235. Google ScholarDigital Library
- Herbert Jaeger. 2001a. The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. German National Research Center for Information Technology GMD Technical Report 148 13, Bonn, Germany.Google Scholar
- Herbert Jaeger. 2001b. Short Term Memory in Echo State Networks, GMD-Forschungszentrum Informationstechnik.Google Scholar
- Herbert Jaeger. 2007. Echo state network. Scholarpedia 2, 2330.Google ScholarCross Ref
- Herbert Jaeger and Harald Haas. 2004. Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication'. Science 304, 78--80.Google ScholarCross Ref
- Yu Jin, Qingchun Zhao, Hongxi Yin, and Hehe Yue. 2015. Handwritten numeral recognition utilizing reservoir computing subject to optoelectronic feedback. In Proceedings of the 11th International Conference on Natural Computation (ICNC’15). IEEE, 1165--1169.Google Scholar
- Leandro Junges and Jason A. C. Gallas. 2012. Intricate routes to chaos in the Mackey--glass delayed feedback system. Phys. Lett. A, 376, 2109--2116.Google ScholarCross Ref
- Eric R. Kandel, James H. Schwartz, Thomas M. Jessell, Steven A. Siegelbaum, and A. James Hudspeth. 2000. Principles of Neural Science, McGraw-Hill, New York.Google Scholar
- Laszlo B. Kish. 2002. End of Moore's law: Thermal (noise) death of integration in micro and nano electronics. Phys. Lett. A 305, 144--149.Google ScholarCross Ref
- Christof Koch and Idan Segev. 2000. The role of single neurons in information processing. Nature Neurosci. 3, 1171--1177.Google ScholarCross Ref
- Shumin Kong and Masahiro Takatsuka. 2017. Hexpo: A vanishing-proof activation function. In Proceedings of the International Joint Conference on Neural Networks (IJCNN’17). IEEE, 2562--2567.Google ScholarCross Ref
- Laurent Larger and John M. Dudley. 2010. Nonlinear dynamics: Optoelectronic chaos. Nature 465, 41--42.Google ScholarCross Ref
- Laurent Larger, Miguel C. Soriano, Daniel Brunner, Lennert Appeltant, Jose M. Gutiérrez, Luis Pesquera, Claudio R. Mirasso, and Ingo Fischer. 2012. Photonic information processing beyond Turing: An optoelectronic implementation of reservoir computing. Optics Express 20, 3241--3249.Google ScholarCross Ref
- Robert Legenstein and Wolfgang Maass. 2007a. Edge of chaos and prediction of computational performance for neural circuit models. Neural Netw. 20, 323--334. Google ScholarDigital Library
- Robert Legenstein and Wolfgang Maass. 2007b. What makes a dynamical system computationally powerful. New Directions in Statistical Signal Processing: From Systems to Brain. 127--154.Google Scholar
- Xiaowei Lin, Zehong Yang, and Yixu Song. 2011. Intelligent stock trading system based on improved technical analysis and echo state network. Expert Syst. Appl. 38, 11347--11354. Google ScholarDigital Library
- Mantas Lukoševicius. 2012. Reservoir computing and self-organized neural hierarchies. Jacobs University, Bremen.Google Scholar
- Wolfgang Maass, Thomas Natschläger, and Henry Markram. 2002. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Comput. 14 2531--60. Google ScholarDigital Library
- M. Mayberry. 2017. Intel's new self-learning chip promises to accelerate artificial intelligence. Intel.Google Scholar
- Carver Mead. 1990. Neuromorphic electronic systems. Proc. IEEE 78, 1629--1636.Google ScholarCross Ref
- Paul A. Merolla, John V. Arthur, Rodrigo Alvarez-Icaza, Andrew S. Cassidy, Jun Sawada, Filipp Akopyan, Bryan L. Jackson, Nabil Imam, Chen Guo, and Yutaka Nakamura. 2014. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668--673.Google ScholarCross Ref
- John G. Milton. 2015. Time delays and the control of biological systems. IFAC PapersOnLine 48, 87--92.Google ScholarCross Ref
- Dharmendra S. Modha. 2017. Introducing a brain-inspired computer. Retrieved from http://www.research.Ibm.com/articles/brain-chip.shtml.Google Scholar
- D. J. Myers and R. A. Hutchinson. 1989. Efficient implementation of piecewise linear activation function for digital VLSI neural networks. Electron. Lett. 25, 1662.Google ScholarCross Ref
- Nasser M. Nasrabadi. 2007. Pattern recognition and machine learning. J. Electron. Imag. 16, 049901.Google ScholarCross Ref
- Silvia Ortín and Luis Pesquera. 2017. Reservoir computing with an ensemble of time-delay reservoirs. Cogn. Comput. 9, 3 (2017), 327--336.Google ScholarCross Ref
- G. Overton. 2014. Photonic reservoir computing--A new tool for speech recognition. Retrieval from https://www.laserfocusworld.com/articles/2014/09/photonic-reservoir-computing-a-new-tool-for-speech-recognition.html.Google Scholar
- Yvan Paquot, Francois Duport, Antoneo Smerieri, Joni Dambre, Benjamin Schrauwen, Marc Haelterman, and Serge Massar. 2012. Optoelectronic reservoir computing. Sci. Rep. 2, 287.Google Scholar
- Ali Rodan and Peter Tino. 2011. Minimum complexity echo state network. IEEE Trans. Neural Netw. 22, 131--144. Google ScholarDigital Library
- Chih-Tang Sah. 1991. Fundamentals of Solid State Electronics, World Scientific Publishing Company, Singapore.Google Scholar
- Benjamin Schrauwen, Michiel D'Haene, David Verstraeten, and Jan Van Campenhout. 2008. Compact hardware liquid state machines on FPGA for real-time speech recognition. Neural Netw. 21, 511--523. Google ScholarDigital Library
- Benjamin Schrauwen and Robert Legenstein. 2010. Connectivity, dynamics, and memory in reservoir computing with binary and analog neurons. Neural computation 22, 5 (2010), 1272--1311. Google ScholarDigital Library
- Benjamin Schrauwen and Dirk Stroobandt. 2008. Using reservoir computing in a decomposition approach for time series prediction. In Proceedings of the European Symposium on Time Series Prediction (ESTSP’08). Multiprint Oy/Otamedia, 149--158.Google Scholar
- Benjamin Schrauwen, David Verstraeten, and Jan Van Campenhout. 2007. An overview of reservoir computing: Theory, applications and implementations. In Proceedings of the 15th European Symposium on Artificial Neural Networks. 471--482.Google Scholar
- Mark D. Skowronski and John G. Harris. 2007. Automatic speech recognition using a predictive echo state network classifier. Neural Netw. 20 414--23. Google ScholarDigital Library
- Miguel C. Soriano, Silvia Ortín, Lars Keuninckx, Lennert Appeltant, Jan Danckaert, Luis Pesquera, and Guy Van derSande. 2015. Delay-based reservoir computing: Noise effects in a combined analog and digital implementation. IEEE Trans. Neural Netw.d Learn. Syst. 26, 388--393.Google ScholarCross Ref
- Kristof Vandoorne, Wouter Dierckx, Benjamin Schrauwen, David Verstraeten, Roel Baets, Peter Bienstman, and Jan Van Campenhout. 2008. Toward optical signal processing using photonic reservoir computing. Optics Express 16, 11182--11192.Google ScholarCross Ref
- David Verstraeten, Benjamin Schrauwen, Michiel d'Haene, and Dirk Stroobandt. 2007. An experimental unification of reservoir computing methods. Neural Netw. 20, 391--403. Google ScholarDigital Library
- David Verstraeten, Benjamin Schrauwen, and Dirk Stroobandt. 2005. Reservoir computing with stochastic bitstream neurons. In Proceedings of the 16th Annual Prorisc Workshop. 454--459.Google Scholar
- Qian Wang, Youjie Li, and Peng Li. 2016. Liquid state machine based pattern recognition on FPGA with firing-activity dependent power gating and approximate computing. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS’16). IEEE, 361--364.Google ScholarDigital Library
- Jianhong Wu. 1998. Symmetric functional differential equations and neural networks with memory. Trans. Amer. Math. Soc. 350, 4799--4838.Google ScholarCross Ref
- Daoyi Xu and Zhichun Yang. 2005. Impulsive delay differential inequality and stability of neural networks. J. Math. Anal. Appl. 305, 107--120.Google ScholarCross Ref
- Shimeng Yu, Yi Wu, Rakesh Jeyasingh, Duygu Kuzum, and H.-S. Philip Wong. 2011. An electronic synapse device based on metal oxide resistive switching memory for neuromorphic computation. IEEE Trans. Electron. Dev. 58, 2729--2737.Google ScholarCross Ref
- Zhang Yong, Peng Li, Yingyezhe Jin, and Yoonsuck Choe. 2015. A digital liquid state machine with biologically inspired learning and its application to speech recognition. IEEE Trans. Netw. Learn. Syst. 26, 2635--2649.Google ScholarCross Ref
- Chenyuan Zhao, Jialing Li, Lingjia Liu, Lakshmi Sravanthi Koutha, Jian Liu, and Yang Yi. 2016. Novel spike based reservoir node design with high performance spike delay loop. In Proceedings of the 3rd ACM International Conference on Nanoscale Computing and Communication. ACM, 14. Google ScholarDigital Library
- Chenyuan Zhao, Yang Yi, Jialing Li, Xin Fu, and Lingjia Liu. 2017. Interspike-interval-based analog spike-time-dependent encoder for neuromorphic processors. IEEE Trans. Very Large Scale Integr. Syst. 25, 2193--2205.Google ScholarDigital Library
Index Terms
- DFR: An Energy-efficient Analog Delay Feedback Reservoir Computing System for Brain-inspired Computing
Index terms have been assigned to the content through auto-classification.
Recommendations
A Novel Single-Resistance-Controlled Sinusoidal Oscillator Employing Single Operational Transresistance Amplifier
A novel operational transresistance amplifier (OTRA)-based single-resistance-controlled sinusoidal oscillator (SRCO) is proposed. It uses single OTRA, three resistors and two capacitors. To the best knowledge of author, this is the first oscillator ...
Recurrent spiking neural network with dynamic presynaptic currents based on backpropagation
AbstractIn recent years, spiking neural networks (SNNs), which originated from the theoretical basis of neuroscience, have attracted neuromorphic computing and brain‐like computing due to their advantages, such as neural dynamics and coding mechanism, ...
Research on General-Purpose Brain-Inspired Computing Systems
AbstractBrain-inspired computing is a new technology that draws on the principles of brain science and is oriented to the efficient development of artificial general intelligence (AGI), and a brain-inspired computing system is a hierarchical system ...
Comments