ABSTRACT
Materials discovery is crucial for making scientific advances in many domains. Collections of data from experiments and first-principle computations have spurred interest in applying machine learning methods to create predictive models capable of mapping from composition and crystal structures to materials properties. Generally, these are regression problems with the input being a 1D vector composed of numerical attributes representing the material composition and/or crystal structure. While neural networks consisting of fully connected layers have been applied to such problems, their performance often suffers from the vanishing gradient problem when network depth is increased. Hence, predictive modeling for such tasks has been mainly limited to traditional machine learning techniques such as Random Forest. In this paper, we study and propose design principles for building deep regression networks composed of fully connected layers with numerical vectors as input. We introduce a novel deep regression network with individual residual learning, IRNet, that places shortcut connections after each layer so that each layer learns the residual mapping between its output and input. We use the problem of learning properties of inorganic materials from numerical attributes derived from material composition and/or crystal structure to compare IRNet's performance against that of other machine learning techniques. Using multiple datasets from the Open Quantum Materials Database (OQMD) and Materials Project for training and evaluation, we show that IRNet provides significantly better prediction performance than the state-of-the-art machine learning approaches currently used by domain scientists. We also show that IRNet's use of individual residual learning leads to better convergence during the training phase than when shortcut connections are between multi-layer stacks while maintaining the same number of parameters.
- 2016. Materials Genome Initiative. https://www.whitehouse.gov/mgiGoogle Scholar
- Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).Google Scholar
- Ankit Agrawal and Alok Choudhary. 2016. Perspective: Materials informatics and big data: Realization of the "fourth paradigm" of science in materials science. APL Materials 4, 5 (2016), 053208.Google ScholarCross Ref
- Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks 5, 2 (1994), 157--166. Google ScholarDigital Library
- G. Bergerhoff, R. Hundt, R. Sievers, and I. D. Brown. 1983. The inorganic crystal structure data base. Journal of Chemical Information and Computer Sciences 23, 2 (1983), 66--69. arXiv:http://dx.doi.org/10.1021/ci00038a003Google ScholarCross Ref
- B Blaiszik, K Chard, J Pruyne, R Ananthakrishnan, S Tuecke, and I Foster. 2016. The Materials Data Facility: Data services to advance materials science research. JOM 68, 8 (2016), 2045--2052.Google ScholarCross Ref
- Wouter Boomsma and Jes Frellsen. 2017. Spherical convolutions and their application in molecular modelling. In Advances in Neural Information Processing Systems. 3433--3443. Google ScholarDigital Library
- Venkatesh Botu and Rampi Ramprasad. 2015. Adaptive machine learning framework to accelerate ab initio molecular dynamics. International Journal of Quantum Chemistry 115, 16 (2015), 1074--1083.Google ScholarCross Ref
- Ryan Chard, Zhuozhao Li, Kyle Chard, Logan T. Ward, Yadu N. Babuji, Anna Woodard, Steven Tuecke, Ben Blaiszik, Michael J. Franklin, and Ian T. Foster. 2019. DLHub: Model and data serving for science. In 33rd IEEE International Parallel and Distributed Processing Symposium.Google Scholar
- Stefano Curtarolo, Gus LW Hart, Marco Buongiorno Nardelli, Natalio Mingo, Stefano Sanvito, and Ohad Levy. 2013. The high-throughput highway to computational materials design. Nature materials 12, 3 (2013), 191.Google Scholar
- Alden Dima, Sunil Bhaskarla, Chandler Becker, Mary Brady, Carelyn Campbell, Philippe Dessauw, Robert Hanisch, Ursula Kattner, Kenneth Kroenlein, Marcus Newrock, et al. 2016. Informatics infrastructure for the Materials Genome Initiative. JOM 68, 8 (2016), 2053--2064.Google ScholarCross Ref
- Felix Faber, Alexander Lindmaa, O Anatole von Lilienfeld, and Rickard Armiento. 2015. Crystal structure representations for machine learning models of formation energies. International Journal of Quantum Chemistry 115, 16 (2015), 1094--1101.Google ScholarCross Ref
- Felix A Faber, Alexander Lindmaa, O Anatole Von Lilienfeld, and Rickard Armiento. 2016. Machine Learning Energies of 2 Million Elpasolite (A B C 2 D 6) Crystals. Physical review letters 117, 13 (2016), 135502.Google Scholar
- Luca M Ghiringhelli, Jan Vybiral, Sergey V Levchenko, Claudia Draxl, and Matthias Scheffler. 2015. Big data of materials science: Critical role of the descriptor. Physical review letters 114, 10 (2015), 105503.Google Scholar
- Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249--256.Google Scholar
- Garrett B Goh, Nathan O Hodas, Charles Siegel, and Abhinav Vishnu. 2017. SMILES2Vec: An Interpretable General-Purpose Deep Neural Network for Predicting Chemical Properties. arXiv preprint arXiv:1712.02034 (2017).Google Scholar
- Katja Hansen, Franziska Biegler, Raghunathan Ramakrishnan, Wiktor Pronobis, O. Anatole Von Lilienfeld, Klaus-Robert Robert Müller, and Alexandre Tkatchenko. 2015. Machine Learning Predictions of Molecular Properties: Accurate Many-Body Potentials and Nonlocality in Chemical Space. The Journal of Physical Chemistry Letters 6, 12 (jun 2015), 2326--2331. arXiv:1109.2618Google ScholarCross Ref
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.Google ScholarCross Ref
- Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700--4708.Google ScholarCross Ref
- Lu Huang, Ji Xu, Jiasong Sun, and Yi Yang. 2017. An improved residual LSTM architecture for acoustic modeling. In Computer and Communication Systems (ICCCS), 2017 2nd International Conference on. IEEE, 101--105.Google ScholarCross Ref
- Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).Google ScholarDigital Library
- Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, and Kristin a. Persson. 2013. The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials 1, 1 (2013), 011002.Google ScholarCross Ref
- Dipendra Jha, Saransh Singh, Reda Al-Bahrani, Wei-keng Liao, Alok Choudhary, Marc De Graef, and Ankit Agrawal. 2018. Extracting grain orientations from ebsd patterns of polycrystalline materials using convolutional neural networks. Microscopy and Microanalysis 24, 5 (2018), 497--502.Google ScholarCross Ref
- Dipendra Jha, LoganWard, Arindam Paul,Wei-keng Liao, Alok Choudhary, Chris Wolverton, and Ankit Agrawal. 2018. ElemNet: Deep Learning the Chemistry of Materials From Only Elemental Composition. Scientific reports 8, 1 (2018), 17593.Google Scholar
- Surya R Kalidindi. 2015. Data science and cyberinfrastructure: critical enablers for accelerated development of hierarchical materials. International Materials Reviews 60, 3 (2015), 150--168.Google ScholarCross Ref
- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Scott Kirklin, James E Saal, Bryce Meredig, Alex Thompson, Jeff W Doak, Muratahan Aykol, Stephan Rühl, and Chris Wolverton. 2015. The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies. npj Computational Materials 1 (2015), 15010.Google Scholar
- Ruoqian Liu, Abhishek Kumar, Zhengzhang Chen, Ankit Agrawal, Veera Sundararaghavan, and Alok Choudhary. 2015. A predictive machine learning approach for microstructure optimization and materials design. Scientific reports 5 (2015).Google Scholar
- Bryce Meredig, AnkitAgrawal, Scott Kirklin, James E Saal, JW Doak,AThompson, Kunpeng Zhang, Alok Choudhary, and Christopher Wolverton. 2014. Combinatorial screening for new materials in unconstrained composition space with machine learning. Physical Review B 89, 9 (2014), 094104.Google ScholarCross Ref
- Grégoire Montavon, Matthias Rupp, Vivekanand Gobre, Alvaro Vazquez- Mayagoitia, Katja Hansen, Alexandre Tkatchenko, Klaus-Robert Müller, and O Anatole von Lilienfeld. 2013. Machine learning of molecular electronic properties in chemical compound space. New Journal of Physics 15, 9 (2013), 095003.Google ScholarCross Ref
- Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10). 807--814. Google ScholarDigital Library
- NMNusran, K R Joshi, K Cho,MA Tanatar,WR Meier, S L Bud'ko, P C Canfield, Y Liu, T A Lograsso, and R Prozorov. 2018. Spatially-resolved study of the Meissner effect in superconductors using NV-centers-in-diamond optical magnetometry. New Journal of Physics 20, 4 (2018), 043010. http://stacks.iop.org/1367--2630/20/ i=4/a=043010Google ScholarCross Ref
- David W Oxtoby, H Pat Gillis, and Laurie J Butler. 2015. Principles of modern chemistry. Cengage Learning.Google Scholar
- Arindam Paul, Dipendra Jha, Reda Al-Bahrani, Wei-keng Liao, Alok Choudhary, and Ankit Agrawal. 2018. CheMixNet: Mixed DNN Architectures for Predicting Chemical Properties using Multiple Molecular Representations. In Proceedings of the Workshop on Molecules and Materials at the 32nd Conference on Neural Information Processing Systems.Google Scholar
- F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830. Google ScholarDigital Library
- Edward O Pyzer-Knapp, Kewei Li, and Alan Aspuru-Guzik. 2015. Learning from the harvard clean energy project: The use of neural networks to accelerate materials discovery. Advanced Functional Materials 25, 41 (2015), 6495--6502.Google ScholarCross Ref
- Zhao Qin, Gang Seob Jung, Min Jeong Kang, and Markus J Buehler. 2017. The mechanics and design of a lightweight three-dimensional graphene assembly. Science advances 3, 1 (2017), e1601536.Google Scholar
- Krishna Rajan. 2015. Materials informatics: The materials "gene" and big data. Annual Review of Materials Research 45 (2015), 153--169.Google ScholarCross Ref
- Rampi Ramprasad, Rohit Batra, Ghanshyam Pilania, Arun Mannodi- Kanakkithodi, and Chiho Kim. 2017. Machine learning in materials informatics: recent applications and prospects. npj Computational Materials 3, 1 (dec 2017), 54.Google Scholar
- KT Schütt, H Glawe, F Brockherde, A Sanna, KR Müller, and EKU Gross. 2014. How to represent crystal structures for machine learning: Towards fast prediction of electronic properties. Physical Review B 89, 20 (2014), 205118.Google ScholarCross Ref
- Kristof T. Schütt, Huziel E. Sauceda, Pieter-Jan Kindermans, Alexandre Tkatchenko, and Klaus-Robert Müller. 2017. SchNet - a deep learning architecture for molecules and materials. (2017), 1--10. arXiv:1712.06113 http: //arxiv.org/abs/1712.06113Google Scholar
- Atsuto Seko, Hiroyuki Hayashi, Keita Nakayama, Akira Takahashi, and Isao Tanaka. 2017. Representation of compounds for machine-learning prediction of physical properties. Physical Review B 95, 14 (2017), 144110.Google ScholarCross Ref
- Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Advances in neural information processing systems. 2377--2385. Google ScholarDigital Library
- Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning.. In AAAI, Vol. 4. 12. Google ScholarDigital Library
- Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--9.Google ScholarCross Ref
- Yiren Wang and Fei Tian. 2016. Recurrent residual learning for sequence classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 938--943.Google ScholarCross Ref
- Logan Ward, Ankit Agrawal, Alok Choudhary, and Christopher Wolverton. 2016. A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials. npj Computational Materials 2, August (2016), 16028. arXiv:1606.09551Google Scholar
- Logan Ward, Ruoqian Liu, Amar Krishna, Vinay I Hegde, Ankit Agrawal, Alok Choudhary, and Chris Wolverton. 2017. Including crystal structure attributes in machine learning models of formation energies via Voronoi tessellations. Physical Review B 96, 2 (2017), 024104.Google ScholarCross Ref
- Logan Ward and Chris Wolverton. 2016. Atomistic calculations and materials informatics: A review. Current Opinion in Solid State and Materials Science (2016).Google Scholar
- Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated residual transformations for deep neural networks. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on. IEEE, 5987--5995.Google ScholarCross Ref
- Dezhen Xue, Prasanna V Balachandran, John Hogden, James Theiler, Deqing Xue, and Turab Lookman. 2016. Accelerated search for materials with targeted properties by adaptive design. Nature communications 7 (2016).Google Scholar
- Quan Zhou, Peizhe Tang, Shenxiu Liu, Jinbo Pan, Qimin Yan, and Shou-Cheng Zhang. 2018. Learning atoms for materials discovery. Proceedings of the National Academy of Sciences 115, 28 (2018), E6411--E6417.Google ScholarCross Ref
Index Terms
IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery
Recommendations
Physics-based Data-Augmented Deep Learning for Enhanced Autogenous Shrinkage Prediction on Experimental Dataset
IC3-2023: Proceedings of the 2023 Fifteenth International Conference on Contemporary ComputingPrediction of the autogenous shrinkage referred to as the reduction of apparent volume of concrete under seal and isothermal conditions is of great significance in the service life analysis and design of durable concrete structures, especially with the ...
The nature of unsupervised learning in deep neural networks: A new understanding and novel approach
Over the last decade, the deep neural networks are a hot topic in machine learning. It is breakthrough technology in processing images, video, speech, text and audio. Deep neural network permits us to overcome some limitations of a shallow neural ...
Deep Learning: Methods and Applications
This monograph provides an overview of general deep learning methodology and its applications to a variety of signal and information processing tasks. The application areas are chosen with the following three criteria in mind: (1) expertise or knowledge ...
Comments