Large-Scale Learnable Graph Convolutional Networks

ABSTRACT

Convolutional neural networks (CNNs) have achieved great success on grid-like data such as images, but face tremendous challenges in learning from more generic data such as graphs. In CNNs, the trainable local filters enable the automatic extraction of high-level features. The computation with filters requires a fixed number of ordered units in the receptive fields. However, the number of neighboring units is neither fixed nor are they ordered in generic graphs, thereby hindering the applications of convolutional operations. Here, we address these challenges by proposing the learnable graph convolutional layer (LGCL). LGCL automatically selects a fixed number of neighboring nodes for each feature based on value ranking in order to transform graph data into grid-like structures in 1-D format, thereby enabling the use of regular convolutional operations on generic graphs. To enable model training on large-scale graphs, we propose a sub-graph training method to reduce the excessive memory and computational resource requirements suffered by prior methods on graph convolutions. Our experimental results on node classification tasks in both transductive and inductive learning settings demonstrate that our methods can achieve consistently better performance on the Cora, Citeseer, Pubmed citation network, and protein-protein interaction network datasets. Our results also indicate that the proposed methods using sub-graph training strategy are more efficient as compared to prior approaches.

References

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio . 2015. Neural machine translation by jointly learning to align and translate. International Conference on Learning Representations (2015).
2. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille . 2016. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. Transactions on Pattern Analysis and Machine Intelligence (2016).
3. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio . 2014. On the Properties of Neural Machine Translation: Encoder--Decoder Approaches. Syntax, Semantics and Structure in Statistical Translation (2014), 103.
4. Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst . 2016. Convolutional neural networks on graphs with fast localized spectral filtering Advances in Neural Information Processing Systems. 3844--3852.
5. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei . 2009. ImageNet: A Large-Scale Hierarchical Image Database Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
6. Jonas Gehring, Michael Auli, David Grangier, and Yann N Dauphin . 2017. A convolutional encoder model for neural machine translation. Annual Meeting of the Association for Computational Linguistics (2017).
7. Xavier Glorot and Yoshua Bengio . 2010. Understanding the difficulty of training deep feedforward neural networks Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. 249--256.
8. Aditya Grover and Jure Leskovec . 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 855--864.
9. William L. Hamilton, Rex Ying, and Jure Leskovec . 2017. Inductive Representation Learning on Large Graphs. In NIPS.
10. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick . 2017. Mask r-cnn. IEEE International Conference on Computer Vision (2017).
11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun . 2016 a. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun . 2016 b. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
13. Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten . 2017. Densely connected convolutional networks. IEEE Conference on Computer Vision and Pattern Recognition (2017).
14. Diederik Kingma and Jimmy Ba . 2015. Adam: A method for stochastic optimization. The International Conference on Learning Representations (2015).
15. Thomas N Kipf and Max Welling . 2017. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations (2017).
16. Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton . 2012 a. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, bibfieldeditorP. Bartlett, F.C.N. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger (Eds.). 1106--1114.
17. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton . 2012 b. Imagenet classification with deep convolutional neural networks Advances in neural information processing systems. 1097--1105.
18. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner . 1998. Gradient-based learning applied to document recognition. Proc. IEEE Vol. 86, 11 (1998), 2278--2324.
19. Minh-Thang Luong, Hieu Pham, and Christopher D Manning . 2015. Effective approaches to attention-based neural machine translation. Conference on Empirical Methods in Natural Language Processing (2015).
20. Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov . 2016. Learning convolutional neural networks for graphs. In International Conference on Machine Learning. 2014--2023.
21. Bryan Perozzi, Rami Al-Rfou, and Steven Skiena . 2014. Deepwalk: Online learning of social representations Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 701--710.
22. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun . 2015. Faster R-CNN: Towards real-time object detection with region proposal networks Advances in neural information processing systems. 91--99.
23. Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad . 2008. Collective classification in network data. AI magazine Vol. 29, 3 (2008), 93.
24. Karen Simonyan and Andrew Zisserman . 2015. Very deep convolutional networks for large-scale image recognition Proceedings of the International Conference on Learning Representations.
25. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov . 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research Vol. 15, 1 (2014), 1929--1958.
26. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich . 2015. Going Deeper With Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--9.
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin . 2017. Attention is all you need. In Advances in Neural Information Processing Systems. 6000--6010.
28. Petar Velivcković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio . 2017. Graph Attention Networks. arXiv preprint arXiv:1710.10903 (2017).
29. Tao Wang, David J Wu, Adam Coates, and Andrew Y Ng . 2012. End-to-end text recognition with convolutional neural networks Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 3304--3308.
30. Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov . 2016. Revisiting semi-supervised learning with graph embeddings. International Conference on Machine Learning (2016).
31. Marinka Zitnik and Jure Leskovec . 2017. Predicting multicellular function through multi-layer tissue networks. Bioinformatics Vol. 33, 14 (2017), i190--i198.

Index Terms

• Computing methodologies

  • Artificial intelligence

  • Machine learning

    • Learning paradigms

      • Supervised learning

        • Structured outputs

    • Machine learning approaches

      • Neural networks