Voxel Deconvolutional Networks for 3D Brain Image Labeling

ABSTRACT

Deep learning methods have shown great success in pixel-wise prediction tasks. One of the most popular methods employs an encoder-decoder network in which deconvolutional layers are used for up-sampling feature maps. However, a key limitation of the deconvolutional layer is that it suffers from the checkerboard artifact problem, which harms the prediction accuracy. This is caused by the independency among adjacent pixels on the output feature maps. Previous work only solved the checkerboard artifact issue of deconvolutional layers in the 2D space. Since the number of intermediate feature maps needed to generate a deconvolutional layer grows exponentially with dimensionality, it is more challenging to solve this issue in higher dimensions. In this work, we propose the voxel deconvolutional layer (VoxelDCL) to solve the checkerboard artifact problem of deconvolutional layers in 3D space. We also provide an efficient approach to implement VoxelDCL. To demonstrate the effectiveness of VoxelDCL, we build four variations of voxel deconvolutional networks (VoxelDCN) based on the U-Net architecture with VoxelDCL. We apply our networks to address volumetric brain images labeling tasks using the ADNI and LONI LPBA40 datasets. The experimental results show that the proposed iVoxelDCNa achieves improved performance in all experiments. It reaches 83.34% in terms of dice ratio on the ADNI dataset and 79.12% on the LONI LPBA40 dataset, which increases 1.39% and 2.21% respectively compared with the baseline. In addition, all the variations of VoxelDCN we proposed outperform the baseline methods on the above datasets, which demonstrates the effectiveness of our methods.

References

1. Andrew Aitken, Christian Ledig, Lucas Theis, Jose Caballero, Zehan Wang, and Wenzhe Shi . 2017. Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize. arXiv preprint arXiv:1707.02937 (2017).
2. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille . 2016. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint arXiv:1606.00915 (2016).
3. Dan Ciresan, Alessandro Giusti, Luca M Gambardella, and Jürgen Schmidhuber . 2012. Deep neural networks segment neuronal membranes in electron microscopy images Advances in neural information processing systems. 2843--2851.
4. Angela Dai, Daniel Ritchie, Martin Bokeloh, Scott Reed, Jürgen Sturm, and Matthias Nießner . 2017. ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans. arXiv preprint arXiv:1712.10215 (2017).
5. Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox . 2015. Flownet: Learning optical flow with convolutional networks Proceedings of the IEEE International Conference on Computer Vision. 2758--2766.
6. Christine Fennema-Notestine, Donald J Hagler, Linda K McEvoy, Adam S Fleisher, Elaine H Wu, David S Karow, and Anders M Dale . 2009. Structural MRI biomarkers for preclinical and mild Alzheimer's disease. Human brain mapping Vol. 30, 10 (2009), 3238--3253.
7. Hongyang Gao, Hao Yuan, Zhengyang Wang, and Shuiwang Ji . 2017. Pixel Deconvolutional Networks. arXiv preprint arXiv:1705.06820 (2017).
8. Yaozong Gao, Shu Liao, and Dinggang Shen . 2012. Prostate segmentation by sparse representation based classification. Medical physics Vol. 39, 10 (2012), 6372--6387.
9. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio . 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
10. Sergey Ioffe and Christian Szegedy . 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. 448--456.
11. Md Amirul Islam, Neil Bruce, and Yang Wang . 2016. Dense Image Labeling Using Deep Convolutional Neural Networks Computer and Robot Vision (CRV), 2016 13th Conference on. IEEE, 16--23.
12. Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu . 2013. 3D Convolutional Neural Networks for Human Action Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 35, 1 (2013), 221--231.
13. Justin Johnson, Alexandre Alahi, and Li Fei-Fei . 2016. Perceptual losses for real-time style transfer and super-resolution European Conference on Computer Vision. Springer, 694--711.
14. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton . 2012. Imagenet classification with deep convolutional neural networks Advances in neural information processing systems. 1097--1105.
15. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner . 1998. Gradient-based learning applied to document recognition. Proc. IEEE Vol. 86, 11 (1998), 2278--2324.
16. Kisuk Lee, Jonathan Zung, Peter Li, Viren Jain, and H Sebastian Seung . 2017. Superhuman Accuracy on the SNEMI3D Connectomics Challenge. arXiv preprint arXiv:1706.00120 (2017).
17. Rongjian Li, Wenlu Zhang, Heung-Il Suk, Li Wang, Jiang Li, Dinggang Shen, and Shuiwang Ji . 2014. Deep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis Proceedings of the 17th International Conference on Medical Image Computing and Computer Assisted Intervention. 305--312.
18. Guangkai Ma, Yaozong Gao, Guorong Wu, Ligang Wu, and Dinggang Shen . 2016. Nonlocal atlas-guided multi-channel forest learning for human brain labeling. Medical physics Vol. 43, 2 (2016), 1003--1019.
19. Vincent A Magnotta, Dan Heckel, Nancy C Andreasen, Ted Cizadlo, Patricia Westmoreland Corson, James C Ehrhardt, and William TC Yuh . 1999. Measurement of brain structures with artificial neural networks: two-and three-dimensional applications. Radiology Vol. 211, 3 (1999), 781--790.
20. Susanne G Mueller, Michael W Weiner, Leon J Thal, Ronald C Petersen, Clifford Jack, William Jagust, John Q Trojanowski, Arthur W Toga, and Laurel Beckett . 2005. The Alzheimer's disease neuroimaging initiative. Neuroimaging Clinics of North America Vol. 15, 4 (2005), 869--877.
21. Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han . 2015. Learning deconvolution network for semantic segmentation Proceedings of the IEEE International Conference on Computer Vision. 1520--1528.
22. Augustus Odena, Vincent Dumoulin, and Chris Olah . 2016. Deconvolution and Checkerboard Artifacts. Distill (2016).
23. Olaf Ronneberger, Philipp Fischer, and Thomas Brox . 2015. U-net: Convolutional networks for biomedical image segmentation International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 234--241.
24. Gerard Sanroma, Guorong Wu, Yaozong Gao, and Dinggang Shen . 2014. Learning to rank atlases for multiple-atlas segmentation. IEEE transactions on medical imaging Vol. 33, 10 (2014), 1939--1953.
25. David W Shattuck, Mubeena Mirza, Vitria Adisetiyo, Cornelius Hojatkashani, Georges Salamon, Katherine L Narr, Russell A Poldrack, Robert M Bilder, and Arthur W Toga . 2008. Construction of a 3D probabilistic atlas of human cortical structures. Neuroimage Vol. 39, 3 (2008), 1064--1080.
26. Evan Shelhamer, Jonathan Long, and Trevor Darrell . 2017. Fully convolutional networks for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence Vol. 39, 4 (2017), 640--651.
27. Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang . 2016 a. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874--1883.
28. Wenzhe Shi, Jose Caballero, Lucas Theis, Ferenc Huszar, Andrew Aitken, Christian Ledig, and Zehan Wang . 2016 b. Is the deconvolution layer the same as a convolutional layer? arXiv preprint arXiv:1609.07009 (2016).
29. Tong Tong, Robin Wolz, Joseph V Hajnal, and Daniel Rueckert . 2012. Segmentation of brain MR images via sparse patch representation MICCAI Workshop on Sparsity Techniques in Medical Imaging (STMI).
30. Hongzhi Wang, Jung W Suh, Sandhitsu R Das, John B Pluta, Caryne Craige, and Paul A Yushkevich . 2013. Multi-atlas segmentation with joint label fusion. IEEE transactions on pattern analysis and machine intelligence Vol. 35, 3 (2013), 611--623.
31. Li Wang, Yaozong Gao, Feng Shi, Gang Li, John H Gilmore, Weili Lin, and Dinggang Shen . 2015. LINKS: Learning-based multi-source IntegratioN frameworK for Segmentation of infant brain images. NeuroImage Vol. 108 (2015), 160--172.
32. Guorong Wu, Minjeong Kim, Gerard Sanroma, Qian Wang, Brent C Munsell, Dinggang Shen, Alzheimer's Disease Neuroimaging Initiative, et almbox. . 2015. Hierarchical multi-atlas label fusion with multi-scale feature representation and label-specific patch partition. NeuroImage Vol. 106 (2015), 34--46.
33. Guorong Wu, Qian Wang, Daoqiang Zhang, Feiping Nie, Heng Huang, and Dinggang Shen . 2014. A generative probability model of joint label fusion for multi-atlas based brain segmentation. Medical image analysis Vol. 18, 6 (2014), 881--890.
34. Matthew D Zeiler, Graham W Taylor, and Rob Fergus . 2011. Adaptive deconvolutional networks for mid and high level feature learning Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 2018--2025.
35. Tao Zeng, Bian Wu, and Shuiwang Ji . 2017. DeepEM3D: Approaching human-level performance on 3D anisotropic EM image segmentation. Bioinformatics Vol. 33, 16 (2017), 2555--2562.
36. Lichi Zhang, Qian Wang, Yaozong Gao, Guorong Wu, and Dinggang Shen . 2016. Automatic labeling of MR brain images by hierarchical learning of atlas forests. Medical physics Vol. 43, 3 (2016), 1175--1186.
37. Shaoting Zhang, Yiqiang Zhan, Maneesh Dewan, Junzhou Huang, Dimitris N Metaxas, and Xiang Sean Zhou . 2012. Towards robust and effective shape modeling: Sparse shape composition. Medical image analysis Vol. 16, 1 (2012), 265--277.
38. Wenlu Zhang, Rongjian Li, Houtao Deng, Li Wang, Weili Lin, Shuiwang Ji, and Dinggang Shen . 2015. Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage Vol. 108 (2015), 214--224.

Index Terms

• Computer systems organization

  • Architectures

    • Other architectures

      • Neural networks

• Computing methodologies

  • Artificial intelligence

• Theory of computation

  • Theory and algorithms for application domains

    • Machine learning theory

      • Structured prediction