3D Mesh Labeling via Deep Convolutional Neural Networks

Abstract

This article presents a novel approach for 3D mesh labeling by using deep Convolutional Neural Networks (CNNs). Many previous methods on 3D mesh labeling achieve impressive performances by using predefined geometric features. However, the generalization abilities of such low-level features, which are heuristically designed to process specific meshes, are often insufficient to handle all types of meshes. To address this problem, we propose to learn a robust mesh representation that can adapt to various 3D meshes by using CNNs. In our approach, CNNs are first trained in a supervised manner by using a large pool of classical geometric features. In the training process, these low-level features are nonlinearly combined and hierarchically compressed to generate a compact and effective representation for each triangle on the mesh. Based on the trained CNNs and the mesh representations, a label vector is initialized for each triangle to indicate its probabilities of belonging to various object parts. Eventually, a graph-based mesh-labeling algorithm is adopted to optimize the labels of triangles by considering the label consistencies. Experimental results on several public benchmarks show that the proposed approach is robust for various 3D meshes, and outperforms state-of-the-art approaches as well as classic learning algorithms in recognizing mesh labels.

References

1. O. K.-C. Au, Y. Zheng, M. Chen, P. Xu, and C.-L. Tai. 2012. Mesh segmentation with concavity-aware fields. IEEE TVCG 18, 7, 1125--1134.
2. J. M. Baker, L. Deng, J. Glass, S. Khudanpur, C.-H. Lee, N. Morgan, and D. O. Shaughnessy. 2009. Developments and directions in speech recognition and understanding, part 1. IEEE Signal Processing Magazine 26, 3, 75--80.
3. S. Belongie, J. Malik, and J. Puzicha. 2002. Shape matching and object recognition using shape contexts. IEEE TPAMI 24, 4, 509--522.
4. M. Ben-Chen and C. Gotsman. 2008. Characterizing shape using conformal factors. In Proc. Eurographics 3DOR. 1--8.
5. Y. Bengio. 2009. Learning deep architectures for AI. Foundations and Trends® in Machine Learning 2, 1, 1--127.
6. Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE TPAMI 23, 11, 1222--1239.
7. J. Bruna, W. Zaremba, A. Szlam, and Y. Lecun. 2014. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203.
8. C.-C. Chang and C.-J. Lin. 2011. LIBSVM: A library for support vector machines. ACM TIST 2, 27:1--27:27. Software available at www.csie.ntu.edu.tw∼cjlin/libsvm.
9. X. Chen, A. Golovinskiy, and T. Funkhouser. 2009. A benchmark for 3D mesh segmentation. ACM Trans. Graph. 28, 3, 73:1--73:12.
10. X. Chen, Y. Guo, B. Zhou, and Q. Zhao. 2013. Deformable model for estimating clothed and naked human shapes from a single image. The Visual Computer 29, 11, 1187--1196.
11. X. Chen, J. Li, Q. Li, B. Gao, D. Zou, and Q. Zhao. 2015a. Image2scene: Transforming style of 3D room. In Proceedings of ACM MM. 321--330.
12. X. Chen, B. Zhou, F. Lu, L. Wang, L. Bi, and P. Tan. 2015b. Garment modeling with a depth camera. ACM Trans. Graph. 34, 6.
13. L. Deng. 2004. Switching dynamic system models for speech articulation and acoustics. In Proceedings of the IMA Workshop. Springer, 115--134.
14. C. Farabet, C. Couprie, L. Najman, and Y. Lecun. 2013. Learning hierarchical features for scene labeling. IEEE TPAMI 35, 8, 1915--1929.
15. R. Gal and D. Cohen-Or. 2006. Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. 25, 1, 130--150.
16. M. Hilaga, Y. Shinagawa, T. Kohmura, and T. L. Kunii. 2001. Topology matching for fully automatic similarity estimation of 3D shapes. In Proc. SIGGRAPH. 203--212.
17. G. Hinton. 2010. A practical guide to training restricted Boltzmann machines. Momentum 9, 1, 926.
18. R. Hu, L. Fan, and L. Liu. 2012. Co-segmentation of 3D shapes via subspace clustering. CGF 31, 5, 1703--1713.
19. Q. Huang, V. Koltun, and L. Guibas. 2011. Joint shape segmentation with linear programming. ACM Trans. Graph. 30, 6, 125:1--125:12.
20. Q.-X. Huang, H. Su, and L. Guibas. 2013. Fine-grained semisupervised labeling of large shape collections. ACM Trans. Graph. 32, 6, 190:1--190:10.
21. Q.-X. Huang, M. Wicke, B. Adams, and L. Guibas. 2009. Shape decomposition using modal analysis. CGF 28, 2, 407--416.
22. A. E. Johnson and M. Hebert. 1999. Using spin images for efficient object recognition in cluttered 3D scenes. IEEE TPAMI 21, 5, 433--449.
23. E. Kalogerakis, A. Hertzmann, and K. Singh. 2010. Learning 3D mesh segmentation and labeling. ACM Trans. Graph. 29, 4, 102:1--102:12.
24. S. Katz and A. Tal. 2003. Hierarchical mesh decomposition using fuzzy clustering and cuts. ACM Trans. Graph. 22, 3, 954--961.
25. K. Kavukcuoglu, M. Ranzato, R. Fergus, and Y. Lecun. 2009. Learning invariant features through topographic filter maps. In Proc. CVPR. 1605--1612.
26. K. Kavukcuoglu, M. Ranzato, and Y. Lecun. 2010. Fast inference in sparse coding algorithms with applications to object recognition. arXiv preprint arXiv:1010.3467.
27. V. G. Kim, W. Li, N. J. Mitra, S. Chaudhuri, S. Diverdi, and T. Funkhouser. 2013. Learning part-based templates from large collections of 3D shapes. ACM Trans. Graph. 32, 4, 70.
28. A. Krizhevsky, I. Sutskever, and G. E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proc. NIPS. 1106--1114.
29. J. D. Lafferty, A. McCallum, and F. C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. ICML. 282--289.
30. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. 2009. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proc. ICML. 609--616.
31. R. Liu, H. Zhang, A. Shamir, and D. Cohen-Or. 2009. A part-aware surface metric for shape analysis. CGF 28, 2, 397--406.
32. J. Lv, X. Chen, J. Huang, and H. Bao. 2012. Semi-supervised mesh segmentation and labeling. CGF 31, 7, 2241--2248.
33. S. Lyu and E. P. Simoncelli. 2008. Nonlinear image representation using divisive normalization. In Proc. CVPR. 1--8.
34. L. Shapira, S. Shalom, A. Shamir, D. Cohen-Or, and H. Zhang. 2010. Contextual part analogies in 3D objects. IJCV 89, 2--3, 309--326.
35. O. Sidi, O. Van Kaick, Y. Kleiman, H. Zhang, and D. Cohen-Or. 2011. Unsupervised co-segmentation of a set of shapes via descriptor-space spectral clustering. ACM Trans. Graph. 30, 6, 126:1--126:10.
36. R. Socher, B. Huval, B. P. Bath, C. D. Manning, and A. Y. Ng. 2012. Convolutional-recursive deep learning for 3D object classification. In Proc. NIPS. 665--673.
37. A. Torralba, K. P. Murphy, and W. T. Freeman. 2007. Sharing visual features for multiclass and multiview object detection. IEEE TPAMI 29, 5, 854--869.
38. O. van Kaick, A. Tagliasacchi, O. Sidi, H. Zhang, D. Cohen-Or, L. Wolf, and G. Hamarneh. 2011. Prior knowledge for part correspondence. CGF 30, 2, 553--562.
39. O. van Kaick, K. Xu, H. Zhang, Y. Wang, S. Sun, A. Shamir, and D. Cohen-Or. 2013. Co-hierarchical analysis of shape structures. ACM Trans. Graph. 32, 4, 69:1--69:10.
40. Y. Wang, S. Asafi, O. van Kaick, H. Zhang, D. Cohen-Or, and B. Chen. 2012. Active co-analysis of a set of shapes. ACM Trans. Graph. 31, 6, 165:1--165:10.
41. Y. Wang, M. Gong, T. Wang, D. Cohen-Or, H. Zhang, and B. Chen. 2013. Projective analysis for 3D shape segmentation. ACM Trans. Graph. 32, 6, 192:1--192:12.
42. Z. Xie, K. Xu, L. Liu, and Y. Xiong. 2014. 3D shape segmentation and labeling via extreme learning machine. CGF 33, 5, 85--95.
43. Y. Yang, W. Xu, X. Guo, K. Zhou, and B. Guo. 2013. Boundary-aware multidomain subspace deformation. IEEE TVCG 19, 10, 1633.
44. Y. Yu, K. Zhou, D. Xu, X. Shi, H. Bao, B. Guo, and H.-Y. Shum. 2004. Mesh editing with poisson-based gradient field manipulation. ACM Trans. Graph. 23, 3, 644--651.
45. M. D. Zeiler, G. W. Taylor, and R. Fergus. 2011. Adaptive deconvolutional networks for mid and high level feature learning. In Proc. ICCV. 2018--2025.
46. J. Zhang, J. Zheng, C. Wu, and J. Cai. 2012. Variational mesh decomposition. ACM Trans. Graph. 31, 3, 21:1--21:14.

Index Terms

• Computing methodologies

  • Computer graphics

    • Shape modeling

• Theory of computation

  • Randomness, geometry and discrete structures

    • Computational geometry