Floating-point Precision and Deformation Awareness for Scalable and Robust 3D Face Alignment

ABSTRACT

This paper improves the accuracy of heatmap-based 3D face alignment neural networks. Many current approaches in face alignment are limited by two major problems, quantization and the lack of regularization of heatmaps. The first limitation is caused by the non-differentiable argmax function, which extracts landmark coordinates from heatmaps as integer indices. Heatmaps are generated at low-resolution to reduce the memory and computational costs, which results in heatmaps far lower than the input image’s resolution. We propose a heatmap generator network producing floating-point precision heatmaps that are scalable to higher-resolutions. To resolve the second limitation, we propose a novel deformation constraint on heatmaps. The constraint is based on graph-Laplacian and enables a heatmap generator to regularize overall shape of the output face landmarks using the global face structure. By eliminating quantization and including regularization, our method can vastly improve landmark localization accuracy, and achieves the state-of-the-art performance without adding complex network structures.

References

1. Adrian Bulat and Georgios Tzimiropoulos. 2017. How Far Are We From Solving the 2D & 3D Face Alignment Problem? (And a Dataset of 230,000 3D Facial Landmarks). In Proc. ICCV.
2. Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. 2018. Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network. In Proc. ECCV.
3. Shiming Ge, Jia Li, Qiting Ye, and Zhao Luo. 2017. Detecting Masked Faces in the Wild with LLE-CNNs. In Proc. CVPR.
4. Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. 2018. 3D-Coded: 3D Correspondences by Deep Deformation. In Proc. ECCV.
5. Lie Gu and Takeo Kanade. 2006. 3D Alignment of Face in a Single Image. In Proc. CVPR.
6. Jia Guo, Jiankang Deng, Niannan Xue, and Stefanos Zafeiriou. 2018. Stacked Dense U-Nets with Dual Transformers for Robust Face Alignment. In Proc. BMVC.
7. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proc. CVPR.
8. Patrik Huber, Philipp Kopp, William Christmas, Matthias Rätsch, and Josef Kittler. 2016. Real-Time 3D Face Fitting and Texture Fusion on in-the-Wild Videos. IEEE Signal Processing Letters 24, 4 (2016), 437–441.
9. Vahid Kazemi and Josephine Sullivan. 2014. One Millisecond Face Alignment with an Ensemble of Regression Trees. In Proc. CVPR.
10. Martin Köstinger, Paul Wohlhart, Peter M. Roth, and Horst Bischof. 2011. Annotated Facial Landmarks in the Wild: A Large-scale, Real-World Database for Facial Landmark Localization. In Proc. First IEEE International Workshop on Benchmarking Facial Image Analysis Technologies.
11. Der-Tsai Lee and Bruce J Schachter. 1980. Two Algorithms for Constructing a Delaunay Triangulation. International Journal of Computer & Information Sciences 9, 3(1980), 219–242.
12. Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler. 2013. Realtime Facial Animation with On-the-fly Correctives. ACM Transactions on Graphics 32, 4, Article 42(2013), 42:1–42:10 pages.
13. Yaojie Liu, Amin Jourabloo, William Ren, and Xiaoming Liu. 2017. Dense Face Alignment. In Proc. ICCV Workshops.
14. Diogo C. Luvizon, David Picard, and Hedi Tabia. 2018. 2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning. In Proc. CVPR.
15. Diogo C. Luvizon, Hedi Tabia, and David Picard. 2019. Human Pose Regression by Combining Indirect Part Detection and Contextual Information. Computers & Graphics(2019).
16. Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016. Stacked Hourglass Networks for Human Pose Estimation. In Proc. ECCV.
17. Aiden Nibali, Zhen He, Stuart Morgan, and Luke Prendergast. 2019. 3D Human Pose Estimation with 2D Marginal Heatmaps. In Proc. WACV.
18. Kyle Olszewski, Joseph J. Lim, Shunsuke Saito, and Hao Li. 2016. High-fidelity Facial and Speech Animation for VR HMDs. ACM Transactions on Graphics 35, 6, Article 221(2016), 221:1–221:14 pages.
19. Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniilidis. 2017. Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose. In Proc. CVPR.
20. Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D Face Model for Pose and Illumination Invariant Face Recognition. In Proc. AVSS.
21. Christos Sagonas, Epameinondas Antonakos, Georgios Tzimiropoulos, Stefanos P. Zafeiriou, and Maja Pantic. 2016. 300 Faces in-the-Wild Challenge: Database and Results. Image and Vision Computing 47 (2016), 3–18.
22. Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 2013a. A Semi-automatic Methodology for Facial Landmark Annotation. In Proc. CVPR, 5th Workshop on Analysis and Modeling of Faces and Gestures (AMFG 2013).
23. Christos Sagonas, Georgios Tzimiropoulos, Stefanos P. Zafeiriou, and Maja Pantic. 2013b. 300 Faces in-the-Wild Challenge: The First Facial Landmark Localization Challenge. In Proc. ICCV, 300 Faces in-the-Wild Challenge (300-W).
24. Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and Hans-Peter Seidel. 2004. Laplacian Surface Editing. In Proc. SGP.
25. Ying Tai, Yicong Liang, Xiaoming Liu, Lei Duan, Jilin Li, Chengjie Wang, Feiyue Huang, and Yu Chen. 2019. Towards Highly Accurate and Stable Face Alignment for High-Resolution Videos. In Proc. AAAI.
26. Zhiqiang Tang, Xi Peng, Shijie Geng, Yizhe Zhu, and Dimitris Metaxas. 2018. CU-Net: Coupled U-Nets. In Proc. BMVC.
27. Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Niessner. 2016. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. In Proc. CVPR.
28. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-RMSProp: Divide the Gradient by a Running Average of its Recent Magnitude. COURSERA: Neural Networks for Machine Learning 4, 2 (2012), 26–31.
29. Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. 2018. Look at Boundary: A Boundary-Aware Face Alignment Algorithm. In Proc. CVPR.
30. Yue Wu and Qiang Ji. 2019. Facial Landmark Detection: A Literature Survey. International Journal of Computer Vision 127, 2 (2019), 115–142.
31. Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In Proc. ICCV.
32. Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, and Stan Z. Li. 2016. Face Alignment Across Large Poses: A 3D Solution. In Proc. CVPR. 146–155.