Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations

ABSTRACT

Recognizing spontaneous micro-expression in video sequences is a challenging problem. In this paper, we propose a new method of small scale spatio-temporal feature learning. The proposed learning method consists of two parts. First, the spatial features of micro-expressions at different expression-states (i.e., onset, onset to apex transition, apex, apex to offset transition and offset) are encoded using convolutional neural networks (CNN). The expression-states are taken into account in the objective functions, to improve the expression class separability of the learned feature representation. Next, the learned spatial features with expression-state constraints are transferred to learn temporal features of micro-expression. The temporal feature learning encodes the temporal characteristics of the different states of the micro-expression using long short-term memory (LSTM) recurrent neural networks. Extensive and comprehensive experiments have been conducted on the publically available CASME II micro-expression dataset. The experimental results showed that the proposed method outperformed state-of-the-art micro-expression recognition methods in terms of recognition accuracy.

References

1. Bartlett, M.S., Movellan, J.R., and Sejnowski, T.J., 2002. Face recognition by independent component analysis. Neural Networks, IEEE Transactions on 13, 6, 1450--1464.
2. Chherawala, Y., Roy, P.P., and Cheriet, M., 2013. Feature design for offline Arabic handwriting recognition: handcrafted vs automated? In Document Analysis and Recognition (ICDAR), 2013 12th International Conference on IEEE, 290--294.
3. Chollet, F., 2015. Keras. Available: https://github.com/fchollet/keras
4. Demyanov, S., ConvNet. Available: https://github.com/sdemyanov/ConvNet
5. Gers, F.A., Schraudolph, N.N., and Schmidhuber, J., 2003. Learning precise timing with LSTM recurrent networks. The Journal of Machine Learning Research 3, 115--143.
6. Huang, X., Wang, S.J., Zhao, G., Piteik, M., and Inen, 2015. Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection. In 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), 1--9.
7. Huang, X., Zhao, G., Hong, X., Zheng, W., and Pietikäinen, M., 2016. Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing 175, Part A(1/29/), 564--578.
8. Khorrami, P., Paine, T., and Huang, T., 2015. Do Deep Neural Networks Learn Facial Action Units When Doing Expression Recognition? In Proceedings of the IEEE International Conference on Computer Vision Workshops, 19--27.
9. Le Ngo, A.C., Liong, S.-T., See, J., and Phan, R.C.-W., 2015. Are subtle expressions too sparse to recognize? In Digital Signal Processing (DSP), 2015 IEEE International Conference on IEEE, 1246--1250.
10. Mignon, A. and Jurie, F., 2012. Pcca: A new approach for distance learning from sparse pairwise constraints. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on IEEE, 2666--2672.
11. Nair, V. and Hinton, G.E., 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), 807--814.
12. Oh, Y.-H., Le Ngo, A.C., See, J., Liong, S.-T., Phan, R.C.-W., and Ling, H.-C., 2015. Monogenic Riesz wavelet representation for micro-expression recognition. In Digital Signal Processing (DSP), 2015 IEEE International Conference on IEEE, 1237--1241.
13. Park, S.Y., Lee, S.H., and Ro, Y.M., 2015. Subtle Facial Expression Recognition Using Adaptive Magnification of Discriminative Facial Motion. In Proceedings of the 23rd Annual ACM Conference on Multimedia Conference ACM, 911--914.
14. Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., and Fergus, R., 2013. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning (ICML-13), 1058--1066.
15. Wang, Y., See, J., Phan, R.C.-W., and Oh, Y.-H., 2014. LBP with Six Intersection Points: Reducing Redundant Information in LBP-TOP for Micro-expression Recognition. In Computer Vision-ACCV 2014 Springer, 525--537.
16. Wang, Y., See, J., Phan, R.C.-W., and Oh, Y.-H., 2015. Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition. PloS One 10, 5, e0124674.
17. Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., and Fu, X., 2014. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PloS One 9, 1, e86041.
18. Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., and Fu, X., 2013. How fast are the leaked facial expressions: The duration of micro-expressions. Journal of Nonverbal Behavior 37, 4, 217--230.
19. Zhao, G. and Pietikainen, M., 2007. Dynamic texture recognition using local binary patterns with an application to facial expressions. Pattern Analysis and Machine Intelligence, IEEE Transactions on 29, 6, 915--928.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

  • Machine learning

    • Machine learning approaches

      • Learning latent representations

      • Neural networks