SAAMEAT: Active Feature Transformation and Selection Methods for the Recognition of User Eating Conditions

ABSTRACT

Automatic recognition of eating conditions of humans could be a useful technology in health monitoring. The audio-visual information can be used in automating this process, and feature engineering approaches can reduce the dimensionality of audio-visual information. The reduced dimensionality of data (particularly feature subset selection) can assist in designing a system for eating conditions recognition with lower power, cost, memory and computation resources than a system which is designed using full dimensions of data. This paper presents Active Feature Transformation (AFT) and Active Feature Selection (AFS) methods, and applies them to all three tasks of the ICMI 2018 EAT Challenge for recognition of user eating conditions using audio and visual features. The AFT method is used for the transformation of the Mel-frequency Cepstral Coefficient and ComParE features for the classification task, while the AFS method helps in selecting a feature subset. Transformation by Principal Component Analysis (PCA) is also used for comparison. We find feature subsets of audio features using the AFS method (422 for Food Type, 104 for Likability and 68 for Difficulty out of 988 features) which provide better results than the full feature set. Our results show that AFS outperforms PCA and AFT in terms of accuracy for the recognition of user eating conditions using audio features. The AFT of visual features (facial landmarks) provides less accurate results than the AFS and AFT sets of audio features. However, the weighted score fusion of all the feature set improves the results.

References

1. Nathalie T. Burkert, Johanna Muckenhuber, Franziska Großschädl, Éva Rásky, and Wolfgang Freidl. 2014. Nutrition and Health textendash The Association between Eating Behavior and Various Health Parameters: A Matched Sample Study. PLoS ONE Vol. 9, 2 (feb. 2014), e88278.
2. Florian Eyben, Felix Weninger, Florian Groß, and Björn Schuller. 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In Proceedings of the 21st ACM international conference on Multimedia. ACM, 835--838.
3. Fasih Haider, Fahim Salim, Owen Conlan, and Saturnino Luz. 2017. An Active Feature Transformation Method For Attitude Recognition of Video Bloggers Proc. Interspeech 2017.
4. Simone Hantke, Maximilian Schmitt, Panagiotis Tzirakis, and Björn Schuller. 2018. EAT - The ICMI 2018 Eating Analysis and Tracking Challenge Proceedings of the 2018 ACM on International Conference on Multimodal Interaction. ACM.
5. Simone Hantke, Felix Weninger, Richard Kurle, Fabien Ringeval, Anton Batliner, Amr El-Desoky Mousa, and Björn Schuller. 2016. I Hear You Eat and Speak: Automatic Recognition of Eating Condition and Food Type, Use-Cases, and Impact on ASR Performance. PLOS ONE Vol. 11, 5 (may. 2016), e0154486.
6. Guang-Bin Huang, Hongming Zhou, Xiaojian Ding, and Rui Zhang. 2012. Extreme learning machine for regression and multiclass classification. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) Vol. 42, 2 (2012), 513--529.
7. Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Siew. 2006. Extreme learning machine: theory and applications. Neurocomputing Vol. 70, 1--3 (2006), 489--501.
8. Heysem Kaya, Alexey A. Karpov, and Albert Ali Salah. 2015. Fisher vectors with cascaded normalization for paralinguistic analysis Sixteenth Annual Conference of the International Speech Communication Association.
9. Teuvo Kohonen. 1998. The self-organizing map. Neurocomputing Vol. 21, 1-3 (1998), 1--6.
10. Mengyi Liu, Ruiping Wang, Shaoxin Li, Shiguang Shan, Zhiwu Huang, and Xilin Chen. 2014. Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In Proceedings of the 16th International Conference on Multimodal Interaction. ACM, 494--501.
11. Sarunas Raudys and Robert P. W. Duin. 1998. Expected classification error of the Fisher linear classifier with pseudo-inverse covariance matrix. Pattern Recognition Letters Vol. 19, 5-6 (April. 1998), 385--392.
12. Björn W. Schuller, Stefan Steidl, Anton Batliner, Simone Hantke, Florian Hönig, Juan Rafael Orozco-Arroyave, Elmar Nöth, Yue Zhang, and Felix Weninger. 2015. The INTERSPEECH 2015 computational paralinguistics challenge: nativeness, parkinson's & eating condition. In INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015. 478--482. http://www.isca-speech.org/archive/interspeech_2015/i15_0478.html
13. Herman Wold. 1985. Partial least squares. Encyclopedia of statistical sciences (1985).

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Speech recognition

  • Machine learning

    • Learning paradigms

      • Supervised learning

    • Machine learning algorithms

      • Feature selection

• Information systems

  • Information systems applications

    • Multimedia information systems

      • Multimedia databases