Johannes Meyer
Skip Abstract Section
Abstract
Smart glasses are considered the next breakthrough in wearables. As the successor of smart watches and smart ear wear, they promise to extend reality by immersive embedding of content in the user's field of view. While advancements in display technology seems to fulfill this promises, interaction concepts are derived from established wearable concepts like touch interaction or voice interaction, preventing full immersion as they require the user to frequently interact with the glasses. To minimize interactions, we propose to add context-awareness to smart glasses through human activity recognition (HAR) by combining head- and eye movement features to recognize a wide range of activities. To measure eye movements in unobtrusive way, we propose laser feedback interferometry (LFI) sensors. These tiny low power sensors are highly robust to ambient light. We combine LFI sensors and an IMU to collect eye and head movement features from 15 participants performing 7 cognitive and physical activities, leading to a unique data set. To recognize activities we propose a 1D-CNN model and apply transfer learning to personalize the classification, leading to an outstanding macro-F1 score of 88.15 % which outperforms state of the art methods. Finally, we discuss the applicability of the proposed system in a smart glasses setup.
- Amazon. 2019. Echo Frames - Eyeglasses with Alexa. online. https://www.amazon.com/Staging-Product-Not-Retail-Sale/dp/B07W72XKPJGoogle Scholar
- Sina Mokhtarzadeh Azar, Mina Ghadimi Atigh, Ahmad Nickabadi, and Alexandre Alahi. 2019. Convolutional Relational Machine for Group Activity Recognition. arXiv:1904.03308 [cs.CV]Google Scholar
- Ling Bao and Stephen S. Intille. 2004. Activity Recognition from User-Annotated Acceleration Data. In Pervasive Computing, Alois Ferscha and Friedemann Mattern (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1--17.Google Scholar
- Behnam Behroozpour, Phillip A. M. Sandborn, Ming C. Wu, and Bernhard E. Boser. 2017. Lidar System Architectures and Circuits. IEEE Communications Magazine 55, 10 (2017), 135--142. https://doi.org/10.1109/MCOM.2017.1700030Google ScholarCross Ref
- C. Braunagel, D. Geisler, W. Rosenstiel, and E. Kasneci. 2017. Online Recognition of Driver-Activity Based on Visual Scanpath Classification. IEEE Intelligent Transportation Systems Magazine 9, 4 (2017), 23--36. https://doi.org/10.1109/MITS.2017.2743171Google ScholarCross Ref
- C. Braunagel, E. Kasneci, W. Stolzmann, and W. Rosenstiel. 2015. Driver-Activity Recognition in the Context of Conditionally Autonomous Driving. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems. 1652--1657. https://doi.org/10.1109/ITSC.2015.268Google ScholarDigital Library
- Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A Tutorial on Human Activity Recognition Using Body-Worn Inertial Sensors. ACM Comput. Surv. 46, 3, Article 33 (Jan. 2014), 33 pages. https://doi.org/10.1145/2499621Google ScholarDigital Library
- Andreas Bulling, Jamie Ward, Hans Gellersen, and Gerhard Tröster. 2011. Eye Movement Analysis for Activity Recognition Using Electrooculography. Pattern Analysis and Machine Intelligence, IEEE Transactions on 33 (05 2011), 741 - 753. https://doi.org/10.1109/TPAMI.2010.86Google ScholarDigital Library
- Andreas Bulling, Jamie A. Ward, and Hans Gellersen. 2012. Multimodal Recognition of Reading Activity in Transit Using Body-Worn Sensors. ACM Trans. Appl. Percept. 9, 1, Article 2 (March 2012), 21 pages. https://doi.org/10.1145/2134203.2134205Google ScholarDigital Library
- Andreas Bulling, Jamie A. Ward, Hans Gellersen, and Gerhard Tröster. 2008. Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography. In Pervasive Computing, Jadwiga Indulska, Donald J. Patterson, Tom Rodden, and Max Ott (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 19--37.Google Scholar
- Andreas Bulling, Christian Weichel, and Hans Gellersen. 2013. EyeContext: Recognition of High-Level Contextual Cues from Human Visual Behaviour. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 305--308. https://doi.org/10.1145/2470654.2470697Google ScholarDigital Library
- J. Chen and X. Ran. 2019. Deep Learning With Edge Computing: A Review. Proc. IEEE 107, 8 (2019), 1655--1674. https://doi.org/10.1109/JPROC.2019.2921977Google ScholarCross Ref
- Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2020. Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities. arXiv:2001.07416 [cs.HC]Google Scholar
- Belkacem Chikhaoui, Frank Gouineau, and Martin Sotir. 2018. A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors. In Machine Learning and Data Mining in Pattern Recognition, Petra Perner (Ed.). Springer International Publishing, Cham, 302--315.Google Scholar
- Epson. 2020. Moverio BT-300 Smart Glasses. https://www.epson.com.hk/For-Home/Wearables/Smart-Glasses/Moverio-BT-300-Smart-Glasses/p/V11H756060Google Scholar
- Tonsen M. Bulling A. et al. Fuhl, W. 2016. Pupil detection for head-mounted eye tracking in the wild: an evaluation of the state of the art. Springer Machine Vision and Applications 27 (2016).Google Scholar
- Angelo Garofalo, Manuele Rusci, Francesco Conti, Davide Rossi, and Luca Benini. 2020. PULP-NN: accelerating quantized neural networks on parallel ultra-low-power RISC-V processors. Philosophical Transactions of the Royal Society A 378, 2164 (2020), 20190155.Google Scholar
- Guido Giuliani, Michele Norgia, Silvano Donati, and Thierry Bosch. 2002. Laser diode self-mixing technique for sensing applications. Journal of Optics A: Pure and Applied Optics 4, 6 (nov 2002), S283-S294. https://doi.org/10.1088/1464-4258/4/6/371Google ScholarCross Ref
- Google LLC. 2021. Glass - Glass. https://www.google.com/glass/start/. Online; accessed 15 April 2021.Google Scholar
- Tomoki Hayashi, Masafumi Nishida, Norihide Kitaoka, Tomoki TODA, and Kazuya Takeda. 2018. Daily Activity Recognition with Large-Scaled Real-Life Recording Datasets Based on Deep Neural Network Using Multi-Modal Signals. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E101.A (01 2018), 199--210. https://doi.org/10.1587/transfun.E101.A.199Google ScholarCross Ref
- Albrecht Schmidt Heiko Drewes. 2007. Interacting with the Computer Using Gaze Gestures. In International Conference on Human Computer Interaction.Google Scholar
- Andrey Ignatov. 2018. Real-time human activity recognition from accelerometer data using Convolutional Neural Networks. Applied Soft Computing 62 (2018), 915--922. https://doi.org/10.1016/j.asoc.2017.09.027Google ScholarCross Ref
- Shoya Ishimaru, Kai Kunze, Koichi Kise, Jens Weppner, Andreas Dengel, Paul Lukowicz, and Andreas Bulling. 2014. In the Blink of an Eye - Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass. ACM International Conference Proceeding Series, 150--153. https://doi.org/10.1145/2582051.2582066Google ScholarDigital Library
- Shoya Ishimaru, Kai Kunze, Katsuma Tanaka, Yuji Uema, Koichi Kise, and Masahiko Inami. 2014. Smarter Eyewear- Using Commercial EOG Glasses for Activity Recognition. UbiComp 2014 - Adjunct Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, 153--156. https://doi.org/10.1145/2638728.2638795Google ScholarDigital Library
- Md Islam, Shuji Sakamoto, Yoshihiro Yamada, Andrew Vargo, Motoi Iwata, Masakazu Iwamura, Koichi Kise, et al. 2020. Self-supervised Deep Learning for Reading Activity Classification. arXiv preprint arXiv:2012.03598 (2020).Google Scholar
- Hans Spruit Jochen Hellmig Johannes Meyer, Thomas Schlebusch and Enkelejda Kasneci. 2021. A Novel Gaze Gesture Sensor for Smart Glasses Based on Laser Self-Mixing. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI '21 Extended Abstracts). ACM, ACM, New York, NY, USA. https://doi.org/10.1145/3411763.3451621Google ScholarDigital Library
- Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-Based Interaction. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (Seattle, Washington) (UbiComp '14 Adjunct). Association for Computing Machinery, New York, NY, USA, 1151--1160. https://doi.org/10.1145/2638728.2641695Google ScholarDigital Library
- Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG]Google Scholar
- Ashish Kumar, Saurabh Goyal, and Manik Varma. 2017. Resource-efficient Machine Learning in 2 KB RAM for the Internet of Things. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70), Doina Precup and Yee Whye Teh (Eds.). PMLR, 1935--1944. http://proceedings.mlr.press/v70/kumar17a.htmlGoogle Scholar
- Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. 2011. Activity Recognition Using Cell Phone Accelerometers. SIGKDD Explor. Newsl. 12, 2 (March 2011), 74--82. https://doi.org/10.1145/1964897.1964918Google ScholarDigital Library
- Guohao Lan, Bailey Heit, Tim Scargill, and Maria Gorlatova. 2020. GazeGraph: Graph-Based Few-Shot Cognitive Context Sensing from Human Visual Behavior. Association for Computing Machinery, New York, NY, USA, 422--435. https://doi.org/10.1145/3384419.3430774Google ScholarDigital Library
- R. Lang and K. Kobayashi. 1980. External optical feedback effects on semiconductor injection laser properties. IEEE Journal of Quantum Electronics 16, 3 (1980), 347--355.Google ScholarCross Ref
- L. Lee and P. Hui. 2018. Interaction Methods for Smart Glasses: A Survey. IEEE Access 6 (2018), 28712--28732.Google ScholarCross Ref
- Xinyu Li, Yanyi Zhang, Ivan Marsic, Aleksandra Sarcevic, and Randall Burd. 2016. Deep Learning for RFID-Based Activity Recognition. Proceedings of the ... International Conference on Embedded Networked Sensor Systems. International Conference on Embedded Networked Sensor Systems 2016. https://doi.org/10.1145/2994551.2994569Google ScholarDigital Library
- Konstantin Lübeck and Oliver Bringmann. 2019. A Heterogeneous and Reconfigurable Embedded Architecture for Energy-Efficient Execution of Convolutional Neural Networks. In Architecture of Computing Systems - ARCS 2019, Martin Schoeberl, Christian Hochberger, Sascha Uhrig, Jürgen Brehm, and Thilo Pionteck (Eds.). Springer International Publishing, Cham, 267--280.Google ScholarCross Ref
- Päivi Majaranta and Andreas Bulling. 2014. Eye Tracking and Eye-Based Human-Computer Interaction. Springer London, London, 39--65. https://doi.org/10.1007/978-1-4471-6392-3_3Google ScholarCross Ref
- J. Meyer, T. Schlebusch, W. Fuhl, and E. Kasneci. 2020. A Novel Camera-Free Eye Tracking Sensor for Augmented Reality Based on Laser Scanning. IEEE Sensors Journal 20, 24 (2020), 15204--15212. https://doi.org/10.1109/JSEN.2020.3011985Google ScholarCross Ref
- Johannes Meyer, Thomas Schlebusch, Hans Spruit, Jochen Hellmig, and Enkelejda Kasneci. 2020. A Novel -Eye-Tracking Sensor for AR Glasses Based on Laser Self-Mixing Showing Exceptional Robustness Against Illumination. In ACM Symposium on Eye Tracking Research and Applications (Stuttgart, Germany) (ETRA '20 Short Papers). Association for Computing Machinery, New York, NY, USA, Article 31, 5 pages. https://doi.org/10.1145/3379156.3391352Google ScholarDigital Library
- Johannes Meyer, Thomas Schlebusch, Hans Spruit, Jochen Hellmig, and Enkelejda Kasneci. 2021. A compact low-power gaze gesture sensor based on laser feedback interferometry for smart glasses. In Digital Optical Technologies 2021, Bernard C. Kress and Christophe Peroz (Eds.), Vol. 11788. International Society for Optics and Photonics, SPIE, 57 - 69. https://doi.org/10.1117/12.2593772Google ScholarCross Ref
- Microsoft Corporation. 2021. Azure Kinect DK - Develop AI models | Microsoft Azure. https://azure.microsoft.com/en-gb/services/kinect-dk/. Online; accessed 14 April 2021.Google Scholar
- Microsoft Corporation. 2021. Kinect - Windows app development. https://developer.microsoft.com/en-gb/windows/kinect/. Online; accessed 14 April 2021.Google Scholar
- Diederick C Niehorster, Thiago Santini, Roy S Hessels, Ignace TC Hooge, Enkelejda Kasneci, and Marcus Nyström. 2020. The impact of slippage on the data quality of head-worn eye trackers. Behavior Research Methods 52, 3 (2020), 1140--1160.Google ScholarCross Ref
- Nintendo Co., Ltd. 2021. Nintendo Switch | Nintendo Switch Family | Nintendo. https://www.nintendo.co.uk/Nintendo-Switch/Nintendo-Switch-1148779.html. Online; accessed 14 April 2021.Google Scholar
- Francisco Javier OrdÃşÃśez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors 16, 1 (2016). https://doi.org/10.3390/s16010115Google ScholarCross Ref
- Mathias Perslev, Sune Darkner, Lykke Kempfner, Miki Nikolic, Poul Jørgen Jennum, and Christian Igel. 2021. U-Sleep: resilient high-frequency sleep staging. NPJ digital medicine 4, 1 (2021), 1--12.Google Scholar
- Armand Pruijmboom, Marcel Schemmann, Jochen Hellmig, Jeroen Schutte, Holger Moench, and Joseph Pankert. 2008. VCSEL-based miniature laser-Doppler interferometer. In Vertical-Cavity Surface-Emitting Lasers XII, Chun Lei and James K. Guenter (Eds.), Vol. 6908. International Society for Optics and Photonics, SPIE, 157 - 163. https://doi.org/10.1117/12.775131Google ScholarCross Ref
- Johann Riemensberger, Anton Lukashchuk, Maxim Karpov, Wenle Weng, Erwan Lucas, Junqiu Liu, and Tobias J Kippenberg. 2020. Massively parallel coherent laser ranging using a soliton microcomb. Nature 581, 7807 (2020), 164--170.Google ScholarCross Ref
- Charissa Ann Ronao and Sung-Bae Cho. 2016. Human activity recognition with smartphone sensors using deep learning neural networks. Expert Systems with Applications 59 (2016), 235--244. https://doi.org/10.1016/j.eswa.2016.04.032Google ScholarDigital Library
- Runtopia Technology Ltd. 2021. RUNTOPIA|REACH. Faster, Longer, Safer. https://landing.runtopia.net/. Online; accessed 14 April 2021.Google Scholar
- SALTED Ltd. 2021. SALTED. https://www.salted.ltd/en/insole. Online; accessed 14 April 2021.Google Scholar
- N. Sarkar, B. O'Hanlon, A. Rohani, D. Strathearn, G. Lee, M. Olfat, and R. R. Mansour. 2017. A resonant eye-tracking microsystem for velocity estimation of saccades and foveated rendering. In 2017 IEEE 30th International Conference on Micro Electro Mechanical Systems (MEMS). 304--307. https://doi.org/10.1109/MEMSYS.2017.7863402Google ScholarCross Ref
- Sensoria Inc. 2021. Sensoria Home Page. https://www.sensoriafitness.com/. Online; accessed 14 April 2021.Google Scholar
- Bosch Sensortec. 2020. BMI270 6-axis, smart, low-power Inertial Measurement Unit for high-performance applications. https://www.bosch-sensortec.com/products/motion-sensors/imus/bmi270Google Scholar
- Julian Steil and Andreas Bulling. 2015. Discovery of Everyday Human Activities from Long-Term Visual Behaviour Using Topic Models. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Osaka, Japan) (UbiComp '15). Association for Computing Machinery, New York, NY, USA, 75--85. https://doi.org/10.1145/2750858.2807520Google ScholarDigital Library
- Julian Steil, Philipp Müller, Yusuke Sugano, and Andreas Bulling. 2018. Forecasting User Attention during Everyday Mobile Interactions Using Device-Integrated and Wearable Sensors. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (Barcelona, Spain) (MobileHCI '18). Association for Computing Machinery, New York, NY, USA, Article 1, 13 pages. https://doi.org/10.1145/3229434.3229439Google ScholarDigital Library
- Thomas Taimre, Milan Nikolić, Karl Bertling, Yah Leng Lim, Thierry Bosch, and Aleksandar D. Rakić. 2015. Laser feedback interferometry: a tutorial on the self-mixing effect for coherent sensing. Adv. Opt. Photon. 7, 3 (Sep 2015), 570--631. https://doi.org/10.1364/AOP.7.000570Google ScholarCross Ref
- Marc Tonsen, Chris Kay Baumann, and Kai Dierkes. 2020. A High-Level Description and Performance Evaluation of Pupil Invisible. arXiv:2009.00508 [cs.CV]Google Scholar
- Florian Wahl, Martin Freund, and Oliver Amft. 2015. WISEglass: Smart eyeglasses recognising context. EAI Endorsed Transactions on Pervasive Health and Technology 2, 5 (12 2015). https://doi.org/10.4108/eai.28-9-2015.2261470Google ScholarDigital Library
- Shaohua Wan, Lianyong Qi, Xiaolong Xu, Chao Tong, and Zonghua Gu. 2020. Deep Learning Models for Real-time Human Activity Recognition with Smartphones. Mobile Networks and Applications 25 (04 2020). https://doi.org/10.1007/s11036-019-01445-xGoogle ScholarDigital Library
- Mei Wang and Weihong Deng. 2020. Deep Face Recognition: A Survey. Neurocomputing (2020).Google Scholar
- Jian-Bo Yang, Nguyen Nhut, Phyo San, Xiaoli li, and Priyadarsini Shonali. 2015. Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition. IJCAI (07 2015).Google Scholar
Index Terms
- A CNN-based Human Activity Recognition System Combining a Laser Feedback Interferometry Eye Movement Sensor and an IMU for Context-aware Smart Glasses
Recommendations
Eye movement analysis for activity recognition
UbiComp '09: Proceedings of the 11th international conference on Ubiquitous computingIn this work we investigate eye movement analysis as a new modality for recognising human activity. We devise 90 different features based on the main eye movement characteristics: saccades, fixations and blinks. The features are derived from eye ...
A Highly Integrated Ambient Light Robust Eye-Tracking Sensor for Retinal Projection AR Glasses Based on Laser Feedback Interferometry
ETRARobust and highly integrated eye-tracking is a key technology to improve resolution of near-eye-display technologies for augmented reality (AR) glasses such as focus-free retinal projection as it enables display enhancements like foveated rendering. ...
Eye Blink Detection for Smart Glasses
ISM '13: Proceedings of the 2013 IEEE International Symposium on MultimediaEye blink is a quick action of closing and opening of the eyelids. Eye blink detection has a wide range of applications in human computer interaction and human vision health care research. Existing approaches to eye blink detection often cannot suit ...
Comments