Exploring Context-Aware User Interfaces for Smartphone-Smartwatch Cross-Device Interaction

Abstract

In this study, we explore context-aware cross-device interactions between a smartphone and smartwatch. We present 24 contexts, and then examine and prioritize suitable user interfaces (UIs) for each. In addition, we present example applications, including a map, notification management system, multitasking application, music player, and video chat application, each of which has its own context-aware UIs. To support these context-aware UIs, we investigate the performance of our context recognizer in which recognition is based on machine-learning using the accelerometers in a smartphone and smartwatch. We conduct seven different evaluations using four machine-learning algorithms: J48 decision tree, sequential minimal optimization (SMO)-based support vector machine (SVM), random forest, and multilayer perceptron. With each algorithm, we conduct a long-interval experiment to examine the level of accuracy at which each context is recognized using data previously collected for training. The results show that SMO-based SVM is suitable for recognizing the 24 contexts considered in this study.

References

1. Ian Anderson, Julie Maitland, Scott Sherwood, Louise Barkhuus, Matthew Chalmers, Malcolm Hall, Barry Brown, and Henk Muller. 2007. Shakra: Tracking and Sharing Daily Activity Levels with Unaugmented Mobile Phones. Mobile Networks and Applications 12, 2 (2007), 185--s199.
2. Ling Bao and Stephen S. Intille. 2004. Activity Recognition from User-Annotated Acceleration Data. In Proceedings of the 2nd International Conference on Pervasive Computing (Pervasive ’04). Springer, 1--17.
3. Sebastian Boring, David Ledo, Xiang ‘Anthony’ Chen, Nicolai Marquardt, Anthony Tang, and Saul Greenberg. 2012. The Fat Thumb: Using the Thumb’s Contact Size for Single-handed Mobile Interaction. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI ’12). ACM, New York, NY, USA, 39--48.
4. Agata Brajdic and Robert Harle. 2013. Walk Detection and Step Counting on Unconstrained Smartphones. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’13). ACM, New York, NY, USA, 225--234.
5. Xiang ‘Anthony’ Chen, Tovi Grossman, Daniel J. Wigdor, and George Fitzmaurice. 2014. Duet: Exploring Joint Interactions on a Smart Phone and a Smart Watch. In Proceedings of the 32nd SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 159--168.
6. Lung-Pan Cheng, Hsiang-Sheng Liang, Che-Yang Wu, and Mike Y. Chen. 2013. iGrasp: Grasp-based Adaptive Keyboard for Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). ACM, New York, NY, USA, 3037--3046.
7. Rajkumar Darbar and Debasis Samanta. 2015. SurfaceSense: Smartphone Can Recognize Where It Is Kept. In Proceedings of the 7th International Conference on HCI, IndiaHCI 2015 (IndiaHCI’15). ACM, New York, NY, USA, 39--46.
8. Anind K. Dey, Gregory D. Abowd, and Daniel Salber. 2001. A Conceptual Framework and a Toolkit for Supporting the Rapid Prototyping of Context-aware Applications. Human-Computer Interaction 16, 2 (Dec. 2001), 97--166.
9. Mayank Goel, Jacob Wobbrock, and Shwetak Patel. 2012. GripSense: Using Built-in Sensors to Detect Hand Posture and Pressure on Commodity Mobile Phones. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST ’12). ACM, New York, NY, USA, 545--554.
10. Mitsuhiro Goto, Hideaki Kimata, Masahiko Toyoshi, Tatsuya Nishikiori, Kosuke Moriwaki, and Koji Nakamura. 2015. A Wearable Action Support System for Business Use by Context-aware Computing Based on Web Schedule. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (UbiComp/ISWC’15 Adjunct). ACM, New York, NY, USA, 53--56.
11. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations Newsletter 11, 1 (Nov. 2009), 10--18.
12. Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST ’11). ACM, New York, NY, USA, 627--636.
13. Seongkook Heo, Jiseong Gu, and Geehyuk Lee. 2014. Expanding Touch Input Vocabulary by using Consecutive Distant Taps. In Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 2597--2606.
14. Ken Hinckley, Seongkook Heo, Michel Pahud, Christian Holz, Hrvoje Benko, Abigail Sellen, Richard Banks, Kenton O’Hara, Gavin Smyth, and William Buxton. 2016. Pre-Touch Sensing for Mobile Interaction. In Proceedings of the 34th SIGCHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 2869--2881.
15. Ken Hinckley, Michel Pahud, Hrvoje Benko, Pourang Irani, François Guimbretière, Marcel Gavriliu, Xiang ‘Anthony’ Chen, Fabrice Matulic, William Buxton, and Andrew Wilson. 2014. Sensing Techniques for Tablet+Stylus Interaction. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST ’14). ACM, New York, NY, USA, 605--614.
16. Ken Hinckley, Jeff Pierce, Mike Sinclair, and Eric Horvitz. 2000. Sensing Techniques for Mobile Interaction. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST ’00). ACM, New York, NY, USA, 91--100.
17. Han-Jong Kim, Seijin Cha, Richard C. Park, Tek-Jin Nam, Woohun Lee, and Geehyuk Lee. 2016. Mo-Bi: Contextual Mobile Interfaces Through Bimanual Posture Sensing with Wrist-Worn Devices. In Proceedings of HCI Korea (HCIK ’16). Hanbit Media, Inc., South Korea, 94--99.
18. Yuki Kubo, Ryosuke Takada, Buntarou Shizuki, and Shin Takahashi. 2017. SynCro: Context-Aware User Interface System for Smartphone-Smartwatch Cross-Device Interaction. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHIEA ’17). ACM, New York, NY, USA, 1794--1801.
19. Jennifer R. Kwapisz, Gary M. Weiss, and Samuel A. Moore. 2011. Activity Recognition using Cell Phone Accelerometers. SIGKDD Explorations Newsletter 12, 2 (March 2011), 74--82.
20. Hosub Lee, Young Sang Choi, Sunjae Lee, and Eunsoo Shim. 2013. Smart Pose: Mobile Posture-aware System for Lowering Physical Health Risk of Smartphone Users. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (CHIEA ’13). ACM, New York, NY, USA, 2257--2266.
21. Seungwoo Lee, Yungeun Kim, Daye Ahn, Rhan Ha, Kyoungwoo Lee, and Hojung Cha. 2015. Non-obstructive Room-level Locating System in Home Environments using Activity Fingerprints from Smartwatch. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 939--950.
22. Hyunchul Lim, Gwangseok An, Yoonkyong Cho, Kyogu Lee, and Bongwon Suh. 2016. WhichHand: Automatic Recognition of a Smartphone’s Position in the Hand using a Smartwatch. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI ’16). ACM, New York, NY, USA, 675--681.
23. Luyang Liu, Cagdas Karatas, Hongyu Li, Sheng Tan, Marco Gruteser, Jie Yang, Yingying Chen, and Richard P. Martin. 2015. Toward Detection of Unsafe Driving with Wearables. In Proceedings of the 1st Workshop on Wearable Systems and Applications (WearSys ’15). ACM, New York, NY, USA, 27--32.
24. Bilge Mutlu, Andreas Krause, Jodi Forlizzi, Carlos Guestrin, and Jessica Hodgins. 2007. Robust, Low-cost, Non-intrusive Sensing and Recognition of Seated Postures. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST ’07). ACM, New York, NY, USA, 149--158.
25. Woori Noh, Mankyung Lee, Hyelim Cheon, Joohee Kim, Kwangjae Lee, and Jundong Cho. 2016. TakeOut: Drawing Application using Distributed User Interface for Being Close to Real Experience. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp ’16).ACM, New York, NY, USA, 173--176.
26. Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch 8 Activate: Adding Interactivity to Existing Objects using Active Acoustic Sensing. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13). ACM, New York, NY, USA, 31--40.
27. Chanho Park and Takefumi Ogawa. 2015. A Study on Grasp Recognition Independent of Users’ Situations using Built-in Sensors of Smartphones. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST ’15 Adjunct). ACM, New York, NY, USA, 69--70.
28. Jun-geun Park, Ami Patel, Dorothy Curtis, Seth Teller, and Jonathan Ledlie. 2012. Online Pose Classification and Walking Speed Estimation using Handheld Devices. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp’12). ACM, New York, NY, USA, 113--122.
29. Anand Ranganathan and Roy H. Campbell. 2003. An Infrastructure for Context-awareness Based on First Order Logic. Personal Ubiquitous Comput. 7, 6 (Dec. 2003), 353--364.
30. Juhi Ranjan and Kamin Whitehouse. 2015. Object Hallmarks: Identifying Object Users using Wearable Wrist Sensors. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 51--61.
31. Jun Rekimoto. 1996. Tilting Operations for Small Screen Interfaces. In Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology (UIST ’96). ACM, New York, NY, USA, 167--168.
32. Jun Rekimoto. 1997. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology (UIST ’97). ACM, New York, NY, USA, 31--39.
33. Nazir Saleheen, Amin Ahsan Ali, Syed Monowar Hossain, Hillol Sarker, Soujanya Chatterjee, Benjamin Marlin, Emre Ertin, Mustafa al’Absi, and Santosh Kumar. 2015. puffMarker: A Multi-sensor Approach for Pinpointing the Timing of First Lapse in Smoking Cessation. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 999--1010.
34. Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY, USA, 483--492.
35. Bill N. Schilit, Norman Adams, and Roy Want. 1994. Context-Aware Computing Applications. In Proceedings of the 1994 First Workshop on Mobile Computing Systems and Applications (WMCSA ’94). IEEE Computer Society, Washington, DC, USA, 85--90.
36. Albrecht Schmidt, Kofi Asante Aidoo, Antti Takaluoma, Urpo Tuomela, Kristof Van Laerhoven, and Walter Van de Velde. 1999. Advanced Interaction in Context. In Proceedings of the 1st International Symposium on Handheld and Ubiquitous Computing (HUC ’99). Springer-Verlag, London, UK, UK, 89--101. http://dl.acm.org/citation.cfm?id=647985.743849
37. Dominik Schmidt, Julian Seifert, Enrico Rukzio, and Hans Gellersen. 2012. A Cross-device Interaction Style for Mobiles and Surfaces. In Proceedings of the Designing Interactive Systems Conference (DIS ’12). ACM, New York, NY, USA, 318--327.
38. Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I Am a Smartwatch and I Can Track My User’s Arm. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys ’16). ACM, New York, NY, USA, 85--96.
39. Katsuma Tanaka, Shoya Ishimaru, Koichi Kise, Kai Kunze, and Masahiko Inami. 2015. Nekoze!: Monitoring and Detecting Head Posture While Working with Laptop and Mobile Phone. In Proceedings of the 9th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth ’15). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, Belgium, 237--240. http://dl.acm.org/citation.cfm?id=2826165.2826203
40. Edison Thomaz, Irfan Essa, and Gregory D. Abowd. 2015. A Practical Approach for Recognizing Eating Moments with Wrist-mounted Inertial Sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp ’15). ACM, New York, NY, USA, 1029--1040.
41. Raphael Wimmer and Sebastian Boring. 2009. HandSense: Discriminating Different Ways of Grasping and Holding a Tangible User Interface. In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (TEI ’09). ACM, New York, NY, USA, 359--362.
42. Yoonsik Yang, Seungho Chae, Jinwook Shim, and Tack-Don Han. 2015. EMG Sensor-based Two-Hand Smart Watch Interaction. In Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST ’15 Adjunct). ACM, New York, NY, USA, 73--74.
43. Dongwook Yoon, Ken Hinckley, Hrvoje Benko, François Guimbretière, Pourang Irani, Michel Pahud, and Marcel Gavriliu. 2015. Sensing Tablet Grasp + Micro-mobility for Active Reading. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST ’15). ACM, New York, NY, USA, 477-487.

Index Terms

• Human-centered computing

  • Interaction design

    • Interaction design process and methods

      • Activity centered design

      • Contextual design

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile devices

      • Mobile devices