Wall++: Room-Scale Interactive and Context-Aware Sensing

ABSTRACT

Human environments are typified by walls, homes, offices, schools, museums, hospitals and pretty much every indoor context one can imagine has walls. In many cases, they make up a majority of readily accessible indoor surface area, and yet they are static their primary function is to be a wall, separating spaces and hiding infrastructure. We present Wall++, a low-cost sensing approach that allows walls to become a smart infrastructure. Instead of merely separating spaces, walls can now enhance rooms with sensing and interactivity. Our wall treatment and sensing hardware can track users' touch and gestures, as well as estimate body pose if they are close. By capturing airborne electromagnetic noise, we can also detect what appliances are active and where they are located. Through a series of evaluations, we demonstrate Wall++ can enable robust room-scale interactive and context-aware applications.

References

1. Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Trans. Graph. 34, 6, Article 219 (October 2015), 13 pages.
2. Ramezani Akhmareh, Alireza, Mihai Teodor Lazarescu, Osama Bin Tariq, and Luciano Lavagno. "A Tagless Indoor Localization System Based on Capacitive Sensing Technology." Sensors 16, no. 9 (2016): 1448.
3. Analog Devices. Datasheet. Retrieved September 18, 2017 from http://www.analog.com/media/en/technicaldocumentation/data-sheets/AD5930.pdf
4. Analog Devices. Datasheet. Retrieved September 18, 2017 from http://www.analog.com/media/en/technicaldocumentation/data-sheets/AD637.pdf
5. Thomas Augsten, Konstantin Kaefer, René Meusel, Caroline Fetzer, Dorian Kanitz, Thomas Stoff, Torsten Becker, Christian Holz, and Patrick Baudisch. 2010. Multitoe: high-precision interaction with back-projected floors based on high-resolution multi-touch input. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 209--218.
6. Gary Barrett, and Ryomei Omote. "Projected-capacitive touch technology." Information Display 26, no. 3 (2010): 16--21.
7. Hrvoje Benko, Andrew D. Wilson, and Federico Zannier. 2014. Dyadic projected spatial augmented reality. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14). ACM, New York, NY, USA, 645--655.
8. Larry K. Baxter "Capacitive sensors." Ann Arbor 1001 (2000): 48109.
9. Alan Bränzel, Christian Holz, Daniel Hoffmann, Dominik Schmidt, Marius Knaust, Patrick Lühne, René Meusel, Stephan Richter, and Patrick Baudisch. 2013. GravitySpace: tracking users and their poses in a smart room using a pressure-sensing floor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 725--734.
10. Andreas Braun, Sebastian Zander-Walz, Stefan Krepp, Silvia Rus, Reiner Wichert, and Arjan Kuijper. 2016. CapTap: combining capacitive gesture recognition and acoustic touch detection. In Proceedings of the 3rd International Workshop on Sensor-based Activity Recognition and Interaction (iWOAR '16). ACM, New York, NY, USA, Article 6, 6 pages.
11. Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. "Realtime multi-person 2d pose estimation using part affinity fields." arXiv preprint arXiv:1611.08050 (2016).
12. Circuit Scribe. Product Information. Retrieved September 18,, 2017 from https://www.circuitscribe.com/product/circuit-scribepen-single/
13. COMSOL multiphysics user's guide. Version: September 10 (2005): 333.
14. Gabriel Deak, Kevin Curran, and Joan Condell. 2012. Review: A survey of active and passive indoor localization systems. Comput. Commun. 35, 16 (September 2012), 1939--1954.
15. Freescale Semiconductor. Datasheet. Retrieved September 18, 2017 from http://cache.freescale.com/files/32bit/doc/data_sheet/K2 0P64M72SF1.pdf
16. Adrian A. de Freitas, Michael Nebeling, Xiang 'Anthony' Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K. Dey. 2016. Snap-To-It: A User-Inspired Platform for Opportunistic Device Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5909--5920.
17. Harald. T. Friis, "A Note on a Simple Transmission Formula" in Proceedings of the IRE, vol. 34, no. 5, pp. 254--256, May 1946.
18. Albert Glinsky. 2000. Theremin: Ether Music and Espionage. University of Illinois Press. http://books.google.de/books?id=6DHlQJcMpBQC
19. Tobias Grosse-Puppendahl, Sebastian Herber, Raphael Wimmer, Frank Englert, Sebastian Beck, Julian von Wilmsdorff, Reiner Wichert, and Arjan Kuijper. 2014. Capacitive near-field communication for ubiquitous interaction and perception. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). ACM, New York, NY, USA, 231--242.
20. Tobias Grosse-Puppendahl, Alexander Marinc, and Andreas Braun. 2011. Classification of user poses with capacitive proximity sensors in AAL-Environments. In Proceedings of the Second international conference on Ambient Intelligence (AmI'11). Springer-Verlag, Berlin, Heidelberg, 314--323.
21. Tobias Grosse-Puppendahl, Christian Holz, Gabe Cohn, Raphael Wimmer, Oskar Bechtold, Steve Hodges, Matthew S. Reynolds, and Joshua R. Smith. 2017. Finding Common Ground: A Survey of Capacitive Sensing in Human-Computer Interaction. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3293- 3315.
22. Nan-Wei Gong, Steve Hodges, and Joseph A. Paradiso. 2011. Leveraging conductive inkjet technology to build a scalable and versatile surface for ubiquitous sensing. In Proceedings of the 13th international conference on Ubiquitous computing (UbiComp '11). ACM, New York, NY, USA, 45--54.
23. Manoj Gulati, Shobha Sundar Ram, and Amarjeet Singh. 2014. An in depth study into using EMI signatures for appliance identification. In Proceedings of the 1st ACM Conference on Embedded Systems for Energy-Efficient Buildings (BuildSys '14). ACM, New York, NY, USA, 70--79.
24. Sidhant Gupta, Matthew S. Reynolds, and Shwetak N. Patel. 2010. ElectriSense: single-point sensing using EMI for electrical event detection and classification in the home. In Proceedings of the 12th ACM international conference on Ubiquitous computing (UbiComp '10). ACM, New York, NY, USA, 139--148.
25. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: an update. SIGKDD Explor. Newsl. 11, 1 (November 2009), 10--18.
26. Josef Hallberg, Marcus Nilsson, and Kåre Synnes. "Positioning with bluetooth." In Telecommunications, 2003. ICT 2003. 10th International Conference on, vol. 2, pp. 954--958. IEEE, 2003.
27. Chris Harrison and Scott E. Hudson. 2008. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST '08). ACM, New York, NY, USA, 205- 208.
28. George Hart. "Nonintrusive appliance load monitoring." Proceedings of the IEEE 80, no. 12 (1992): 1870--1891.
29. Hiroshi Ishii, Craig Wisneski, Julian Orbanes, Ben Chun, and Joe Paradiso. 1999. PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 394--401.
30. Brett Jones, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, Nikunj Raghuvanshi, and Lior Shapira. 2014. RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST '14). ACM, New York, NY, USA, 637--644.
31. Younghun Kim, Thomas Schmid, Zainul M. Charbiwala, and Mani B. Srivastava. 2009. ViridiScope: design and implementation of a fine grained power monitoring system for homes. In Proceedings of the 11th international conference on Ubiquitous computing (UbiComp '09). ACM, New York, NY, USA, 245--254.
32. Antti Kotanen, Marko Hannikainen, Helena Leppakoski, and Timo D. Hamalainen. "Experiments on local positioning with Bluetooth." In Information Technology: Coding and Computing {Computers and Communications}, 2003. Proceedings. ITCC 2003. International Conference on, pp. 297--303. IEEE, 2003.
33. Stacey Kuznetsov and Eric Paulos. 2010. UpStream: motivating water conservation with low-cost water flow sensing and persuasive displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1851- 1860.
34. Gierad Laput, Walter S. Lasecki, Jason Wiese, Robert Xiao, Jeffrey P. Bigham, and Chris Harrison. 2015. Zensors: Adaptive, Rapidly Deployable, Human-Intelligent Sensor Feeds. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1935--1944.
35. Gierad Laput, Chouchang Yang, Robert Xiao, Alanson Sample, and Chris Harrison. 2015. EM-Sense: Touch Recognition of Uninstrumented, Electrical and Electromechanical Objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 157- 166.
36. Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 321--333.
37. Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak Patel. 2011. HeatWave: thermal imaging for surface user interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2565--2574.
38. Hanchuan Li, Can Ye, and Alanson P. Sample. 2015. IDSense: A Human Object Interaction Detection System Based on Passive UHF RFID. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2555--2564.
39. Hanchuan Li, Eric Brockmeyer, Elizabeth J. Carter, Josh Fromm, Scott E. Hudson, Shwetak N. Patel, and Alanson Sample. 2016. PaperID: A Technique for Drawing Functional Battery-Free Wireless Interfaces on Paper. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5885--5896.
40. Hanchuan Li, Peijin Zhang, Samer Al Moubayed, Shwetak N. Patel, and Alanson P. Sample. 2016. ID-Match: A Hybrid Computer Vision and RFID System for Recognizing Individuals in Groups. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4933- 4944.
41. Living Wall, Online Documentation. Retrieved January 5, 2018 from http://highlowtech.org/?p=27
42. Hui Liu, H. Darabi, P. Banerjee, and Jing Liu. 2007. Survey of Wireless Indoor Positioning Techniques and Systems. Trans. Sys. Man Cyber Part C 37, 6 (November 2007), 1067--1080.
43. Takuya Maekawa, Yutaka Yanagisawa, Yasue Kishino, Katsuhiko Ishiguro, Koji Kamei, Yasushi Sakurai, and Takeshi Okadome. 2010. Object-based activity recognition with heterogeneous sensors on wrist. In Proceedings of the 8th international conference on Pervasive Computing (Pervasive'10). Springer-Verlag, Berlin, Heidelberg, 246--264.
44. Jon Moeller and Andruid Kerne. 2012. ZeroTouch: an optical multi-touch and free-air interaction architecture. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI '12). ACM, New York, NY, USA, 2165--2174.
45. Adiyan Mujibiya and Jun Rekimoto. 2013. Mirage: exploring interaction modalities using off-body static electric field sensing. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13). ACM, New York, NY, USA, 211- 220.
46. Lionel M. Ni, Yunhao Liu, Yiu Cho Lau, and Abhishek P. Patil. 2004. LANDMARC: indoor location sensing using active RFID. Wirel. Netw. 10, 6 (November 2004), 701--710.
47. MG Chemicals. Product Information. Retrieved September 18, 2017 from http://www.mgchemicals.com/products/emi-and-rfishielding/acrylic-conductive-coatings-ar-series/838artotal-ground-carbon-conductive-coating
48. MG Chemicals. Product Information. Retrieved September 18, 2017 from http://www.mgchemicals.com/products/emi-and-rfishielding/water-based-conductive-coatings-wbseries/841wb-super-shield-water-based-nickelconductive-coating
49. MG Chemicals. Product Information. Retrieved September 18, 2017 from http://www.mgchemicals.com/products/emi-and-rfishielding/acrylic-conductive-coatings-ar-series/841arsuper-shield-nickel-conductive-coating
50. MG Chemicals. Product Information. Retrieved September 18, 2017 from http://www.mgchemicals.com/products/emi-and-rfishielding/water-based-conductive-coatings-wbseries/842wb-super-shield-water-based-silverconductive-coating
51. O. Omojola, E. Rehmi Post, M. D. Hancher, Y. Maguire, R. Pappu, B. Schoner, P. R. Russo, R. Fletcher, and N. Gershenfeld. 2000. An installation of interactive furniture. IBM Syst. J. 39, 3--4 (July 2000), 861--879.
52. Sophocles J. Orfanidis Electromagnetic waves and antennas. New Brunswick, NJ: Rutgers University, 2002.
53. Joseph A. Paradiso, Kai-yuh Hsiao, Joshua Strickon, Joshua Lifton, and Ari Adler. "Sensor systems for interactive surfaces." IBM Systems Journal 39, no. 3.4 (2000): 892--914.
54. Joseph A. Paradiso, Che King Leo, Nisha Checka, and Kaijen Hsiao. 2002. Passive acoustic knock tracking for interactive windows. In CHI '02 Extended Abstracts on Human Factors in Computing Systems (CHI EA '02). ACM, New York, NY, USA, 732--733.
55. PJRC. Product Information. Retrieved September 18, 2017 from https://www.pjrc.com/teensy/teensy31.html
56. Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking (MobiCom '13). ACM, New York, NY, USA, 27--38.
57. Tauhidur Rahman, Alexander T. Adams, Ruth Vinisha Ravichandran, Mi Zhang, Shwetak N. Patel, Julie A. Kientz, and Tanzeem Choudhury. 2015. DoppleSleep: a contactless unobtrusive sleep sensing system using short-range Doppler radar. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 39--50.
58. Niranjini Rajagopal, Suman Giri, Mario Berges, and Anthony Rowe. 2013. A magnetic field-based appliance metering system. In Proceedings of the ACM/IEEE 4th International Conference on Cyber-Physical Systems (ICCPS '13). ACM, New York, NY, USA, 229--238.
59. Jun Rekimoto. 2002. SmartSkin: an infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '02). ACM, New York, NY, USA, 113--120.
60. Jun Rekimoto, and Nobuyuki Matsushita. "Perceptual surfaces: Towards a human and object sensitive interactive display." In Workshop on Perceptual User Interfaces (PUI'97), pp. 30--32. 1997.
61. Michael Rohs, and Beat Gfeller. Using camera-equipped mobile phones for interacting with real-world objects. na, 2004.
62. Anthony Rowe, Mario Berges, and Raj Rajkumar. 2010. Contactless sensing of appliance state transitions through variations in electromagnetic fields. In Proceedings of the 2nd ACM Workshop on Embedded Sensing Systems for Energy-Efficiency in Building (BuildSys '10). ACM, New York, NY, USA, 19--24.
63. Muhamad Risqi Utama Saputra, Guntur Dharma Putra, and Paulus Insap Santosa. "Indoor human tracking application using multiple depth-cameras." In Advanced Computer Science and Information Systems (ICACSIS), 2012 International Conference on, pp. 307--312. IEEE, 2012.
64. SciPy.org Retrieved September 19, 2017 from https://docs.scipy.org/doc/scipy/reference/optimize.html
65. Joshua R. Smith, Tom White, Christopher Dodge, Joseph Paradiso, Neil Gershenfeld, and David Allport. 1998. Electric Field Sensing For Graphical Interfaces. IEEE Comput. Graph. Appl. 18, 3 (May 1998), 54--60.
66. Joshua R. Smith. 1999. Electric Field Imaging. Ph.D. Dissertation. Massachusetts Institute of Technology, Cambridge, MA, USA. AAI0800637.
67. Andrew Spielberg, Alanson Sample, Scott E. Hudson, Jennifer Mankoff, and James McCann. 2016. RapID: A Framework for Fabricating Low-Latency Interactive Objects with RFID Tags. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5897- 5908.
68. Shih-Wei Sun, Chien-Hao Kuo, and Pao-Chi Chang. 2016. People tracking in an environment with multiple depth cameras. J. Vis. Comun. Image Represent. 35, C (February 2016), 36--54.
69. Miika Valtonen, Jaakko Maentausta, and Jukka Vanhala. 2009. TileTrack: Capacitive human tracking using floor tiles. In Proceedings of the 2009 IEEE International Conference on Pervasive Computing and Communications (PERCOM '09). IEEE Computer Society, Washington, DC, USA, 1--10.
70. Miika Valtonen, Lasse Kaila, Jaakko Mäentausta, and Jukka Vanhala. 2011. Unobtrusive human height and pose recognition with a capacitive sensor. J. Ambient Intell. Smart Environ. 3, 4 (December 2011), 305--332.
71. Cati Vaucelle, Hiroshi Ishii, and Joseph A. Paradiso. 2009. Cost-effective wearable sensor to detect EMF. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, New York, NY, USA, 4309--4314.
72. Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RFIDraw: virtual touch screen in the air using RF signals. SIGCOMM Comput. Commun. Rev. 44, 4 (August 2014), 235--246.
73. Joey Wilson and Neal Patwari. 2010. Radio Tomographic Imaging with Wireless Networks. IEEE Transactions on Mobile Computing 9, 5 (May 2010), 621--632.
74. Andrew Wilson and Hrvoje Benko. 2010. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST '10). ACM, New York, NY, USA, 273--282.
75. Andrew Wilson, Hrvoje Benko, Shahram Izadi, and Otmar Hilliges. 2012. Steerable augmented reality with the beamatron. In Proceedings of the 25th annual ACM symposium on User interface software and technology (UIST '12). ACM, New York, NY, USA, 413--422.
76. Robert Xiao, Chris Harrison, and Scott E. Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 879--888.
77. Robert Xiao, Greg Lew, James Marsanico, Divya Hariharan, Scott Hudson, and Chris Harrison. 2014. Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (MobileHCI '14). ACM, New York, NY, USA, 67--76.
78. Robert Xiao, Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4000--4008.
79. Chouchang Yang, and Alanson P. Sample. "EM-ID: Tag-less identification of electrical devices via electromagnetic emissions." In RFID (RFID), 2016 IEEE International Conference on, pp. 1--8. IEEE, 2016.
80. Moustafa Youssef, Matthew Mah, and Ashok Agrawala. 2007. Challenges: device-free passive localization for wireless environments. In Proceedings of the 13th annual ACM international conference on Mobile computing and networking (MobiCom '07). ACM, New York, NY, USA, 222--229.
81. Yang Zhang, Gierad Laput, and Chris Harrison. 2017. Electrick: Low-Cost Touch Sensing Using Electric Field Tomography. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1--14.
82. Nan Zhao, Gershon Dublon, Nicholas Gillian, Artem Dementyev, and Joseph A. Paradiso. "EMI Spy: Harnessing electromagnetic interference for low-cost, rapid prototyping of proxemic interaction." In Wearable and Implantable Body Sensor Networks (BSN), 2015 IEEE 12th International Conference on, pp. 1--6. IEEE, 2015.
83. Thomas G. Zimmerman, Joshua R. Smith, Joseph A. Paradiso, David Allport, and Neil Gershenfeld. 1995. Applying electric field sensing to human-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 280--287.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)