SurfaceFusion: unobtrusive tracking of everyday objects in tangible user interfaces

ABSTRACT

Interactive surfaces and related tangible user interfaces often involve everyday objects that are identified, tracked, and augmented with digital information. Traditional approaches for recognizing these objects typically rely on complex pattern recognition techniques, or the addition of active electronics or fiducials that alter the visual qualities of those objects, making them less practical for real-world use. Radio Frequency Identification (RFID) technology provides an unobtrusive method of sensing the presence of and identifying tagged nearby objects but has no inherent means of determining the position of tagged objects. Computer vision, on the other hand, is an established approach to track objects with a camera. While shapes and movement on an interactive surface can be determined from classic image processing techniques, object recognition tends to be complex, computationally expensive and sensitive to environmental conditions. We present a set of techniques in which movement and shape information from the computer vision system is fused with RFID events that identify what objects are in the image. By synchronizing these two complementary sensing modalities, we can associate changes in the image with events in the RFID data, in order to recover position, shape and identification of the objects on the surface, while avoiding complex computer vision processes and exotic RFID solutions.

References

1. Boukraa, M. and Ando, S. Tag-based vision: assisting 3D scene analysis with radio-frequency tags. Image Processing 2002 (2002), I-269--I-2722.
2. Dietz, P. and Leigh, D. DiamondTouch: a multi-user touch technology. UIST '01 (2001), 219--226.
3. Fishkin, K., Jiang, b., Philipose, M. and Roy, S. I Sense a Disturbance in the Force: Unobtrusive Detection of Interactions with RFID-tagged Objects. IRS-TR-04-013. (2004).
4. Gonzalez, R. C., and Woods, G. Digital Image Processing. Addison Wesley (1993).
5. Ishii, H. and Ullmer, B. Tangible bits: towards seamless interfaces between people, bits and atoms. CHI '97 (1997), 234--241.
6. Krahnstoever, N., Rittscher, J., Tu, P., Chean, K. and Tomlinson, T. Activity Recognition using Visual Tracking and RFID. WACV/MOTIONS '05 (2005), 494--500.
7. Microsoft Surface. http://www.microsoft.com/surface/. (Sep 2007).
8. Olwal, A. LightSense: Enabling Spatially Aware Handheld Interaction Devices. ISMAR '06 (2006), 119--122.
9. Patten, J., Ishii, H., Hines, J., and Pangaro, G. Sensetable: a wireless object tracking platform for tangible user interfaces. CHI '01 (2001), 253--260.
10. Philips Entertaible. http://www.research.philips.com/initiatives/entertaible/. (Sep 2007).
11. Rahimi, A, and Recht, B. Estimating Observation Functions in Dynamical Systems using Unsupervised Regression. NIPS (2006).
12. Raskar, R., Beardsley, P., Dietz, P., and van Baar, J. Photosensing wireless tags for geometric procedures. Commun. ACM 48, 9 (2005), 46--51.
13. Reilly, D., Rodgers, M., Argue, R., Nunes, M., and Inkpen, K. Marked-up maps: combining paper maps and electronic information resources. Personal Ubiquitous Comput. 10, 4 (2006), 215--226.
14. Rekimoto, J. and Saitoh, M. Augmented surfaces: a spatially continuous work space for hybrid computing environments. CHI '99 (1999), 378--385.
15. Rekimoto, J., Ullmer, B., and Oba, H. DataTiles: a modular platform for mixed physical and graphical interactions. CHI '01 (2001), 269--276.
16. Sugimoto, M., Kusunoki, F,. and Hashizume, H. Supporting Face-toface Group Activities with a Sensor-Embedded Board. CSCW Workshop on Shared Environments to Support Face-to-Face Collaboration (2000).
17. Ullmer, B., Ishii, H., and Jacob, R. Tangible Query Interfaces: Physically Constrained Tokens for Manipulating Database Queries. INTERACT '03 (2003), 279--286.
18. Ullmer, B. and Ishii, H. 1997. The metaDESK: models and prototypes for tangible user interfaces. UIST '97. (1997) 223--232.
19. Ullmer, B., Ishii, H., and Glas, D. mediaBlocks: physical containers, transports, and controls for online media. SIGGRAPH '98 (1998), 379--386.
20. Want, R., Fishkin, K. P., Gujar, A., and Harrison, B. L. Bridging physical and virtual worlds with electronic tags. CHI '99 (1999), 370--377.
21. Wellner, P. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (1993), 87--96.
22. Wilson, A. D. and Sarin, R. BlueTable: connecting wireless mobile devices on interactive surfaces using vision-based handshaking. Graphics Interface 2007 (2007), 119--125.
23. Wilson, A. D. PlayAnywhere: a compact interactive tabletop projection-vision system. UIST '05. (2005), 83--92.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Graphical user interfaces