FlexSense: a transparent self-sensing deformable surface

ABSTRACT

We present FlexSense, a new thin-film, transparent sensing surface based on printed piezoelectric sensors, which can reconstruct complex deformations without the need for any external sensing, such as cameras. FlexSense provides a fully self-contained setup which improves mobility and is not affected from occlusions. Using only a sparse set of sensors, printed on the periphery of the surface substrate, we devise two new algorithms to fully reconstruct the complex deformations of the sheet, using only these sparse sensor measurements. An evaluation shows that both proposed algorithms are capable of reconstructing complex deformations accurately. We demonstrate how FlexSense can be used for a variety of 2.5D interactions, including as a transparent cover for tablets where bending can be performed alongside touch to enable magic lens style effects, layered input, and mode switching, as well as the ability to use our device as a high degree-of-freedom input controller for gaming and beyond.

References

1. Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G., and Singh, K. Exploring Interactive Curve and Surface Manipulation Using a Bend and Twist Sensitive Input Strip. In I3D'99, ACM, 1999, 111--118.
2. Caglioti, V., Giusti, A., Mureddu, L., and Taddei, P. A Manipulable Vision-Based 3D Input Device for Space Curves. In Articulated Motion and Deformable Objects. Springer, 2008, 309--318.
3. Danisch, L. A., Englehart, K., and Trivett, A. Spatially continuous six-degrees-of-freedom position and orientation sensor. In Photonics East, International Society for Optics and Photonics, 1999, 48--56.
4. Evgeniou, T., Pontil, M., and Poggio, T. Regularization Networks and Support Vector Machines. In Advances in Computational Mathematics, 2000.
5. Follmer, S., Leithinger, D., Olwal, A., Cheng, N., and Ishii, H. Jamming User Interfaces: Programmable Particle Stiffness and Sensing for Malleable and Shape-changing Devices. In UIST'12, ACM, 2012, 519--528.
6. Gallant, D. T., Seniuk, A. G., and Vertegaal, R. Towards More Paper-like Input: Flexible Input Devices for Foldable Interaction Styles. In UIST'08, ACM, Oct. 2008, 283.
7. Gomes, A., Nesbitt, A., and Vertegaal, R. MorePhone: A Study of Actuated Shape Deformations for Flexible Thin-Film Smartphone Notifications. In CHI'13, ACM, Apr. 2013, 583.
8. Herkenrath, G., Karrer, T., and Borchers, J. TWEND: Twisting and Bending as new Interaction Gesture in Mobile Devices. In CHI'08 EA, ACM, Apr. 2008, 3819.
9. Holman, D., Vertegaal, R., Altosaar, M., Troje, N., and Johns, D. PaperWindows: Interaction Techniques for Digital Paper. In CHI'05, ACM, 2005, 591--599.
10. Kabsch, W. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica (1976).
11. Kato, H., and Billinghurst, M. Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. In IWAR'99, IEEE Computer Society, 1999.
12. Kato, T., Yamamoto, A., and Higuchi, T. Shape recognition using piezoelectric thin films. In IEEE Industrial Technology, vol. 1, IEEE, 2003, 112--116.
13. Khalilbeigi, M., Lissermann, R., Kleine, W., and Steimle, J. Foldme: Interacting with double-sided foldable displays. In TEI'12, ACM, 2012, 33--40.
14. Khalilbeigi, M., Lissermann, R., Muhlhauser, M., and Steimle, J. Xpaaand: Interaction Techniques for Rollable Displays. In CHI'11, ACM, 2011, 2729--2732.
15. Kildal, J., Paasovaara, S., and Aaltonen, V. Kinetic Device: Designing Interactions with a Deformable Mobile Interface. In CHI EA'12, May 2012.
16. Konieczny, J., Shimizu, C., Meyer, G., and Colucci, D. A Handheld Flexible Display System, 2005.
17. Lahey, B., Girouard, A., Burleson, W., and Vertegaal, R. PaperPhone: Understanding the Use of Bend Gestures in Mobile Devices with Flexible Electronic Paper Displays. In CHI'11, ACM, May 2011, 1303.
18. Leal, A., Bowman, D., Schaefer, L., Quek, F., and Stiles, C. K. 3D Sketching Using Interactive Fabric for Tangible and Bimanual Input. In GI'11, Canadian Human-Computer Communications Society, 2011, 49--56.
19. Lee, J. C., Hudson, S. E., and Tse, E. Foldable interactive displays. In UIST'08, ACM, 2008, 287--290.
20. Lee, S.-S. et al. How Users Manipulate Deformable Displays as Input Devices. In CHI'10, ACM, Apr. 2010, 1647.
21. Lee, S.-S. et al. FlexRemote: Exploring the effectiveness of deformable user interface as an input device for TV. In HCI International 2011--Posters' Extended Abstracts. Springer, 2011, 62--65.
22. Punpongsanon, P., Iwai, D., and Sato, K. DeforMe: Projection-based Visualization of Deformable Surfaces Using Invisible Textures. In ETech SA'13, ACM, 2013.
23. Rendl, C. et al. PyzoFlex: Printed Piezoelectric Pressure Sensing Foil. In UIST'12, ACM, 2012.
24. Rosenberg, I., and Perlin, K. The UnMousePad: An Interpolating Multi-touch Force-sensing Input Pad. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 2009, 65.
25. Roudaut, A., Karnik, A., Lochtefeld, M., and Subramanian, S. Morphees: Toward High "Shape Resolution" in Self-Actuated Flexible Mobile Devices. In CHI'13, ACM, Apr. 2013, 593.
26. Sato, T., Mamiya, H., Koike, H., and Fukuchi, K. PhotoelasticTouch: Transparent Rubbery Tangible Interface Using an LCD and Photoelasticity. In UIST'09, ACM, 2009, 43--50.
27. Schölkopf, B., Herbrich, R., and Smola, A. A Generalized Representer Theorem. In Conference on Computational Learning Theory, 2001.
28. Schwesig, C., Poupyrev, I., and Mori, E. Gummi: A Bendable Computer. In CHI'04, ACM, Apr. 2004, 263--270.
29. Smith, R. T., Thomas, B. H., and Piekarski, W. Digital Foam Interaction Techniques for 3D Modeling. In VRST'08, ACM, 2008, 61--68.
30. Sorkine, O., and Alexa, M. As-rigid-as-possible surface modeling. In SGP'07, 2007.
31. Steimle, J., Jordt, A., and Maes, P. Flexpad: Highly Flexible Bending Interactions for Projected Handheld Displays. In CHI'13, ACM, Apr. 2013, 237.
32. Tajika, T., Yonezawa, T., and Mitsunaga, N. Intuitive Page-turning Interface of E-books on Flexible E-paper based on User Studies. In MM'08, ACM, Oct. 2008, 793.
33. Tarun, A. P. et al. PaperTab: An Electronic Paper Computer with Multiple Large Flexible Electrophoretic Displays. In CHI EA'13, ACM, 2013, 3131--3134.
34. Taylor, J. et al. User-specific hand modeling from monocular depth sequences. In CVPR, 2014.
35. Tikhonov, A., Leonov, A., and A.G., Y. Nonlinear Ill-Posed Problems. In Kluwer Academic Publishers, 1998.
36. Warren, K., Lo, J., Vadgama, V., and Girouard, A. Bending the Rules: Bend Gesture Classification for Flexible Displays. In CHI'13, ACM, Apr. 2013, 607.
37. Watanabe, J., Mochizuki, A., and Horry, Y. Bookisheet: Bendable Device for Browsing Content Using the Metaphor of Leafing Through the Pages. In UbiComp'08, ACM, Sept. 2008, 360.
38. Wellner, P. Interacting with paper on the DigitalDesk. Communications of the ACM 36, 7 (1993), 87--96.
39. Ye, Z., and Khalid, H. Cobra: Flexible Displays for Mobile Gaming Scenarios. In CHI EA'10, ACM, 2010, 4363--4368.
40. Zimmerman, T. G., Lanier, J., Blanchard, C., Bryson, S., and Harvill, Y. A Hand Gesture Interface Device. In CHI'87, ACM, 1987, 189--192.
41. Zirkl, M. et al. An All-Printed Ferroelectric Active Matrix Sensor Network Based on Only Five Functional Materials Forming a Touchless Control Interface. Advanced Materials, Volume 23, Issue 18 (2011), 2069--2074.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)