RealPen: Providing Realism in Handwriting Tasks on Touch Surfaces using Auditory-Tactile Feedback

ABSTRACT

We present RealPen, an augmented stylus for capacitive tablet screens that recreates the physical sensation of writing on paper with a pencil, ball-point pen or marker pen. The aim is to create a more engaging experience when writing on touch surfaces, such as screens of tablet computers. This is achieved by regenerating the friction-induced oscillation and sound of a real writing tool in contact with paper. To generate realistic tactile feedback, our algorithm analyzes the frequency spectrum of the friction oscillation generated when writing with traditional tools, extracts principal frequencies, and uses the actuator's frequency response profile for an adjustment weighting function. We enhance the realism by providing the sound feedback aligned with the writing pressure and speed. Furthermore, we investigated the effects of superposition and fluctuation of several frequencies on human tactile perception, evaluated the performance of RealPen, and characterized users' perception and preference of each feedback type.

References

1. Akay, A. 2002. Acoustics of friction. The Journal of the Acoustical Society of America. 111, 4, 1525--1548.
2. Andersen, T.H. and Zhai, S. 2008. Writing with Music: Exploring the Use of Auditory Feedback in Gesture Interfaces. ACM Trans. Appl. Percept. 7, 3, 17:1--17:24.
3. Arasan, A., Basdogan, C. and Sezgin, T.M. 201 Haptic stylus with inertial and vibro-tactile feedback. Proc. of WHC '13. IEEE, 425--430.
4. Bau, O., Poupyrev, I., Israr, A. and Harrison, C. 2010. TeslaTouch: Electrovibration for Touch Surfaces. Proc. of UIST '10. ACM, 283--292.
5. Baxter, B., Scheib, V., Lin, M.C. and Manocha, D. 2001. DAB: Interactive Haptic Painting with 3D Virtual Brushes. Proc. of SIGGRAPH '01. ACM, 461--468.
6. Biet, M., Giraud, F. and Lemaire-Semail, B. 2007. Squeeze film effect for the design of an ultrasonic tactile plate. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control. 54, 12, 2678--2688.
7. Brink, A.A., Smit, J., Bulacu, M.L. and Schomaker, L.R.B. 2012. Writer identification using directional ink-trace width measurements. Pattern Recognition. 45, 1, 162--171.
8. Chang, A. and O'Sullivan, C. 2005. Audio-haptic Feedback in Mobile Phones. Proc. of CHI EA '05. ACM, 1264--1267.
9. Chen, D., Song, A. and Tian, L. 2015. A novel miniature multi-mode haptic pen for image interaction on mobile terminal. Proc. of HAVE '15. IEEE, 1--6.
10. Chigira, Y., Fujii, Y. and Valera, J.D.R. 2004. Direct measurement of friction acting between a ballpoint pen and a paper. Proc. of SICE '04. IEEE, 1518--1521, vol. 2.
11. Cho, Y., Kim, S., Joung, M. and Lee, J. Haptic Cushion: Automatic Generation of Vibro- tactile Feedback Based on Audio Signal for Immersive Interaction with Multimedia. Proc. of ACTUATOR '14. 427--430.
12. Danna, J. and Velay, J.-L. 2015. Basic and supplementary sensory feedback in handwriting. Frontiers in Psychology. 6.
13. DiFilippo, D. and Pai, D.K. 2000. The AHI: An Audio and Haptic Interface for Contact Interactions. Proc. of UIST '00. ACM, 149--158.
14. Dooijes, E.H. 1983. Analysis of handwriting movements. Acta Psychologica. 54, 1, 99--1
15. Fagiani, R., Massi, F., Chatelet, E., Berthier, Y. and Akay, A. 2011. Tactile perception by friction induced vibrations. Tribology International. 44, 10, 1100--1110.
16. Feder, K.P. and Majnemer, A. 2007. Handwriting development, competency, and intervention. Developmental Medicine & Child Neurology. 49, 4, 312--317.
17. Franke, K. and Rose, S. 2004. Ink-deposition model: the relation of writing and ink deposition processes. Proc. of IWFHR '04. IEEE, 173--178.
18. French, A.S., Holden, A.V. and Stein, R.B. 1972. The estimation of the frequency response function of a mechanoreceptor. Kybernetik. 11, 1, 15--23.
19. Harrison, C., Xiao, R. and Hudson, S. 2012. Acoustic Barcodes: Passive, Durable and Inexpensive Notched Identification Tags. Proc. of UIST '12. ACM, 563--568.
20. Hayward, V., Astley, O.R., Cruz-Hernandez, M., Grant, D. and Robles-De-La-Torre, G. 2004. Haptic interfaces and devices. Sensor Review. 24, 1, 16--29.
21. Hinckley, K., Pahud, M., Benko, H., Irani, P., Guimbretière, F., Gavriliu, M., Chen, X., Matulic, F., Buxton, W. and Wilson, A. 2014. Sensing Techniques for Tablet+Stylus Interaction. Proc. of UIST '14. ACM, 605--614.
22. Hinckley, K., Yatani, K., Pahud, M., Coddington, N., Rodenhouse, J., Wilson, A., Benko, H. and Buxton, B. 2010. Pen + Touch = New Tools. Proc. of UIST '10. ACM, 27--36.
23. Hoggan, E., Raisamo, R. and Brewster, S.A. 2009. Mapping Information to Audio and Tactile Icons. Proc. of ICMI-MLMI '09. ACM, 327--334.
24. Hollerbach, J.M. 1981. An Oscillation Theory of Handwriting. Biol. Cybern. 39, 139--156.
25. Ikei, Y., Yamada, M. and Fukuda, S. 2001. A new design of haptic texture display - Texture Display2 - and its preliminary evaluation. Proc. of VR '01. IEEE, 21--28.
26. Kamuro, S., Minamizawa, K., Kawakami, N. and Tachi, S. 2009. Ungrounded kinesthetic pen for haptic interaction with virtual environments. Proc. of RO-MAN '09. IEEE, 436--441.
27. Kwon, D.-S., Yang, T.-H. and Cho, Y.-J. 2010. Mechatronics technology in mobile devices. Industrial Electronics Magazine, IEEE. 4, 2, 36--41.
28. Lee, J.C., Dietz, P.H., Leigh, D., Yerazunis, W.S. and Hudson, S.E. 2004. Haptic Pen: A Tactile Feedback Stylus for Touch Screens. Proc. of UIST '04. ACM, 291--294.
29. Liao, C., Guimbretière, F. and Loeckenhoff, C.E. 2006. Pen-top Feedback for Paper-based Interfaces. Proc. of UIST '06. ACM, 201--210.
30. McMahan, W., Romano, J.M., Abdul Rahuman, A.M. and Kuchenbecker, K.J. 2010. High frequency acceleration feedback significantly increases the realism of haptically rendered textured surfaces. Proc. of Haptics Symposium '10. IEEE, 141--148.
31. 3Murakami, K., Tsuruno, R. and Genda, E. 2005. Multiple illuminated paper textures for drawing strokes. Proc. of Computer Graphics International '05. IEEE, 156--161.
32. Pielot, M., Church, K. and de Oliveira, R. 2014. An In-situ Study of Mobile Phone Notifications. Proc. of MobileHCI '14. ACM, 233--242.
33. Pitts, M.J., Williams, M.A., Wellings, T. and Attridge, A. 2009. Assessing Subjective Response to Haptic Feedback in Automotive Touchscreens. Proc. of AutomotiveUI '09. ACM, 11--18.
34. Rocchesso, D., Delle Monache, S. and Papetti, S. 2016. Multisensory texture exploration at the tip of the pen. International Journal of Human-Computer Studies. 85, 47--56.
35. Romano, J.M. and Kuchenbecker, K.J. 2012. Creating Realistic Virtual Textures from Contact Acceleration Data. IEEE Transactions on Haptics. 5, 2, 109--119.
36. Rosen, S. and Howell, P. 2010. Signals and Systems for Speech and Hearing, second edition. Emerald Press.
37. Schomaker, L.R.B. and Plamondon, R. 1990. The relation between pen force and pen-point kinematics in handwriting. Biological Cybernetics. 63, 4, 277--289.
38. Sharmin, S., Evreinov, G. and Raisamo, R. 2005. Non-visual feedback cues for pen computing. Proc. of EuroHaptics '05. IEEE, 625--628.
39. Tajadura-Jiménez, A., Tsakiris, M., Marquardt, T. and Bianchi-Berthouze, N. 2015. Action sounds update the mental representation of arm dimension: contributions of kinaesthesia and agency. Frontiers in Psychology. 6.
40. Verrillo, R.T. 1965. Temporal Summation in Vibrotactile Sensitivity. The Journal of the Acoustical Society of America. 37, 5, 843--846.
41. Vitense, H.S., Jacko, J.A. and Emery, V.K. 2003. Multimodal feedback: an assessment of performance and mental workload. Ergonomics. 46, 1--3, 68--87.
42. Winfield, L., Glassmire, J., Colgate, J.E. and Peshkin, M. 2007. T-PaD: Tactile Pattern Display through Variable Friction Reduction. Proc. of EuroHaptics '07. IEEE, 421--426.
43. Wintergerst, G., Jagodzinski, R., Hemmert, F., Müller, A. and Joost, G. 2010. Reflective Haptics: Enhancing Stylus-based Interactions on Touch Screens. Proc. of EuroHaptics'10. Springer-Verlag, 360--366.
44. Withana, A., Kondo, M., Makino, Y., Kakehi, G., Sugimoto, M. and Inami, M. 2010. ImpAct: Immersive Haptic Stylus to Enable Direct Touch and Manipulation for Surface Computing. Comput. Entertain. 8, 2, 9:1--9:16.
45. Zwicker, E. 1961. Subdivision of the Audible Frequency Range into Critical Bands (Frequenzgruppen). The Journal of the Acoustical Society of America. 33, 2, 248--248.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)