Seeing double: reconstructing obscured typed input from repeated compromising reflections

ABSTRACT

Of late, threats enabled by the ubiquitous use of mobile devices have drawn much interest from the research community. However, prior threats all suffer from a similar, and profound, weakness - namely the requirement that the adversary is either within visual range of the victim (e.g., to ensure that the pop-out events in reflections in the victim's sunglasses can be discerned) or is close enough to the target to avoid the use of expensive telescopes. In this paper, we broaden the scope of the attacks by relaxing these requirements and show that breaches of privacy are possible even when the adversary is around a corner. The approach we take overcomes challenges posed by low image resolution by extending computer vision methods to operate on small, high-noise, images. Moreover, our work is applicable to all types of keyboards because of a novel application of fingertip motion analysis for key-press detection. In doing so, we are also able to exploit reflections in the eyeball of the user or even repeated reflections (i.e., a reflection of a reflection of the mobile device in the eyeball of the user). Our empirical results show that we can perform these attacks with high accuracy, and can do so in scenarios that aptly demonstrate the realism of this threat.

References

1. D. Asonov and R. Agrawal. Keyboard acoustic emanations. In Proceedings of the IEEE Symposium on Security and Privacy, 2004.
2. M. Backes, M. Durmuth, and D. Unruh. Compromising reflections-or-how to read LCD monitors around the corner. In Proceedings of the IEEE Symposium on Security and Privacy, 2008.
3. M. Backes, T. Chen, M. Dürmuth, H. Lensch, and M. Welk. Tempest in a teapot: Compromising reflections revisited. In Proceedings of the IEEE Symposium on Security and Privacy, 2009.
4. S. Baker and I. Matthews. Lucas-Kanade 20 years on. International Journal of Computer Vision, 56(3):221--255, 2004.
5. D. Balzarotti, M. Cova, and G. Vigna. ClearShot: Eavesdropping on keyboard input from video. In Proceedings of the IEEE Symposium on Security and Privacy, 2008.
6. L. Cai and H. Chen. Touchlogger: inferring keystrokes on touch screen from smartphone motion. In USENIX Workshop on Hot Topics in Security (HotSec), 2011.
7. L. Cai and H. Chen. On the practicality of motion based keystroke inference attack. Trust and Trustworthy Computing, pages 273--290, 2012.
8. A. Chaudhary, J. Raheja, and S. Raheja. A vision based geometrical method to find fingers positions in real time hand gesture recognition. Journal of Software, 7(4): 861--869, 2012.
9. T. Chen and M.-Y. Kan. Creating a live, public short message service corpus: The NUS SMS corpus. Language Resources and Evaluation, 2011.
10. R. Collins and R. Weiss. Vanishing point calculation as a statistical inference on the unit sphere. In Proceedings of the Third International Conference on Computer Vision, 1990.
11. R. O. Duda and P. E. Hart. Use of the Hough transformation to detect lines and curves in pictures. Communications of the ACM, 15(1):11--15, 1972.
12. F. Elibol, U. Sarac, and I. Erer. Realistic eavesdropping attacks on computer displays with low-cost and mobile receiver system. In Proceedings of the 20th European Signal Processing Conference, 2012.
13. M. Fischler and R. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381--395, 1981.
14. W. N. Francis and H. Kucera. Brown corpus manual. Technical report, Dept. of Linguistics, Brown University, 1979.
15. H. Grabner, M. Grabner, and H. Bischof. Real-time tracking via on-line boosting. In Proceedings of the British Machine Vision Conference, volume 1, pages 47--56, 2006.
16. H. J. Highland. Electromagnetic radiation revisited. Computer Security, 5:85--93, June 1986.
17. B. Hoanca and K. J. Mock. Password entry scheme resistant to eavesdropping. In Security and Management, 2008.
18. X. Iturbe, A. Altuna, A. Ruiz de Olano, and I. Martinez. VHDL described finger tracking system for real-time human-machine interaction. In International Conference on Signals and Electronic Systems, 2008.
19. L. Jin, D. Yang, L. Zhen, and J. Huang. A novel vision-based finger-writing character recognition system. Journal of Circuits, Systems, and Computers, 16(03):421--436, 2007.
20. C. Kerdvibulvech and H. Saito. Vision-based detection of guitar players? fingertips without markers. In Computer Graphics, Imaging and Visualisation, 2007.
21. M. Kuhn and C. Kuhn. Compromising emanations: eavesdropping risks of computer displays. Technical report, University of Cambridge, 2003.
22. M. Kumar, T. Garfinkel, D. Boneh, and T. Winograd. Reducing shoulder-surfing by using gaze-based password entry. In Symposium on Usable Privacy and Security, 2007.
23. A. Lavie. Evaluating the output of machine translation systems. AMTA Tutorial, 2010.
24. A. Lavie and M. J. Denkowski. The METEOR metric for automatic evaluation of machine translation. Machine Translation, 23(2--3):105--115, 2009.
25. B. Lee and J. Chun. Manipulation of virtual objects in marker-less AR system by fingertip tracking and hand gesture recognition. In Proceedings of the 2nd International Conference on Interaction Sciences, 2009.
26. T. Lee and T. Hollerer. Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. In IEEE International Symposium on Wearable Computers, 2007.
27. E. Lutton, H. Maitre, and J. Lopez-Krahe. Contribution to the determination of vanishing points using Hough transform. Transactions on Pattern Analysis and Machine Intelligence, 16(4):430--438, 1994.
28. M. Magee and J. Aggarwal. Determining vanishing points from perspective images. Computer Vision, Graphics, and Image Processing, 26(2):256--267, 1984.
29. F. Maggi, A. Volpatto, S. Gasparini, G. Boracchi, and S. Zanero. A fast eavesdropping attack against touchscreens. In Information Assurance and Security (IAS). IEEE, 2011.
30. J. Nakamura. Image sensors and signal processing for digital still cameras. CRC, 2005.
31. D. Nguyen, T. Pham, and J. Jeon. Fingertip detection with morphology and geometric calculation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2009.
32. K. Oka, Y. Sato, and H. Koike. Real-time fingertip tracking and gesture recognition. Computer Graphics and Applications, 22(6):64--71, 2002.
33. E. Owusu, J. Han, S. Das, A. Perrig, and J. Zhang. Accessory: password inference using accelerometers on smartphones. In Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications. ACM, 2012.
34. R. Raguram, A. White, D. Goswami, F. Monrose, and J. Frahm. iSpy: automatic reconstruction of typed input from compromising reflections. In Proceedings of the ACM Conference on Computer and Communications Security, 2011.
35. R. Raguram, A. M. White, Y. Xu, J.-M. Frahm, P. Georgel, and F. Monrose. On the privacy risks of virtual keyboards: automatic reconstruction of typed input from compromising reflections. IEEE Transactions on Dependable and Secure Computing, 2013.
36. L. Sobrado and J.-C. Birget. Graphical passwords. The Rutgers Scholar, 4, 2002.
37. E. Stelzer. Contrast, resolution, pixelation, dynamic range and signal-to-noise ratio: fundamental limits to resolution in fluorescence light microscopy. Journal of Microscopy, 189 (1):15--24, 1998.
38. R. Szeliski. Image alignment and stitching: A tutorial. Foundations and Trends in Computer Graphics and Vision, 2006.
39. D. S. Tan, P. Keyani, and M. Czerwinski. Spy-resistant keyboard: More secure password entry on public touch screen displays. In Proceedings of the 17th Australia Conference on Computer-Human Interaction, 2005.
40. N. Ukita and M. Kidode. Wearable virtual tablet: fingertip drawing on a portable plane-object using an active-infrared camera. In Proceedings of the International Conference on Intelligent User Interfaces. ACM, 2004.
41. W. van Eck. Electromagnetic radiation from video display units: an eavesdropping risk Computer Security, 4: 269--286, December 1985.
42. A. Vedaldi and B. Fulkerson. VLFeat: An open and portable library of computer vision algorithms. In Proceedings of the International Conference on Multimedia, 2010.
43. M. Vuagnoux and S. Pasini. Compromising electromagnetic emanations of wired and wireless keyboards. In Proceedings of the 18th USENIX Security Symposium, 2009.
44. J. Weaver, K. J. Mock, and B. Hoanca. Gaze-based password authentication through automatic clustering of gaze points. In IEEE International Conference on Systems, Man and Cybernetics, 2011.
45. J. Weickert. Anisotropic diffusion in image processing, volume 1. Teubner Stuttgart, 1998.
46. D. Yang, L. Jin, and J. Yin. An effective robust fingertip detection method for finger writing character recognition system. In Proceedings of the International Conference on Machine Learning and Cybernetics, 2005.
47. Y. Zhang, P. Xia, J. Luo, Z. Ling, B. Liu, and X. Fu. Fingerprint attack against touch-enabled devices. In Security and Privacy in Smartphones and Mobile Devices, SPSM '12, 2012.
48. Z. Zhang. Vision-based interaction with fingers and papers. In Proceedings International Symposium on the CREST Digital Archiving Project, 2003.
49. L. Zhuang, F. Zhou, and J. Tygar. Keyboard acoustic emanations revisited. In Proceedings of the 12th ACM Conference on Computer and Communications Security, 2005.

Index Terms

• Security and privacy

  • Human and societal aspects of security and privacy

• Social and professional topics

  • Computing / technology policy

    • Privacy policies