Designing Implicit Interfaces for Physiological Computing: Guidelines and Lessons Learned Using fNIRS

Abstract

A growing body of recent work has shown the feasibility of brain and body sensors as input to interactive systems. However, the interaction techniques and design decisions for their effective use are not well defined. We present a conceptual framework for considering implicit input from the brain, along with design principles and patterns we have developed from our work. We also describe a series of controlled, offline studies that lay the foundation for our work with functional near-infrared spectroscopy (fNIRS) neuroimaging, as well as our real-time platform that serves as a testbed for exploring brain-based adaptive interaction techniques. Finally, we present case studies illustrating the principles and patterns for effective use of brain data in human--computer interaction. We focus on signals coming from the brain, but these principles apply broadly to other sensor data and in domains such as aviation, education, medicine, driving, and anything involving multitasking or varying cognitive workload.

References

1. D. Afergan. 2014. Using brain-computer interfaces for implicit input. Adjunct Proc. ACM UIST 2014, ACM Press.
2. D. Afergan, E. M. Peck, R. Chang, and R. J. K. Jacob. 2013. Using passive input to adapt visualization systems to the individual. In Proceedings of the CHI 2013 Workshop: Many People, Many Eyes: Aggregating Influences of Visual Perception on User Interface Design.
3. D. Afergan, E. M. Peck, E. T. Solovey, A. Jenkins, S. W. Hincks, E. T. Brown, R. Chang, and R. J. K. Jacob. 2014a. Dynamic difficulty using brain metrics of workload. In Proceedings of the ACM CHI 2014. ACM.
4. D. Afergan, T. Shibata, S. W. Hincks, E. M. Peck, B. F. Yuksel, R. Chang, and R. J. K. Jacob. 2014b. Brain-based target expansion. In Proceedings of the ACM UIST 2014. ACM.
5. H. Ayaz, M. Izzetoglu, S. Bunce, T. Heiman-Patterson, and B. Onaral. 2007. Detecting cognitive activity related hemodynamic signal for brain computer interface using functional near infrared spectroscopy. In Proceedings of the IEEE/EMBS Conference on Neural Engineering. 342--345.
6. H. Ayaz, B. Onaral, K. Izzetoglu, P. A. Shewokis, R. McKendrick, and R. Parasuraman. 2013. Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: Empirical examples and a technological development. Frontiers in Human Neuroscience 7, 1--13.
7. H. Ayaz, A. Shewokis, S. Bunce, K. Izzetoglu, B. Willems, and B. Onaral. 2012. Optical brain monitoring for operator training and mental workload assessment. Neuroimage 59, 36--47.
8. H. Ayaz, B. Willems, B. Bunce, P. A. Shewokis, K. Izzetoglu, S. Hah, A. R. Deshmukh, and B. Onaral. 2010. Cognitive workload assessment of air traffic controllers using optical brain imaging sensors. Advances in Understanding Human Performance: Neuroergonomics, Human Factors Design and Special Populations, T. Marke, W. Karwowski, and V. Rice (Eds.). CRC Press Taylor & Francis, 21--31.
9. A. Baddeley. 2003. Working memory: Looking back and looking forward. Nature Reviews Neuroscience 4(10), 829--839.
10. S. C. Bunce, K. Izzetoglu, H. Ayaz, P. Shewokis, M. Izzetoglu, K. Pourrezaei, and B. Onaral. 2011. Implementation of fNIRS for monitoring levels of expertise and mental workload. In Foundations of Augmented Cognition. Directing the Future of Adaptive Systems, Springer, Berlin, 13--22.
11. B. Chance, E. Anday, S. Nioka, S. Zhou, L. Hong, K. Worden, C. Li, T. Murray, Y. Ovetsky, D. Pidikiti, and R. Thomas. A novel method for fast imaging of brain function, non-invasively, with light. Optics Express 10, 2 (1988), 411--423.
12. X. Cui, S. Bray, and A. L. Reiss. 2010. Speeded near infrared spectroscopy (NIRS) response detection. PLOS ONE 5, 11.
13. E. Cutrell and D. Tan. 2008. BCI for passive input in HCI. In Proceedings of the ACM CHI 2008. ACM.
14. D. T. Delpy, M. Cope, P. van der Zee, S. Arridge, S. Wray, and J. Wyatt. 1988. Estimation of optical path length through tissue from direct time of flight measurements. Physics in Medicine and Biology 33 1433--1442.
15. G. Derosiére, K. Mandrick, G. Dray, T. E. Ward, and S. Perrey. 2013. NIRS-measured prefrontal cortex activity in neuroergonomics: Strengths and weaknesses. Frontiers in Human Neuroscience 7, 583.
16. A. C. Ehlis, C. G. Bähne, C. P. Jacob, M. J. Herrmann, and A. J. Fallgatter. 2008. Reduced lateral prefrontal activation in adult patients with attention-deficit/hyperactivity disorder (ADHD) during a working memory task: A functional near-infrared spectroscopy (fNIRS) study. Journal of Psychiatric Research 42, 13, 1060--1067.
17. S. H. Fairclough. 2009. Fundamentals of physiological computing. Interacting with Computers 21, 1--2, 133--145.
18. F. A. Fishburn, M. E. Norr, A. V. Medvedev, and C. J. Vaidya. 2014. Sensitivity of fNIRS to cognitive state and load. Frontiers in Human Neuroscience 8, 76.
19. J. Fogarty, S. E. Hudson, and J. Lai. 2004. Examining the robustness of sensor-based statistical models of human interruptibility. In Proceedings of the ACM CHI 2004. ACM.
20. J. D. Foley and A. Van Dam. 1982. Fundamentals of Interactive Computer Graphics. Addison-Wesley Systems Programming Series, Reading, MA.
21. L. Gagnon, R. J. Cooper, M. A. Yücel, K. L. Perdue, D. N. Greve, and D. A. Boas. 2012. Short separation channel location impacts the performance of short channel regression in NIRS. NeuroImage 59, 3, 2518--2528.
22. A. S. Gevins and B. C. Cutillo. 1993. Neuroelectric evidence for distributed processing in human working memory. Electroencephalography and Clinical Neurophysiology 87, 128--143.
23. A. Girouard, E. T. Solovey, L. M. Hirshfield, K. Chauncey, A. Sassaroli, S. Fantini, and R. J. K. Jacob. 2009. Distinguishing difficulty levels with non-invasive brain activity measurements. In INTERACT 2009.
24. D. Grimes, D. S. Tan, S. Hudson, P. Shenoy, and R. Rao. 2008. Feasibility and pragmatics of classifying working memory load with an electroencephalograph. In Proceedings of the ACM CHI 2008. ACM.
25. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. 2009. The WEKA data mining software: an update. ACM SIGKDD Explorations Newsletter 11, 1, 10--18
26. C. Herff, D. Heger, O. Fortmann, J. Hennrich, F. Putze, and T. Schultz. 2013. Mental workload during n-back task-quantified in the prefrontal cortex using fNIRS. Frontiers in Human Neuroscience 7, 935.
27. M. J. Herrmann, T. Huter, M. M. Plichta, A. C. Ehlis, G. W. Alpers, A. Mühlberger, and A. J. Fallgatter. 2008. Enhancement of activity of the primary visual cortex during processing of emotional stimuli as measured with event-related functional near-infrared spectroscopy and event-related potentials. Human Brain Mapping 29, 1, 28--35
28. L. M. Hirshfield, A. Girouard, E. T. Solovey, R. J. K. Jacob, A. Sassaroli, Y. Tong, and S. Fantini. 2007. Human-computer interaction and brain measurement using functional near-infrared spectroscopy. In Proceedings of the UIST 2007. ACM.
29. L. M. Hirshfield, E. T. Solovey, A. Girouard, J. Kebinger, R. J. K. Jacob, A. Sassaroli, and S. Fantini. 2009. Brain measurement for usability testing and adaptive interfaces: An example of uncovering syntactic workload with functional near infrared spectroscopy. In Proceedings of the ACM CHI 2009. ACM.
30. L. M. Hirshfield, K. Chauncey, R. Gulotta, A. Girouard, E. T. Solovey, R. J. K. Jacob, A. Sassaroli, and S. Fantini. 2009. Combining electroencephalograph and near infrared spectroscopy to explore users’ instantaneous and continuous mental workload states. In Proceedings of the HCII’09.
31. S. Hudson, J. Fogarty, C. Atkeson, D. Avrahami, J. Forlizzi, S. Kiesler, J. Lee, and J. Yang. 2003. Predicting human interruptibility with sensors: A Wizard of Oz feasibility study. In Proceedings of the ACM CHI 2003. ACM Press.
32. K. Izzetoglu, S. Bunce, M. Izzetoglu, B. Onaral, and K. Pourrezaei. 2003. fNIR spectroscopy as a measure of cognitive task load. Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE. IEEE 4, 3431--3434.
33. K. Izzetoglu, S. Bunce, B. Onaral, K. Pourrezaei, and B. Chance. 2004. Functional optical brain imaging using near-infrared during cognitive tasks. International Journal of Human-Computer Interaction 17, 2, 211--227.
34. M. Izzetoglu, K. Izzetoglu, S. Bunce, H. Ayaz, A. Devaraj, B. Onaral, and K. Pourrezaei. 2005. Functional near-infrared neuroimaging. IEEE Transactions on Neural Systems and Rehabilitation Engineering 13, 2, 153--159.
35. R. J. K. Jacob. 1983. Using formal specifications in the design of a human-computer interface. Communications of the ACM 26, 4, 259--264.
36. R. J. K. Jacob. 1986. A specification language for direct-manipulation user interfaces. ACM Transactions on Graphics 5, 4, 283--317.
37. R. J. K. Jacob. 2001. Open Syntax: Improving access for all users. In Proceedings of the WUAUC 2001. ACM.
38. Z. A. Keirn and J. I. Aunon. 1990. A new mode of communication between man and his surroundings. IEEE Transactions on Biomedical Engineering 37, 12, 1209.
39. E. Koechlin, G. Basso, P. Pietrini, S. Panzer, and J. Grafman. 1999. The role of the anterior prefrontal cortex in human cognition. Nature 399, 1--4.
40. E. Koechlin, G. Corrado, P. Pietrini, and J. Grafman, 2000. Dissociating the role of the medial and lateral anterior prefrontal cortex in human planning. Proceedings of the National Academy of Sciences 97, 7651--7656.
41. L. I. Kuncheva, J. J. Rodríguez, C. O. Plumpton, D. E. Linden, and S. J. Johnston. 2010. Random subspace ensembles for fMRI classification. IEEE Transactions on Medical Imaging 29, 2, 531--542.
42. J. C. Lee and D. Tan. 2006. Using a low-cost electroencephalograph for task classification in HCI research. In Proceedings of the ACM Symposium on User Interface Software and Technology 2006. ACM Press.
43. J. Leon-Carrion, J. Damas, K. Izzetoglu, K. Pourrezai, J. F. Martín-Rodríguez, J. M. Barroso y Martin, and M. R. Dominguez-Morales. 2006. Differential time course and intensity of PFC activation for men and women in response to emotional stimuli: A functional near- infrared spectroscopy (fNIRS) study. Neuroscience Letters 403, 1--2, 90--95.
44. C. Li, H. Gong, Z. Gan, and Q. Luo. 2005. Monitoring of prefrontal cortex activation during verbal n-back task with 24-channel functional NIRS imager. Proceedings of SPIE 5630, 883.
45. R. L. Mappus IV, G. R. Venkatesh, C. Shastry, A. Israeli, and M. M. Jackson. 2009. An fNIR based BMI for letter construction using continuous control. In ACM CHI 2009 Extended Abstracts, ACM, 3571--3576.
46. R. Mandryk, M. Atkins, and K. Inkpen. 2006. A continuous and objective evaluation of emotional experience with interactive play environments. In Proceedings of the ACM CHI 2006. ACM.
47. R. McKendrick, H. Ayaz, R. Olmstead, and R. Parasuraman. 2014. Enhancing dual-task performance with verbal and spatial working memory training: Continuous monitoring of cerebral hemodynamics with NIRS. Neuroimage 85, 3, 1014--1026.
48. B. Mehler, B. Reimer, and J. F. Coughlin. 2012. Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task: An on-road study across three age groups. Human Factors: The Journal of the Human Factors and Ergonomics Society 54, 3, 396--412.
49. B. Mehler, B. Reimer, J. F. Coughlin, and J. A. Dusek. 2009. Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transportation Research Record: Journal of the Transportation Research Board 2138, 2, 6--12.
50. E. Molteni, G. Baselli, A. M. Bianchi, M. Caffini, D. Contini, L. Spinelli, A. Torricelli, S. Cerutti, and R. Cubeddu. 2009. Frontal brain activation during a working memory task: a time-domain fNIRS study. Progress in Biomedical Optics and Imaging 10, 1.
51. G. Mueller-Putz, R. Scherer, C. Brunner, R. Leeb, and G. Pfurtscheller. 2008. Better than random: A closer look on BCI results. International Journal of Bioelectromagnetism 10, 52--55.
52. L. E. Nacke, M. Kalyn, C. Lough, and R. L. Mandryk. 2011. Biofeedback game design: Using direct and indirect physiological control to enhance game interaction. In Proceedings of the ACM CHI 2011. ACM.
53. I. Nouretdinov, S. G. Costafreda, A. Gammerman, A. Chervonenkis, V. Vovk, V. Vapnik, and C. H. Fu. 2011. Machine learning classification with confidence: application of transductive conformal predictors to MRI-based diagnostic and prognostic markers in depression. Neuroimage 56, 2, 809--813.
54. A. M. Owen, K. M. McMillan, A. R. Laird, and E. Bullmore. 2005. N-back working memory paradigm: A meta-analysis of normative functional neuroimaging studies. Human Brain Mapping 25, 1, 46--59.
55. E. M. Peck, B. F. Yuksel, L. Harrison, A. Ottley, and R. Chang. 2012. Towards a 3-dimensional model of individual cognitive differences. In BELIV 2012: Beyond Time and Errors: Novel Evaluation Methods for Visualization, 2012.
56. E. M. Peck, B. F. Yuksel, A. Ottley, R. J. K. Jacob, and R. Chang. 2013. Using fNIRS brain sensing to evaluate information visualization interfaces. In Proceedings of the ACM CHI 2013. ACM.
57. E. M. Peck, D. Afergan, and R. J. K. Jacob, 2013. Investigation of fNIRS brain sensing as input to information filtering systems. In Augmented Human 2013.
58. L. J. Prinzel, F. G. Freeman, M. W. Scerbo, P. J. Mikulka, and A. T. Pope. 2003. Effects of a psychophysiological system for adaptive automation on performance, workload, and the event-related potential P300 component. Human Factors: The Journal of the Human Factors and Ergonomics Society 45, 4, 601--613.
59. N. Ramnani and A. M. Owen. 2004. Anterior prefrontal cortex: insights into function from anatomy and neuroimaging. Nature Reviews Neuroscience 5, 3, 184.
60. M. Rötting, T. Zander, S. Trösterer, and J. Dzaack. 2009. Implicit interaction in multimodal human-machine systems. In Industrial Engineering and Ergonomics, Springer, Berlin, 523--536.
61. G. Schalk, D. J. McFarland, T. Hinterberger, N. Birbaumer, and J. R. Wolpaw. 2004. BCI2000: A general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering 51, 6, 1034.
62. A. Schmidt. 2000. Implicit human computer interaction through context. Personal Technologies 4, 2, 191--199.
63. P. Shenoy and D. S. Tan. 2008. Human-aided computing: utilizing implicit human processing to classify images. In Proceedings of the ACM CHI 2008. ACM.
64. T. Shimizu, S. Hirose, H. Obara, K. Yanagisawa, H. Tsunashima, Y. Marumo, and M. Taira. 2009. Measurement of frontal cortex brain activity attributable to the driving workload and increased attention. SAE International Journal of Passenger Cars-Mechanical Systems 2, 1, 736--744.
65. R. Sitaram, H. Zhang, C. Guan, M. Thulasidas, Y. Hoshi, A. Ishikawa, K. Shimizu, N. Birbaumer. 2007. Temporal classification of multichannel near-infrared spectroscopy signals of motor imagery for developing a brain-computer interface. Neuroimage 34, 4, 1416--1427.
66. E. T. Solovey, A. Girouard, K. Chauncey, L. Hirshfield, A. Sassaroli, F. Zheng, S. Fantini, and R. Jacob. 2009. Using fNIRS brain sensing in realistic HCI settings: Experiments and guidelines. In Proceedings of the ACM UIST 2009.
67. E. T. Solovey, F. Lalooses, and K. Chauncey, et al. 2011. Sensing cognitive multitasking for a brain-based adaptive user interface. In Proceedings of the ACM CHI, ACM.
68. E. T. Solovey, P. Schermerhorn, M. Scheutz, A. Sassaroli, S. Fantini, and R. J. K. Jacob. 2012. Brainput: Enhancing interactive systems with streaming fNIRS brain input. In Proceedings of the ACM CHI. ACM.
69. T. Starner, B. Schiele, and A. Pentland. 1998. Visual contextual awareness in wearable computing. In Proceedings of the ISWC.
70. B. Steichen, G. Carenini, and C. Conati. 2013. User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities. In Proceedings of the ACM IUI 2013. ACM.
71. M. St. John, D. A. Kobus, J. G. Morrison, and D. Schmorrow. 2004. Overview of the DARPA augmented cognition technical integration experiment. International Journal of Human-Computer Interaction 17, 2, 131--149.
72. S. Tak and J. Ye. 2014. Statistical analysis of fNIRS data: A comprehensive review. NeuroImage 85, 1, 72--91.
73. D. Tan and A. Nijholt. 2010. Brain-computer interfaces and human-computer interaction. In Brain-Computer Interaction: Applying our Minds to Human-Computer Interaction, D. S. Tan and A. Nijholt (Eds.). Springer, London, 3--19.
74. J. J. Thomas and K. A. Cook, eds. 2005. Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE CS Press. http://nvac.pnl.gov/agenda.stm.
75. D. Toker, C. Conati, G. Carenini, and M. Haraty. 2012. Towards adaptive information visualization: On the influence of user characteristics. In User Modeling, Adaptation, and Personalization.
76. C. T. Vi and S. Subramanian. 2012. Detecting error-related negativity for interaction design. In Proceedings of the CHI 2012. ACM.
77. A. Villringer, J. Planck, C. Hock, L. Schleinkofer, and U. Dirnagl. 1993. Near infrared spectroscopy (NIRS): A new tool to study hemodynamic changes during activation of brain function in human adults. Neuroscience Letters 154, 101--104.
78. G. F. Wilson and C. A.Russell. 2007. Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding. Human Factors: The Journal of the Human Factors and Ergonomics Society 49(6), 1005--1018.
79. M. Wolf, U. Wolf, J. H. Choi, R. Gupta, L. P. Safonova, L. A. Paunescu, A. Michalos, and E. Gratton. 2002. Functional frequency-domain near-infrared spectroscopy detects fast neuronal signal in the motor cortex. Neuroimage 17, 1868--1875.
80. J. R. Wolpaw, D. J. McFarland, G. W. Neat, and C. A. Forneris. 1991. An EEG-based brain-computer interface for cursor control. Electroencephalography and Clinical Neurophysiology 78, 3, 252.
81. T. O. Zander, C. Kothe, S. Welke, and M. Roetting. 2008. Enhancing human-machine systems with secondary input from passive brain-computer interfaces. In Proceedings of the 4th International BCI Workshop & Training Course.
82. T. O. Zander, J. Brönstrup, R. Lorenz, and L. R. Krol. 2014. Towards BCI-based implicit control in human--computer interaction. In Advances in Physiological Computing. Springer, London, 67--90.
83. T. O. Zander and S. Jatzev. 2012. Context-aware brain--computer interfaces: Exploring the information space of user, technical system and environment. Journal of Neural Engineering 9, 1.
84. T. O. Zander and C. Kothe. 2011. Towards passive brain--computer interfaces: Applying brain--computer interface technology to human--machine systems in general. Journal of Neural Engineering 8, 2.
85. T. O. Zander, C. Kothe, S. Jatzev, and M. Gaertner. 2010. Enhancing human-computer interaction with input from active and passive brain-computer interfaces. In Brain-Computer Interfaces, Springer, London, 181--199.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)