Abstract
In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) on an interface. This article contributes individualised predictive models of visual search, and a computational approach to restructure graphical layouts for an individual user such that features on a new, unvisited interface can be found quicker. It explores four technical principles inspired by the human visual system (HVS) to predict expected positions of features and create individualised layout templates: (I) the interface with highest frequency is chosen as the template; (II) the interface with highest predicted recall probability (serial position curve) is chosen as the template; (III) the most probable locations for features across interfaces are chosen (visual statistical learning) to generate the template; (IV) based on a generative cognitive model, the most likely visual search locations for features are chosen (visual sampling modelling) to generate the template. Given a history of previously seen interfaces, we restructure the spatial layout of a new (unseen) interface with the goal of making its features more easily findable. The four HVS principles are implemented in Familiariser, a web browser that automatically restructures webpage layouts based on the visual history of the user. Evaluation of Familiariser (using visual statistical learning) with users provides first evidence that our approach reduces visual search time by over 10%, and number of eye-gaze fixations by over 20%, during web browsing tasks.
- John R. Anderson, D. Bothell, C. Lebiere, and M. Matessa. 1998. An integrated theory of list memory. Journal of Memory and Language 38, 4 (1998), 341--380.Google ScholarCross Ref
- John R. Anderson, Jon M. Fincham, and Scott Douglass. 1999. Practice and retention: A unifying analysis. Journal of Experimental Psychology-Learning Memory and Cognition 25, 5 (1999), 1120--1136.Google ScholarCross Ref
- Yigal Arens, Lawrence Miller, Stuart C. Shapiro, and Norman K. Sondheimer. 1988. Automatic construction of user-interface displays. In Proceedings of the 7th AAAI National Conference on Artificial Intelligence (AAAI’88). AAAI Press, 808--813. http://dl.acm.org/citation.cfm?id=2887965.2888108 Google ScholarDigital Library
- Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. MenuOptimizer: Interactive optimization of menu systems. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST’13). ACM, New York, NY, 331--342. Google ScholarDigital Library
- C. M. Beshers and S. Feiner. 1989. Scope: Automated generation of graphical interfaces. In Proceedings of the 2nd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology (UIST’89). ACM, New York, NY, 76--85. Google ScholarDigital Library
- David M. Blei. 2012. Probabilistic topic models. Communications of the ACM 55, 4 (2012), 77--84. Google ScholarDigital Library
- François Bodart, Anne-Marie Hennebert, Jean-Marie Leheureux, and Jean Vanderdonckt. 1994. Towards a dynamic strategy for computer-aided visual placement. In Proceedings of the Workshop on Advanced Visual Interfaces (AVI’94). ACM, New York, NY, 78--87. Google ScholarDigital Library
- Marvin M. Chun and Yuhong Jiang. 1998. Contextual cueing: Implicit learning and memory of visual context guides spatial attention. Cognitive Psychology 36, 1 (1998), 28--71.Google ScholarCross Ref
- Marvin M. Chun and Yuhong Jiang. 1999. Top-down attentional guidance based on implicit learning of visual covariation. Psychological Science 10, 4 (1999), 360--365.Google ScholarCross Ref
- Morgan Dixon and James Fogarty. 2010. Prefab: Implementing advanced behaviors using pixel-based reverse engineering of interface structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’10). ACM, New York, NY, 1525--1534. Google ScholarDigital Library
- Peter A. Frensch. 1994. Composition during serial learning: A serial position effect. Journal of Experimental Psychology Learning Memory and Cognition 20 (1994), 423--423.Google ScholarCross Ref
- Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically generating user interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI’04). ACM, New York, NY, 93--100. Google ScholarDigital Library
- Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S. Weld. 2006. Exploring the design space for adaptive graphical user interfaces (AVI’06). ACM, New York, NY, 201--208. Google ScholarDigital Library
- Krzysztof Z. Gajos, Jing Jing Long, and Daniel S. Weld. 2006. Automatically generating custom user interfaces for users with physical disabilities. In Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility (Assets’06). ACM, New York, NY, 243--244. Google ScholarDigital Library
- Krzysztof Z. Gajos, Jacob O. Wobbrock, and Daniel S. Weld. 2007. Automatically generating user interfaces adapted to users’ motor and vision capabilities. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST’07). ACM, New York, NY, 231--240. Google ScholarDigital Library
- Saul Greenberg and Ian H. Witten. 1985. Adaptive personalized interfaces—A question of viability. Behaviour and Information Technology 4, 1 (1985), 31--45.Google ScholarCross Ref
- Richard N. A. Henson. 1996. Unchained memory: Error patterns rule out chaining models of immediate serial recall. The Quarterly Journal of Experimental Psychology: Section A 49, 1 (1996), 80--115.Google ScholarCross Ref
- Helene Intraub and Michael Richardson. 1989. Wide-angle memories of close-up scenes. Journal of Experimental Psychology: Learning, Memory, and Cognition 15, 2 (1989), 179.Google ScholarCross Ref
- Laurent Itti and Christof Koch. 2000. A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research 40, 10 (2000), 1489--1506.Google ScholarCross Ref
- Christian Janssen, Anette Weisbecker, and Jürgen Ziegler. 1993. Generating user interfaces from data models and dialogue net specifications. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems (CHI’93). ACM, New York, NY, 418--423. Google ScholarDigital Library
- Jussi P. P. Jokinen, Sayan Sarcar, Antti Oulasvirta, Chaklam Silpasuwanchai, Zhenxin Wang, and Xiangshi Ren. 2017. Modelling learning of new keyboard layouts. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI’17). ACM, New York, NY, 4203--4215. Google ScholarDigital Library
- Andruid Kerne, William A. Hamilton, and Zachary O. Toups. 2012. Culturally based design: Embodying trans-surface interaction in rummy. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). ACM, New York, NY, 509--518. Google ScholarDigital Library
- Eileen Kowler. 2011. Eye movements: The past 25 years. Vision Research 51, 13 (2011), 1457--1483.Google ScholarCross Ref
- Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design mining the web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’13). ACM, New York, NY, 3083--3092. Google ScholarDigital Library
- Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, and Scott R. Klemmer. 2011. Bricolage: Example-based retargeting for web design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 2197--2206. Google ScholarDigital Library
- Talia Lavie and Joachim Meyer. 2010. Benefits and costs of adaptive user interfaces. International Journal of Human-Computer Studies 68, 8 (Aug. 2010), 508--524. Google ScholarDigital Library
- Jeffrey Nichols and Tessa Lau. 2008. Mobilization by demonstration: Using traces to re-author existing web sites. In Proceedings of the 13th International Conference on Intelligent User Interfaces (IUI’08). ACM, New York, NY, 149--158. Google ScholarDigital Library
- Jeffrey Nichols, Brad A. Myers, Michael Higgins, Joseph Hughes, Thomas K. Harris, Roni Rosenfeld, and Mathilde Pignol. 2002. Generating remote control interfaces for complex appliances. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (UIST’02). ACM, New York, NY, 161--170. Google ScholarDigital Library
- Jeffrey Nichols, Brad A. Myers, and Brandon Rothrock. 2006. UNIFORM: Automatically generating consistent remote control user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’06). ACM, New York, NY, 611--620. Google ScholarDigital Library
- Jakob Nielsen. 1993. Usability Engineering. Morgan Kaufmann Publishers, Inc., San Francisco, CA. Google ScholarDigital Library
- Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014. Learning layouts for single-page graphic designs. IEEE Transactions on Visualization and Computer Graphics 20, 8 (Aug. 2014), 1200--1213. Google ScholarDigital Library
- Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with interactive layout suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 1221--1224. Google ScholarDigital Library
- D. R. Olsen, Jr.1989. A programming language basis for user interface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’89). ACM, New York, NY, 171--176. Google ScholarDigital Library
- Angel R. Puerta, Henrik Eriksson, John H. Gennari, and Mark A. Musen. 1994. Model-based automated generation of user interfaces. In Proceedings of the 12h National Conference on Artificial Intelligence (Vol. 1) (AAAI’94). American Association for Artificial Intelligence, 471--477. http://dl.acm.org/citation.cfm?id=199288.184583 Google ScholarDigital Library
- Keith Rayner. 2009. The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology 62, 8 (2009), 1457--1506.Google ScholarCross Ref
- John Rheinfrank and Shelley Evenson. 1996. Design languages. In Bringing Design to Software. ACM, 63--85. Google ScholarDigital Library
- Dario D. Salvucci. 2001. An integrated model of eye movements and visual encoding. Cognitive Systems Research 1, 4 (Feb. 2001), 201--220. Google ScholarDigital Library
- Sayan Sarcar, Jussi Jokinen, Antti Oulasvirta, Xiangshi Ren, Chaklam Silpasuwanchai, and Zhenxin Wang. 2018. Ability-based optimization: Designing smartphone text entry interface for older adults. IEEE Pervasive Computing 17 (2018), 15--26. Google ScholarDigital Library
- A. Sears. 1993. Layout appropriateness: A metric for evaluating user interface widget layout. IEEE Transactions on Software Engineering 19, 7 (July 1993), 707--719. Google ScholarDigital Library
- Andrew Sears, Julie A. Jacko, Josey Chu, and Francisco Moro. 2001. The role of visual search in the design of effective soft keyboards. Behaviour and Information Technology 20, 3 (2001), 159--166.Google ScholarCross Ref
- Kashyap Todi, Jussi Jokinen, Kris Luyten, and Antti Oulasvirta. 2018. Familiarisation: Restructuring layouts with visual learning models. In Proceedings of the 23rd International Conference on Intelligent User Interfaces (IUI’18). ACM, New York, NY, 547--558. Google ScholarDigital Library
- Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and explore with a layout optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS’16). ACM, New York, NY, 543--555. Google ScholarDigital Library
- Anne M. Treisman and Garry Gelade. 1980. A feature-integration theory of attention. Cognitive Psychology 12, 1 (1980), 97--136.Google ScholarCross Ref
- Martijn Van Welie and Gerrit C. Van der Veer. 2003. Pattern languages in interaction design: Structure and organization. In Proceedings of Interact, Vol. 3. 1--5.Google Scholar
- Jacob O. Wobbrock, Shaun K. Kane, Krzysztof Z. Gajos, Susumu Harada, and Jon Froehlich. 2011. Ability-based design: Concept, principles and examples. ACM Transactions on Accessible Computing 3, 3 (April 2011), Article 9, 27 pages. Google ScholarDigital Library
- Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: Using GUI screenshots for search and automation. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST’09). ACM, New York, NY, 183--192. Google ScholarDigital Library
Index Terms
- Individualising Graphical Layouts with Predictive Visual Search Models
Recommendations
Familiarisation: Restructuring Layouts with Visual Learning Models
IUI '18: Proceedings of the 23rd International Conference on Intelligent User InterfacesIn domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) in familiar locations. This paper contributes computational approaches to restructuring ...
Towards accurate and practical predictive models of active-vision-based visual search
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsBeing able to predict the performance of interface designs using models of human cognition and performance is a long-standing goal of HCI research. This paper presents recent advances in cognitive modeling which permit increasingly realistic and ...
Personalizing graphical user interfaces on flexible widget layout
EICS '09: Proceedings of the 1st ACM SIGCHI symposium on Engineering interactive computing systemsThe authors propose a method for personalizing the flexible widget layout (FWL) by adjusting the desirability of widgets with a pairwise comparison method, and show its implementation and that it actually works. Personalization of graphical user ...
Comments