Evaluation Beyond Usability: Validating Sustainable HCI Research

ABSTRACT

The evaluation of research artefacts is an important step to validate research contributions. Sub-disciplines of HCI often pursue primary goals other than usability, such as Sustainable HCI (SHCI), HCI for development, or health and wellbeing. For such disciplines, established evaluation methods are not always appropriate or sufficient, and new conventions for identifying, discussing, and justifying suitable evaluation methods need to be established. In this paper, we revisit the purpose and goals of evaluation in HCI and SHCI, and elicit five key elements that can provide guidance to identifying evaluation methods for SHCI research. Our essay is meant as a starting point for discussing current and improving future evaluation practice in SHCI; we also believe it holds value for other subdisciplines in HCI that encounter similar challenges while evaluating their research.

References

1. Meyer H. Abrams. 1953. The Mirror and the Lamp: Romantic Theory and the Critical Tradition. Oxford University Press, New York.
2. ACM SIGCHI. 2017. Guide to a Successful Paper or Note Submission. Retrieved from https://chi2017.acm.org/guide-paper-submission.html
3. Christopher Alexander. 1979. The Timeless Way of Building. Oxford University Press, New York, NY.
4. Christopher Alexander, Sara Ishikawa, and Murray Silverstein. 1977. A Pattern Language: Towns, Buildings, Construction. Oxford University Press, New York.
5. Yaw Anokwa, Thomas N. Smyth, Divya Ramachandran, Jahanzeb Sherwani, Yael Schwartzman, Rowena Luk, Melissa Ho, Neema Moraveji, and Brian DeRenzi. 2009. Stories from the Field: Reflections on HCI4D Experiences. Information Technologies & International Development 5, 4: 101-- 116.
6. Holly Arrow, Joseph Edward McGrath, and Jennifer L. Berdahl. 2000. Small Groups as Complex Systems: Formation, Coordination, Development, and Adaptation. SAGE Publications, Inc, Thousand Oaks, Calif.
7. Louise Barkhuus and Jennifer A. Rode. 2007. From Mice to Men - 24 Years of Evaluation in CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07).
8. Oliver Bates, Mike Hazas, Adrian Friday, Janine Morley, and Adrian K. Clear. 2014. Towards an Holistic View of the Energy and Environmental Impacts of Domestic Media and IT. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14), 1173--1182.
9. Oliver Bates, Vanessa Thomas, and Christian Remy. 2017. Doing Good in HCI: Can We Broaden Our Agenda? Interactions (New York) 24, 5.
10. Eric P.S. Baumer and M. Six Silberman. 2011. When the Implication is Not to Design (Technology). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 2271--2274.
11. Jack Beaton and Ripul Kumar. 2010. Indian Cultural Effects on User Research Methodologies. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10), 4267--4272.
12. Peter Beigl, Stewart Hickey, Gudrun Obersteiner, Colin Fitzpatrick, Karsten Schischke, Paul Maher, and Jose Ospina. 2013. Towards zero waste in industrial networks: a case study of the D4R laptop. In First International Conference on ICT for Sustainability (ICT4S '13).
13. Enrico Bertini, Catherine Plaisant, and Giuseppe Santucci. 2007. BELIV'06: Beyond Time and Errors; Novel Evaluation Methods for Information Visualization. interactions 14, 3: 59--60.
14. Eli Blevis. 2007. Sustainable interaction design: invention & disposal, renewal & reuse. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07), 503--512.
15. Eli Blevis, Youn-kyung Lim, David Roedl, and Erik Stolterman. 2007. Using design critique as research to link sustainability and interactive technologies. In Proceedings of the 2nd international conference on Online communities and social computing (OCSC '07), 22--31. Retrieved March 25, 2013 from http://dl.acm.org/citation.cfm?id=1784297.1784301
16. Eli Blevis, Kevin Makice, William Odom, David Roedl, Christian Beck, Shunying Blevis, and Arvind Ashok. 2007. Luxury & new luxury, quality & equality. In Proceedings of the 2007 conference on Designing pleasurable products and interfaces (DPPI '07), 296--311.
17. Mark Blythe. 2014. Research Through Design Fiction: Narrative in Real and Imaginary Abstracts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14), 703--712.
18. Leonardo Bonanni, Matthew Hockenberry, David Zwarg, Chris Csikszentmihalyi, and Hiroshi Ishii. 2010. Small business applications of sourcemap: a web tool for sustainable design and supply chain transparency. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 937--946.
19. Hronn Brynjarsdottir, Maria Håkansson, James Pierce, Eric Baumer, Carl DiSalvo, and Phoebe Sengers. 2012. Sustainably unpersuaded: how persuasion narrows our vision of sustainability. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems (CHI '12), 947--956.
20. Jay Chen. 2015. Computing within limits and ICTD. First Monday 20, 8. Retrieved December 15, 2017 from http://firstmonday.org/ojs/index.php/fm/article/view/61
21. Jay Chen. 2016. A Strategy for Limits-aware Computing. In Proceedings of the Second Workshop on Computing Within Limits (LIMITS '16), 1:1--1:6.
22. Paul Cilliers. 2001. Boundaries, hierarchies and networks in complex systems. International Journal of Innovation Management 05, 02: 135--147.
23. Michael Demmer, Eric Brewer, R.J. Honicky, Madelaine Plauch, Melissa Ho, Sonesh Surana, and Joyojeet Pal. 2006. The Challenges of Technology Research for Developing Regions. IEEE Pervasive Computing 5: 15--23.
24. Tawanna Dillahunt, Jennifer Mankoff, and Jodi Forlizzi. 2010. A proposed framework for assessing environmental sustainability in the HCI community. In Examining Appropriation, Re-Use, and Maintenance of Sustainability workshop at CHI 2010 (CHI '10 workshop).
25. Tawanna Dillahunt, Jennifer Mankoff, Eric Paulos, and Susan Fussell. 2009. It's not all about "Green": energy use in low-income communities. In Proceedings of the 11th international conference on Ubiquitous computing (Ubicomp '09), 255--264.
26. Carl DiSalvo, Phoebe Sengers, and Hrönn Brynjarsdóttir. 2010. Mapping the landscape of sustainable HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 1975--1984.
27. Alan Dix. 2004. validity. In CHI2004 Workshop on Reflexive HCI.
28. Alan Dix. 2008. Theoretical analysis and theory creation. In Research Methods for Human-Computer Interaction. Cambridge University Press.
29. Alan J. Dix, Janet E. Finlay, Gregory D. Abowd, Russell Beale, and Janet E. Finley. 1998. HumanComputer Interaction. Prentice Hall, London; New York.
30. Paul Dourish. 2006. Implications for Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06), 541--550.
31. Paul Dourish. 2010. HCI and environmental sustainability: the politics of design and the design of politics. In Proceedings of the 8th ACM Conference on Designing Interactive Systems (DIS '10), 1--10.
32. Hamid Ekbia and Bonnie Nardi. 2015. The Political Economy of Computing: The Elephant in the HCI Room. interactions 22, 6: 46--49.
33. European Commission: COM. 2015. Closing the loop An EU action plan for the Circular Economy. Retrieved October 27, 2016 from http://eurlex.europa.eu/legalcontent/EN/TXT/?uri=CELEX:52015DC0614
34. Jon Froehlich, Leah Findlater, and James Landay. 2010. The design of eco-feedback technology. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 1999--2008.
35. Silke Gegenbauer and Elaine M. Huang. 2012. Inspiring the design of longer-lived electronics through an understanding of personal attachment. In Proceedings of the Designing Interactive Systems Conference (DIS '12), 635--644.
36. Saul Greenberg and Bill Buxton. 2008. Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08), 111--120.
37. Greendelta. 2017. openLCA.org. Retrieved September 10, 2017 from /
38. Lorna A. Greening, David L. Greene, and Carmen Difiglio. 2000. Energy efficiency and consumption - the rebound effect - a survey. Energy Policy 28, 6: 389--401.
39. Barbara Grosse-Hering, Jon Mason, Dzmitry Aliakseyeu, Conny Bakker, and Pieter Desmet. 2013. Slow Design for Meaningful Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 3431--3440.
40. Jonathan M. Harris. 2000. Basic principles of sustainable development. Tufts University.
41. Helen Ai He, Saul Greenberg, and Elaine M. Huang. 2010. One size does not fit all: applying the transtheoretical model to energy feedback technology design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10), 927-- 936.
42. Thomas T. Hewett, Ronald Baecker, Stuart Card, Tom Carey, Jean Gasen, Marilyn Mantei, Gary Perlman, Gary Strong, and William Verplank. 1992. ACM SIGCHI Curricula for Human-Computer Interaction. ACM, New York, NY, USA.
43. Lorenz M. Hilty. 2008. Information Technology and Sustainability. Books On Demand, Norderstedt Germany.
44. Elaine M. Huang. 2011. Building outwards from sustainable HCI. interactions 18, 3: 14--17.
45. International Organization for Standardization. 2006. ISO 14040:2006 - Environmental management -- Life cycle assessment -- Principles and framework. Retrieved September 9, 2017 from https://www.iso.org/standard/37456.html
46. International Organization for Standardization. 2010. ISO 9241--210:2010 - Ergonomics of human-system interaction -- Part 210: Human-centred design for interactive systems. Retrieved August 25, 2017 from https://www.iso.org/standard/52075.html
47. International Organization for Standardization. 2016. ISO 14034:2016 - Environmental management -Environmental technology verification (ETV). Retrieved August 25, 2017 from https://www.iso.org/standard/43256.html
48. Samuel J. Kaufman and M. Six Silberman. 2011. Rebound effects in sustainable HCI. In Sustainable Interaction Design in Professional Domains workshop at CHI 2011 (CHI '11 workshop).
49. Rex B. Kline. 2010. Principles and Practice of Structural Equation Modeling, Third Edition. The Guilford Press, New York.
50. Bran Knowles, Lynne Blair, Stuart Walker, Paul Coulton, Lisa Thomas, and Louise Mullagh. 2014. Patterns of Persuasion for Sustainability. In Proceedings of the 2014 Conference on Designing Interactive Systems (DIS '14), 1035--1044.
51. Bran Knowles and Maria Håkansson. 2016. A Sustainable HCI Knowledge Base in Progress. interactions 23, 3: 74--76.
52. Neha Kumar and Susan Dray. 2017. Crossing Borders, Building Bridges. interactions 24, 3: 42--47.
53. Neha Kumar, Naveena Karusala, Aaditeshwar Seth, and Biswajit Patra. 2017. Usability, Tested? interactions 24, 4: 74--77.
54. Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2010. Research Methods in HumanComputer Interaction. Wiley, Chichester, West Sussex, U.K.
55. Joseph Lindley, Paul Coulton, and Emmett L. Brown. 2016. Peer Review and Design Fiction: "Honestly, they're not just made up." In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16), 583-- 595.
56. Yong Liu, Jorge Goncalves, Denzil Ferreira, Bei Xiao, Simo Hosio, and Vassilis Kostakos. 2014. CHI 19942013: Mapping Two Decades of Intellectual Progress Through Co-word Analysis. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14), 3553--3562.
57. Anton Lundström and Daniel Pargman. 2017. Developing a Framework for Evaluating the Sustainability of Computing Projects. In Proceedings of the 2017 Workshop on Computing Within Limits (LIMITS '17), 111--117.
58. Jennifer Mankoff. 2012. HCI and sustainability: a tale of two motivations. interactions 19, 3: 16--19.
59. Jennifer C. Mankoff, Eli Blevis, Alan Borning, Batya Friedman, Susan R. Fussell, Jay Hasbrouck, Allison Woodruff, and Phoebe Sengers. 2007. Environmental sustainability and interaction. In CHI '07 Extended Abstracts on Human Factors in Computing Systems (CHI EA '07), 2121--2124.
60. Jennifer Mankoff, Jennifer A. Rode, and Haakon Faste. 2013. Looking Past Yesterday's Tomorrow: Using Futures Studies Methods to Extend the Research Horizon. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 1629--1638.
61. Samantha McDonald, Bonnie Nardi, and Bill Tomlinson. 2017. Political Realities of Digital Communication: The Limits of Value from Digital Messages to Members of the US Congress. In Proceedings of the 2017 Workshop on Computing Within Limits (LIMITS '17), 129--138.
62. Bonnie Nardi and Hamid Ekbia. 2017. Developing a political economy perspective for sustainable HCI. In Digital Technology and Sustainability: Embracing the Paradox.
63. National Research Council. 2012. Computing Research for Sustainability. The National Academies Press, Washington, DC, USA.
64. Jakob Nielsen. 1994. Usability engineering. Morgan Kaufmann Publishers, San Francisco, Calif.
65. William Odom, James Pierce, Erik Stolterman, and Eli Blevis. 2009. Understanding why we preserve some things and discard others in the context of interaction design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), 1053--1062.
66. Antti Oulasvirta and Kasper Hornbæk. 2016. HCI Research As Problem-Solving. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 4956--4967.
67. Daniel Pargman, Elina Eriksson, Mattias Höjer, Ulrika Gunnarsson Östling, and Luciane Aguiar Borges. 2017. The (Un)Sustainability of Imagined Future Information Societies. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 773--785.
68. Daniel Pargman and Barath Raghavan. 2014. Rethinking Sustainability in Computing: From Buzzword to Non-negotiable Limits. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordiCHI '14), 638--647.
69. James Pierce. 2012. Undesigning technology: considering the negation of design by design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12), 957--966.
70. James Pierce and Eric Paulos. 2012. Beyond energy monitors: interaction, energy, and emerging energy systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12), 665-- 674.
71. Karl R. Popper. 1962. Conjectures and Refutations: The Growth of Scientific Knowledge. Basic Books.
72. Nicolas Prat, Isabelle Comyn-Wattiau, and Jacky Akoka. 2014. Artifact Evaluation in Information Systems Design-Science Research. PACIS 2014 Proceedings. Retrieved from http://aisel.aisnet.org/pacis2014/23
73. Pré consultants b.v. 2017. SimaPro. SimaPro. Retrieved September 10, 2017 from https://simapro.com/
74. Jenny Preece, Yvonne Rogers, and Helen Sharp. 2002. Interaction Design. Wiley, New York, NY.
75. Chris Preist, Daniel Schien, and Eli Blevis. 2016. Understanding and Mitigating the Effects of Device and Cloud Service Design Decisions on the Environmental Footprint of Digital Infrastructure. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1324--1337.
76. Jan Pries-Heje, Richard Baskerville, and John Venable. 2008. Strategies for Design Science Research Evaluation. ECIS 2008 Proceedings. Retrieved from http://aisel.aisnet.org/ecis2008/87
77. Christian Remy, Oliver Bates, Thomas Vanessa, and Elaine May Huang. 2017. The Limits of Evaluating Sustainability. In Proceedings of the Third Workshop on Computing within Limits.
78. Christian Remy, Silke Gegenbauer, and Elaine M. Huang. 2015. Bridging the Theory-Practice Gap: Lessons and Challenges of Applying the Attachment Framework for Sustainable HCI Design. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 1305--1314.
79. Christian Remy and Elaine M. Huang. 2015. Addressing the Obsolescence of End-User Devices: Approaches from the Field of Sustainable HCI. In ICT Innovations for Sustainability, Lorenz M. Hilty and Bernard Aebischer (eds.). Springer International Publishing, Switzerland, 474.
80. Christian Remy and Elaine M. Huang. 2015. Limits and sustainable interaction design: Obsolescence in a future of collapse and resource scarcity. First Monday 20, 8. Retrieved November 10, 2015 from http://firstmonday.org/ojs/index.php/fm/article/view/61
81. David J. Roedl and Erik Stolterman. 2013. Design Research at CHI and Its Applicability to Design Practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 1951--1954.
82. David Roedl, William Odom, and Eli Blevis. 2017. Three Principles of Sustainable Interaction Design, Revisited. In Digital Technology and Sustainability: Embracing the Paradox.
83. Yvonne Rogers. 2004. New theoretical approaches for human-computer interaction. Annual Review of Information Science and Technology 38, 1: 87--143.
84. Yvonne Rogers, Jenny Preece, and Helen Sharp. 2011. Interaction design. Wiley, Hoboken, N.J.; Chichester.
85. Antti Salovaara, Antti Oulasvirta, and Giulio Jacucci. 2017. Evaluation of Prototypes and the Problem of Possible Futures. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 2064--2077.
86. Phoebe Sengers and Bill Gaver. 2006. Staying Open to Interpretation: Engaging Multiple Meanings in Design and Evaluation. In Proceedings of the 6th Conference on Designing Interactive Systems (DIS '06), 99--108.
87. Ben Shneiderman, Catherine Plaisant, Maxine Cohen, and Steven Jacobs. 2009. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson, Boston.
88. Elizabeth Shove, Mika Pantzar, and Matt Watson. 2012. The Dynamics of Social Practice: Everyday Life and how it Changes. SAGE Publications Ltd, Los Angeles.
89. M. Six Silberman, Lisa Nathan, Bran Knowles, Roy Bendor, Adrian Clear, Maria Håkansson, Tawanna Dillahunt, and Jennifer Mankoff. 2014. Next steps for sustainable HCI. interactions 21, 5: 66--69.
90. M. Six Silberman and Bill Tomlinson. 2010. Toward an ecological sensibility: tools for evaluating sustainable HCI. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10), 3469--3474.
91. Bruce Sterling. 2009. COVER STORY: Design Fiction. interactions 16, 3: 20--24.
92. Erik Stolterman. 2008. The Nature of Design Practice and Implications for Interaction Design Research. International Journal of Dsign 2, 1: 55--65.
93. Alistair Sutcliffe. 2000. On the Effective Use and Reuse of HCI Knowledge. ACM Trans. Comput.-Hum. Interact. 7, 2: 197--221.
94. Vanessa Thomas. 2017. Negotiating and engaging with environmental public policy at different scales. In Digital Technology and Sustainability: Embracing the Paradox.
95. Vanessa Thomas, Christian Remy, Mike Hazas, and Oliver Bates. 2017. HCI and Environmental Public Policy: Opportunities for Engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.
96. Bill Tomlinson, Eli Blevis, Bonnie Nardi, Donald J. Patterson, M. SIX Silberman, and Yue Pan. 2013. Collapse Informatics and Practice: Theory, Method, and Design. ACM Trans. Comput.-Hum. Interact. 20, 4: 24:1--24:26.
97. Bill Tomlinson, M. Six Silberman, Donald Patterson, Yue Pan, and Eli Blevis. 2012. Collapse informatics: augmenting the sustainability & ICT4D discourse in HCI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12), 655-- 664.
98. Kentaro Toyama. 2015. Preliminary thoughts on a taxonomy of value for sustainable computing. First Monday 20, 8. Retrieved August 25, 2017 from http://ojphi.org/ojs/index.php/fm/article/view/6132
99. United Nations. 2015. Paris Agreement.
100. United Nations. 2017. Sustainable development goals. United Nations Sustainable Development. Retrieved September 7, 2017 from http://www.un.org/sustainabledevelopment/sustainable -development-goals/
101. John Venable, Jan Pries-Heje, and Richard Baskerville. 2012. A Comprehensive Framework for Evaluation in Design Science Research. In Proceedings of the 7th International Conference on Design Science Research in Information Systems: Advances in Theory and Practice (DESRIST'12), 423--438.
102. John Venable, Jan Pries-Heje, and Richard Baskerville. 2016. FEDS: a Framework for Evaluation in Design Science Research. European Journal of Information Systems 25, 1: 77--89.
103. Martin Woolley. 2003. Choreographing obsolescence ecodesign: the pleasure/dissatisfaction cycle. In Proceedings of the 2003 international conference on Designing pleasurable products and interfaces (DPPI '03), 77--81.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)