Participatory Sensing or Participatory Nonsense?: Mitigating the Effect of Human Error on Data Quality in Citizen Science

Abstract

Citizen Science with mobile and wearable technology holds the possibility of unprecedented observation systems. Experts and policy makers are torn between enthusiasm and scepticism regarding the value of the resulting data, as their decision making traditionally relies on high-quality instrumentation and trained personnel measuring in a standardized way. In this paper, we (1) present an empirical behavior taxonomy of errors exhibited in non-expert smartphone-based sensing, based on four small exploratory studies, and discuss measures to mitigate their effects. We then present a large summative study (N=535) that compares instructions and technical measures to address these errors, both from the perspective of improvements to error frequency and perceived usability. Our results show that (2) technical measures without explanation notably reduce the perceived usability and (3) technical measures and instructions nicely complement each other: Their combination achieves a significant reduction in observed error rates while not affecting the user experience negatively.

References

1. Harini Alagarai Sampath, Rajeev Rajeshuni, and Bipin Indurkhya. 2014. Cognitively Inspired Task Design to Improve User Performance on Crowdsourcing Platforms. In 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, 3665--3674.
2. Victoria Bellotti and Keith Edwards. 2001. Intelligibility and accountability: human considerations in context-aware systems. Human--Computer Interaction 16, 2-4 (2001), 193--212.
3. Rick Bonney, Jennifer L Shirk, Tina B Phillips, Andrea Wiggins, Heidi L Ballard, Abraham J Miller-Rushing, Julia K Parrish, et al. 2014. Next steps for citizen science. Science 343, 6178 (2014).
4. John Brooke. 1996. SUS -- A quick and dirty usability scale. Usability evaluation in industry 189 (1996).
5. Matthias Budde, Pierre Barbera, Rayan El Masri, Till Riedel, and Michael Beigl. 2013. Retrofitting Smartphones to be Used as Particulate Matter Dosimeters. In International Symposium on Wearable Computers (ISWC’13).
6. Matthias Budde, Marcel Köpke, and Michael Beigl. 2015. Robust In-situ Data Reconstruction from Poisson Noise for Low-cost, Mobile, Non-expert Environmental Sensing. In International Symposium on Wearable Computers (ISWC’15). ACM, 179--182.
7. Matthias Budde, Rikard Öxler, Michael Beigl, and Jussi Holopainen. 2016. Sensified Gaming -- Design Patterns and Game Design Elements for Gameful Environmental Sensing. In 13th Int. Conference on Advances in Computer Entertainment Technology (ACE2016).
8. Matthias Budde, Lin Zhang, and Michael Beigl. 2014. Distributed, Low-cost Particulate Matter Sensing: Scenarios, Challenges, Approaches. In ProScience, Vol. 1. (Proc. 1st Int. Conf. Atmospheric Dust (DUST2014)).
9. Jeffrey A Burke, Deborah Estrin, Mark Hansen, Andrew Parker, Nithya Ramanathan, Sasank Reddy, and Mani B Srivastava. 2006. Participatory sensing. Center for Embedded Network Sensing (2006).
10. Delphine Christin, Andreas Reinhardt, Salil S. Kanhere, and Matthias Hollick. 2011. A survey on privacy in mobile participatory sensing applications. Journal of Systems and Software 84, 11 (2011). Mobile Applications: Status and Trends.
11. Anind K. Dey. 2001. Understanding and Using Context. Personal and Ubiquitous Computing 5, 1 (2001), 4--7.
12. Anind K. Dey and Alan Newberger. 2009. Support for Context-aware Intelligibility and Control. In CHI ’09. ACM, 859--868.
13. J St BT Evans. 1988. The knowledge elicitation problem: a psychological perspective. Behaviour 8 Information Technology 7, 2 (1988).
14. Mary M Gardiner, Leslie L Allee, Peter MJ Brown, John E Losey, Helen E Roy, and Rebecca Rice Smyth. 2012. Lessons from lady beetles: accuracy of monitoring data from US and UK citizen-science programs. Frontiers in Ecology and the Environment 10, 9 (2012), 471--476.
15. Steven K. Gibb. 2015. Volunteers Against Pollution. Chemical 8 Engineering News (C8EN) 93, 36 (Sept. 2015).
16. Mike Harding, Bran Knowles, Nigel Davies, and Mark Rouncefield. 2015. HCI, Civic Engagement 8 Trust. In 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 2833--2842.
17. John D Hoyt and Harry Wechsler. 1994. Detection of human speech in structured noise. In Acoustics, Speech, and Signal Processing, 1994. ICASSP-94., 1994 IEEE International Conference on, Vol. 2. IEEE, II--237.
18. Kuan Lun Huang, Salil S Kanhere, and Wen Hu. 2010. Are you contributing trustworthy data?: the case for a reputation system in participatory sensing. In Modeling, analysis, and simulation of wireless and mobile systems.
19. Eiman Kanjo. 2010. Noisespy: A real-time mobile phone platform for urban noise monitoring and mapping. Mobile Networks and Applications 15, 4 (2010), 562--574.
20. Sunyoung Kim, Jennifer Mankoff, and Eric Paulos. 2013. Sensr: Evaluating a Flexible Framework for Authoring Mobile Data-collection Tools for Citizen Science. In Computer Supported Cooperative Work (CSCW ’13). ACM, 10.
21. Sunyoung Kim, Christine Robson, Thomas Zimmerman, Jeffrey Pierce, and Eben M. Haber. 2011. Creek Watch: Pairing Usefulness and Usability for Successful Citizen Science. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, 2125--2134.
22. Eli Kintisch. 2011. How to Grow Your Own Army of Citizen Scientists. (24 Feb. 2011).
23. Simon Klakegg, Chu Luo, Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2016. Instrumenting Smartphones with Portable NIRS. In Adj. Proceedings UbiComp’16, Workshop in Ubiquitous Mobile Instrumentation (UbiMI).
24. N.D. Lane, E. Miluzzo, Hong Lu, D. Peebles, T. Choudhury, and A.T. Campbell. 2010. A survey of mobile phone sensing. IEEE Communications Magazine 48, 9 (2010).
25. B. Laugwitz, T. Held, and M. Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and Usability Engineering Group. Springer, 63--76.
26. Kurt Luther, Scott Counts, Kristin B. Stecher, Aaron Hoff, and Paul Johns. 2009. Pathfinder: An Online Collaboration Environment for Citizen Scientists. In Human Factors in Computing Systems (CHI ’09). ACM, 239--248.
27. Nicolas Maisonneuve, Matthias Stevens, and Bartek Ochab. 2010. Participatory noise pollution monitoring using mobile phones. In Information Polity. Vol. 15. 51 -- 71. Issue 1.
28. Sebastian Matyas, Peter Kiefer, Christoph Schlieder, and Sara Kleyer. 2011. Wisdom about the Crowd: Assuring Geospatial Data Quality Collected in Location-Based Games. In International Conference on Entertainment Computing (ICEC 2011). 331--336.
29. George A. Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review 91 (1956), 81--97.
30. L. A. Muratori, P. Salomoni, and G. Pau. 2011. Feeling the pack: Strategies for an optimal participatory system to sense and recognize noise pollution. In 2011 IEEE International Conference on Consumer Electronics - Berlin (ICCE -Berlin). 17--21.
31. Donald A. Norman. 1983. Design Rules Based on Analyses of Human Error. Commun. ACM 26, 4 (April 1983), 254--258.
32. Donald A. Norman and Pieter Jan Stappers. 2015. DesignX: Complex Sociotechnical Systems. She Ji 1 (2015), 83--106. Issue 2.
33. DG Novick and K Ward. 2006. Why Don’t People Read the Manual?. In Int. Conference on Design of Communication (SIGDOC ’06). ACM.
34. L. Peterson and M. Peterson. 1959. Short-term retention of individual verbal items. Journal of Experimental Psychology 58 (1959).
35. Rajib Kumar Rana, Chun Tung Chou, Salil S Kanhere, Nirupama Bulusu, and Wen Hu. 2010. Ear-phone: an end-to-end participatory urban noise mapping system. In 9th ACM/IEEE International Conference on Information Processing in Sensor Networks. 105--116.
36. Sasank Reddy, Deborah Estrin, and Mani Srivastava. 2010. Recruitment Framework for Participatory Sensing Data Collections. In Pervasive Computing. LNCS, Vol. 6030.
37. I. Schweizer, R. Bärtl, A. Schulz, F. Probst, and M. Mühläuser. 2011. NoiseMap -- real-time participatory noise maps. In PhoneSense’11.
38. L. See, A. Comber, C. Salk, S. Fritz, M. van der Velde, C. Perger, C. Schill, I. McCallum, F. Kraxner, and M. Obersteiner. 2013. Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-Experts. PLoS ONE 8, 7 (2013).
39. S Andrew Sheppard and Loren Terveen. 2011. Quality is a verb: the operationalization of data quality in a citizen science community. In Proceedings of the 7th International Symposium on Wikis and Open Collaboration. 29--38.
40. S. Andrew Sheppard, Andrea Wiggins, and Loren Terveen. 2014. Capturing Quality: Retaining Provenance for Curated Volunteer Monitoring Data. In 17th ACM Conference on Computer Supported Cooperative Work 8 Social Computing (CSCW ’14). ACM, 1234--1245.
41. F Snik, JHH Rietjens, A Apituley, H Volten, B Mijling, A Di Noia, S Heikamp, RC Heinsbroek, OP Hasekamp, and JM Smit. 2014. Mapping atmospheric aerosols with a citizen science network of smartphone spectropolarimeters. Geophysical Research Letters 41, 20 (2014).
42. John Sweller. 2002. Visualisation and instructional design. In International Workshop on Dynamic Visualizations and Learning.
43. Brett Amy Thelen and Rachel K. Thiet. 2008. Cultivating connection: Incorporating meaningful citizen science into Cape Cod National Seashore’s estuarine research and monitoring programs. Park Science 25, 1 (2008).
44. A. Truskinger, Haofan Yang, J. Wimmer, Jinglan Zhang, I. Williamson, and P. Roe. 2011. Large Scale Participatory Acoustic Sensor Data Analysis: Tools and Reputation Models to Enhance Effectiveness. In E-Science.
45. Kathleen Tuite, Noah Snavely, Dun-yu Hsiao, Nadine Tabing, and Zoran Popovic. 2011. PhotoCity: Training Experts at Large-scale Image Acquisition Through a Competitive Game. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, 10.
46. VDA-QMC. 2011. Messsystem und Messprozess sind zweierlei. QZ 56, 5 (2011).
47. Christopher D Wickens, John Lee, Yili D Liu, and Sallie Gordon-Becker. 2014. Introduction to Human Factors Engineering: Pearson New International Edition. Pearson Higher Ed. Second Edition.
48. Andrea Wiggins and Yurong He. 2016. Community-based Data Validation Practices in Citizen Science. In 19th ACM Conference on Computer-Supported Cooperative Work 8 Social Computing (CSCW ’16). ACM, 1548--1559.
49. Patricia Wright. 1981. “The instructions clearly state …” can’t people read? Applied Ergonomics 12 (September 1981), 131--141. Issue 3.
50. Poonam Yadav and John Darlington. 2016. Design Guidelines for the User-Centred Collaborative Citizen Science Platforms. arXiv preprint arXiv:1605.00910 (2016).

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI

  • Interaction design

    • Interaction design process and methods

      • User centered design

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile computing design and evaluation methods