Abstract
Citizen Science with mobile and wearable technology holds the possibility of unprecedented observation systems. Experts and policy makers are torn between enthusiasm and scepticism regarding the value of the resulting data, as their decision making traditionally relies on high-quality instrumentation and trained personnel measuring in a standardized way. In this paper, we (1) present an empirical behavior taxonomy of errors exhibited in non-expert smartphone-based sensing, based on four small exploratory studies, and discuss measures to mitigate their effects. We then present a large summative study (N=535) that compares instructions and technical measures to address these errors, both from the perspective of improvements to error frequency and perceived usability. Our results show that (2) technical measures without explanation notably reduce the perceived usability and (3) technical measures and instructions nicely complement each other: Their combination achieves a significant reduction in observed error rates while not affecting the user experience negatively.
- Harini Alagarai Sampath, Rajeev Rajeshuni, and Bipin Indurkhya. 2014. Cognitively Inspired Task Design to Improve User Performance on Crowdsourcing Platforms. In 32nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, 3665--3674. Google ScholarDigital Library
- Victoria Bellotti and Keith Edwards. 2001. Intelligibility and accountability: human considerations in context-aware systems. Human--Computer Interaction 16, 2-4 (2001), 193--212. Google ScholarDigital Library
- Rick Bonney, Jennifer L Shirk, Tina B Phillips, Andrea Wiggins, Heidi L Ballard, Abraham J Miller-Rushing, Julia K Parrish, et al. 2014. Next steps for citizen science. Science 343, 6178 (2014).Google Scholar
- John Brooke. 1996. SUS -- A quick and dirty usability scale. Usability evaluation in industry 189 (1996).Google Scholar
- Matthias Budde, Pierre Barbera, Rayan El Masri, Till Riedel, and Michael Beigl. 2013. Retrofitting Smartphones to be Used as Particulate Matter Dosimeters. In International Symposium on Wearable Computers (ISWC’13). Google ScholarDigital Library
- Matthias Budde, Marcel Köpke, and Michael Beigl. 2015. Robust In-situ Data Reconstruction from Poisson Noise for Low-cost, Mobile, Non-expert Environmental Sensing. In International Symposium on Wearable Computers (ISWC’15). ACM, 179--182. Google ScholarDigital Library
- Matthias Budde, Rikard Öxler, Michael Beigl, and Jussi Holopainen. 2016. Sensified Gaming -- Design Patterns and Game Design Elements for Gameful Environmental Sensing. In 13th Int. Conference on Advances in Computer Entertainment Technology (ACE2016). Google ScholarDigital Library
- Matthias Budde, Lin Zhang, and Michael Beigl. 2014. Distributed, Low-cost Particulate Matter Sensing: Scenarios, Challenges, Approaches. In ProScience, Vol. 1. (Proc. 1st Int. Conf. Atmospheric Dust (DUST2014)).Google Scholar
- Jeffrey A Burke, Deborah Estrin, Mark Hansen, Andrew Parker, Nithya Ramanathan, Sasank Reddy, and Mani B Srivastava. 2006. Participatory sensing. Center for Embedded Network Sensing (2006).Google Scholar
- Delphine Christin, Andreas Reinhardt, Salil S. Kanhere, and Matthias Hollick. 2011. A survey on privacy in mobile participatory sensing applications. Journal of Systems and Software 84, 11 (2011). Mobile Applications: Status and Trends. Google ScholarDigital Library
- Anind K. Dey. 2001. Understanding and Using Context. Personal and Ubiquitous Computing 5, 1 (2001), 4--7. Google ScholarDigital Library
- Anind K. Dey and Alan Newberger. 2009. Support for Context-aware Intelligibility and Control. In CHI ’09. ACM, 859--868. Google ScholarDigital Library
- J St BT Evans. 1988. The knowledge elicitation problem: a psychological perspective. Behaviour 8 Information Technology 7, 2 (1988).Google Scholar
- Mary M Gardiner, Leslie L Allee, Peter MJ Brown, John E Losey, Helen E Roy, and Rebecca Rice Smyth. 2012. Lessons from lady beetles: accuracy of monitoring data from US and UK citizen-science programs. Frontiers in Ecology and the Environment 10, 9 (2012), 471--476.Google ScholarCross Ref
- Steven K. Gibb. 2015. Volunteers Against Pollution. Chemical 8 Engineering News (C8EN) 93, 36 (Sept. 2015).Google Scholar
- Mike Harding, Bran Knowles, Nigel Davies, and Mark Rouncefield. 2015. HCI, Civic Engagement 8 Trust. In 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 2833--2842. Google ScholarDigital Library
- John D Hoyt and Harry Wechsler. 1994. Detection of human speech in structured noise. In Acoustics, Speech, and Signal Processing, 1994. ICASSP-94., 1994 IEEE International Conference on, Vol. 2. IEEE, II--237.Google Scholar
- Kuan Lun Huang, Salil S Kanhere, and Wen Hu. 2010. Are you contributing trustworthy data?: the case for a reputation system in participatory sensing. In Modeling, analysis, and simulation of wireless and mobile systems. Google ScholarDigital Library
- Eiman Kanjo. 2010. Noisespy: A real-time mobile phone platform for urban noise monitoring and mapping. Mobile Networks and Applications 15, 4 (2010), 562--574. Google ScholarDigital Library
- Sunyoung Kim, Jennifer Mankoff, and Eric Paulos. 2013. Sensr: Evaluating a Flexible Framework for Authoring Mobile Data-collection Tools for Citizen Science. In Computer Supported Cooperative Work (CSCW ’13). ACM, 10. Google ScholarDigital Library
- Sunyoung Kim, Christine Robson, Thomas Zimmerman, Jeffrey Pierce, and Eben M. Haber. 2011. Creek Watch: Pairing Usefulness and Usability for Successful Citizen Science. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, 2125--2134. Google ScholarDigital Library
- Eli Kintisch. 2011. How to Grow Your Own Army of Citizen Scientists. (24 Feb. 2011).Google Scholar
- Simon Klakegg, Chu Luo, Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2016. Instrumenting Smartphones with Portable NIRS. In Adj. Proceedings UbiComp’16, Workshop in Ubiquitous Mobile Instrumentation (UbiMI). Google ScholarDigital Library
- N.D. Lane, E. Miluzzo, Hong Lu, D. Peebles, T. Choudhury, and A.T. Campbell. 2010. A survey of mobile phone sensing. IEEE Communications Magazine 48, 9 (2010). Google ScholarDigital Library
- B. Laugwitz, T. Held, and M. Schrepp. 2008. Construction and evaluation of a user experience questionnaire. In Symposium of the Austrian HCI and Usability Engineering Group. Springer, 63--76. Google ScholarDigital Library
- Kurt Luther, Scott Counts, Kristin B. Stecher, Aaron Hoff, and Paul Johns. 2009. Pathfinder: An Online Collaboration Environment for Citizen Scientists. In Human Factors in Computing Systems (CHI ’09). ACM, 239--248. Google ScholarDigital Library
- Nicolas Maisonneuve, Matthias Stevens, and Bartek Ochab. 2010. Participatory noise pollution monitoring using mobile phones. In Information Polity. Vol. 15. 51 -- 71. Issue 1. Google ScholarDigital Library
- Sebastian Matyas, Peter Kiefer, Christoph Schlieder, and Sara Kleyer. 2011. Wisdom about the Crowd: Assuring Geospatial Data Quality Collected in Location-Based Games. In International Conference on Entertainment Computing (ICEC 2011). 331--336. Google ScholarDigital Library
- George A. Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review 91 (1956), 81--97.Google ScholarCross Ref
- L. A. Muratori, P. Salomoni, and G. Pau. 2011. Feeling the pack: Strategies for an optimal participatory system to sense and recognize noise pollution. In 2011 IEEE International Conference on Consumer Electronics - Berlin (ICCE -Berlin). 17--21.Google Scholar
- Donald A. Norman. 1983. Design Rules Based on Analyses of Human Error. Commun. ACM 26, 4 (April 1983), 254--258. Google ScholarDigital Library
- Donald A. Norman and Pieter Jan Stappers. 2015. DesignX: Complex Sociotechnical Systems. She Ji 1 (2015), 83--106. Issue 2.Google ScholarCross Ref
- DG Novick and K Ward. 2006. Why Don’t People Read the Manual?. In Int. Conference on Design of Communication (SIGDOC ’06). ACM. Google ScholarDigital Library
- L. Peterson and M. Peterson. 1959. Short-term retention of individual verbal items. Journal of Experimental Psychology 58 (1959).Google Scholar
- Rajib Kumar Rana, Chun Tung Chou, Salil S Kanhere, Nirupama Bulusu, and Wen Hu. 2010. Ear-phone: an end-to-end participatory urban noise mapping system. In 9th ACM/IEEE International Conference on Information Processing in Sensor Networks. 105--116. Google ScholarDigital Library
- Sasank Reddy, Deborah Estrin, and Mani Srivastava. 2010. Recruitment Framework for Participatory Sensing Data Collections. In Pervasive Computing. LNCS, Vol. 6030. Google ScholarDigital Library
- I. Schweizer, R. Bärtl, A. Schulz, F. Probst, and M. Mühläuser. 2011. NoiseMap -- real-time participatory noise maps. In PhoneSense’11.Google Scholar
- L. See, A. Comber, C. Salk, S. Fritz, M. van der Velde, C. Perger, C. Schill, I. McCallum, F. Kraxner, and M. Obersteiner. 2013. Comparing the Quality of Crowdsourced Data Contributed by Expert and Non-Experts. PLoS ONE 8, 7 (2013).Google Scholar
- S Andrew Sheppard and Loren Terveen. 2011. Quality is a verb: the operationalization of data quality in a citizen science community. In Proceedings of the 7th International Symposium on Wikis and Open Collaboration. 29--38. Google ScholarDigital Library
- S. Andrew Sheppard, Andrea Wiggins, and Loren Terveen. 2014. Capturing Quality: Retaining Provenance for Curated Volunteer Monitoring Data. In 17th ACM Conference on Computer Supported Cooperative Work 8 Social Computing (CSCW ’14). ACM, 1234--1245. Google ScholarDigital Library
- F Snik, JHH Rietjens, A Apituley, H Volten, B Mijling, A Di Noia, S Heikamp, RC Heinsbroek, OP Hasekamp, and JM Smit. 2014. Mapping atmospheric aerosols with a citizen science network of smartphone spectropolarimeters. Geophysical Research Letters 41, 20 (2014).Google ScholarCross Ref
- John Sweller. 2002. Visualisation and instructional design. In International Workshop on Dynamic Visualizations and Learning.Google Scholar
- Brett Amy Thelen and Rachel K. Thiet. 2008. Cultivating connection: Incorporating meaningful citizen science into Cape Cod National Seashore’s estuarine research and monitoring programs. Park Science 25, 1 (2008).Google Scholar
- A. Truskinger, Haofan Yang, J. Wimmer, Jinglan Zhang, I. Williamson, and P. Roe. 2011. Large Scale Participatory Acoustic Sensor Data Analysis: Tools and Reputation Models to Enhance Effectiveness. In E-Science. Google ScholarDigital Library
- Kathleen Tuite, Noah Snavely, Dun-yu Hsiao, Nadine Tabing, and Zoran Popovic. 2011. PhotoCity: Training Experts at Large-scale Image Acquisition Through a Competitive Game. In SIGCHI Conference on Human Factors in Computing Systems (CHI ’11). ACM, 10. Google ScholarDigital Library
- VDA-QMC. 2011. Messsystem und Messprozess sind zweierlei. QZ 56, 5 (2011).Google Scholar
- Christopher D Wickens, John Lee, Yili D Liu, and Sallie Gordon-Becker. 2014. Introduction to Human Factors Engineering: Pearson New International Edition. Pearson Higher Ed. Second Edition. Google ScholarDigital Library
- Andrea Wiggins and Yurong He. 2016. Community-based Data Validation Practices in Citizen Science. In 19th ACM Conference on Computer-Supported Cooperative Work 8 Social Computing (CSCW ’16). ACM, 1548--1559. Google ScholarDigital Library
- Patricia Wright. 1981. “The instructions clearly state …” can’t people read? Applied Ergonomics 12 (September 1981), 131--141. Issue 3.Google Scholar
- Poonam Yadav and John Darlington. 2016. Design Guidelines for the User-Centred Collaborative Citizen Science Platforms. arXiv preprint arXiv:1605.00910 (2016).Google Scholar
Index Terms
- Participatory Sensing or Participatory Nonsense?: Mitigating the Effect of Human Error on Data Quality in Citizen Science
Recommendations
Participatory sensing for community building
CHI EA '11: CHI '11 Extended Abstracts on Human Factors in Computing SystemsIn this research, we explore the viability of using participatory sensing as a means to enhance a sense of community. To accomplish this, we are developing and deploying a suite of participatory sensing applications, where users explicitly report on the ...
Integrating participatory sensing in application development practices
FoSER '10: Proceedings of the FSE/SDP workshop on Future of software engineering researchWith the widespread capabilities of commodity mobile devices, applications will increasingly incorporate participatory sensing functionality. Participatory sensing directly involves end-users in collecting (and ultimately sharing) information about the ...
Demonstration abstract: participatory sensing enabled environmental monitoring in smart cities
IPSN '14: Proceedings of the 13th international symposium on Information processing in sensor networksThis work describes a system demonstrator for environmental monitoring based on participatory sensing. Mobile environmental sensors carried by citizens are used to measure pollutant concentrations and provide this data to a Unified Sensing Platform. It ...
Comments