Jorge Goncalves
Niels van Berkel
Vassilis Kostakos
Skip Abstract Section
Abstract
We develop and evaluate a new ubiquitous crowdsourcing platform called CrowdPickUp, that combines the advantages of mobile and situated crowdsourcing to overcome their respective limitations. In a 19-day long field study with 70 participants, we evaluate the quality of work that CrowdPickUp produces. In particular, we measure quality in terms of worker performance in a variety of tasks (requiring local knowledge, location-based, general) while using a number of different quality control mechanisms, and also capture workers’ perceptions of the platform. Our findings show that workers of CrowdPickUp contributed data of comparable quality to previously presented crowdsourcing deployments while at the same time allowing for a wide breadth of tasks to be deployed. Finally, we offer insights towards the continued exploration of this research agenda.
- Florian Alt, Alireza A. S. Shirazi, Albrecht Schmidt, Urs Kramer and Zahid Nawaz. 2010. Location-based Crowdsourcing: Extending Crowdsourcing to the Real World. In NordiCHI, ACM, 13-22. Google Scholar
Digital Library
- Mustafa M. Y. Arslan, Indrajeet Singh, Shailendra Singh, Harsha H. V. Madhyastha, Karthikeyan Sundaresan and Srikanth S. V. Krishnamurthy. 2012. Computing while charging: building a distributed computing infrastructure using smartphones. In Proceedings of the 8th international conference on Emerging networking experiments and technologies, ACM, 193-204. Google ScholarDigital Library
- Jeffrey J. A. Burke, Deborah Estrin, Mark Hansen, Andrew Parker, Nithya Ramanathan, Sasank Reddy and Mani M. B. Srivastava. 2006. Participatory sensing. In First Workshop on World-Sensor-Web: Mobile Device Centric Sensory Networks and Applications at Sensys ‘06.Google Scholar
- Chris Callison-Burch. 2009. Fast, cheap, and creative: evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, 286-295. Google ScholarDigital Library
- Manuel Cebrian, Lorenzo Coviello, Andrea Vattani and Panagiotis Voulgaris. 2012. Finding red balloons with split contracts: crobustness to individuals’ selfishness. In Proceedings of the Forty-Fourth Annual ACM Symposium on Theory of Computing (STOC 2012), 775-788. Google ScholarDigital Library
- Zhao Chen, Rui Fu, Ziyuan Zhao, Zheng Liu, Leihao Xia, Lei Chen, Peng Cheng, Caleb C. C. Cao, Yongxin Tong and Chen C. J. Zhang. 2014. gMission: A General Spatial Crowdsourcing Platform. Proc. VLDB Endow. 7, 13: 1629-1632. Google ScholarDigital Library
- Donkor B. On Social Sentiment and Sentiment Analysis. Retrieved 13/05/2016 from http://brnrd.me/social-sentiment-sentiment-analysis/Google Scholar
- N. Eagle. 2009. txteagle: Mobile Crowdsourcing. In Proceedings of the 3rd International Conference on Internationalization, Design and Global Development: Held as Part of HCI International 2009, Springer-Verlag, 447-456. Google ScholarDigital Library
- Stephen Figlewski. 1979. Subjective information and market efficiency in a betting market. Journal of Political Economy 87, 1: 75-88.Google ScholarCross Ref
- Jorge Goncalves, Michael Feldman, Subingqian Hu, Vassilis Kostakos and Abraham Bernstein. 2017. Task Routing and Assignment in Crowdsourcing based on Cognitive Abilities. In Proceedings of the 26th International Conference on World Wide Web, 1023-1031. Google ScholarDigital Library
- Jorge Goncalves, Denzil Ferreira, Simo Hosio, Yong Liu, Jakob Rogstadius, Hannu Kukka and Vassilis Kostakos. 2013. Crowdsourcing on the spot: altruistic use of public displays, feasibility, performance, and behaviours. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 753-762. Google ScholarDigital Library
- Jorge Goncalves, Simo Hosio, Denzil Ferreira and Vassilis Kostakos. 2014. Game of Words: Tagging Places through Crowdsourcing on Public Displays. In Designing Interactive Systems, 705-714. Google ScholarDigital Library
- Jorge Goncalves, Simo Hosio and Vassilis Kostakos. 2017. Eliciting Structured Knowledge from Situated Crowd Markets. ACM Transactions on Internet Technology, 17(2), article 14. Google ScholarDigital Library
- Jorge Goncalves, Simo Hosio, Yong Liu and Vassilis Kostakos. 2016. Worker performance in a situated crowdsourcing market. Interacting with Computers 28, 5: 612-624.Google ScholarCross Ref
- Jorge Goncalves, Vassilis Kostakos, Simo Hosio, Evangelos Karapanos and Olga Lyra. 2013. IncluCity: Using Contextual Cues to Raise Awareness on Environmental Accessibility. In Proceedings of the ACM SIGACCESS International Conference on Computers and Accessibility, article 17. Google ScholarDigital Library
- Aakar Gupta, William Thies, Edward Cutrell and Ravin Balakrishnan. 2012. mClerk: enabling mobile crowdsourcing in developing regions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, 1843-1852. Google ScholarDigital Library
- Kurtis Heimerl, Brian Gawalt, Kuang Chen, Tapan Parikh and Björn Hartmann. 2012. Community Sourcing: engaging local crowds to perform expert work via physical kiosks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1539-1548. Google ScholarDigital Library
- Simo Hosio, Jorge Goncalves, Vili Lehdonvirta, Denzil Ferreira and Vassilis Kostakos. 2014. Situated Crowdsourcing Using a Market Model. In User Interface Software and Technology, ACM, 55-64. Google ScholarDigital Library
- Simo Hosio, Jorge Goncalves, Niels van Berkel, Simon Klakegg. 2016. Crowdsourcing situated 8 subjective knowledge for decision support. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, ACM, 1478-1483. Google ScholarDigital Library
- Yi-Ching Huang. 2015. Designing a Micro-Volunteering Platform for Situated Crowdsourcing. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work 8 Social Computing, ACM, 73-76. Google ScholarDigital Library
- Yun Huang, Alain Shema and Huichuan Xia. 2016. A proposed genome of mobile and situated crowdsourcing and its design implications for encouraging contributions. International Journal of Human-Computer Studies. Google ScholarDigital Library
- Panagiotis G. Ipeirotis and Evgeniy Gabrilovich. 2014. Quizz: Targeted crowdsourcing with a billion (potential) users. In Proceedings of the 23rd International Conference on World Wide Web, 143-154. Google ScholarDigital Library
- Panagiotis G. Ipeirotis, Foster Provost and Jing Wang. 2010. Quality management on amazon mechanical turk. In Proceedings of the ACM SIGKDD workshop on human computation, 64-67. Google ScholarDigital Library
- Thivya Kandappu, Nikita Jaiman, Randy Tandriansyah, Archan Misra, Shih-Fen Cheng, Cen Chen, Hoong Chuin Lau, Deepthi Chander and Koustov Dasgupta. 2016. TASKer: behavioral insights via campus-based experimental mobile crowd-sourcing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 392-402. Google ScholarDigital Library
- Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work, ACM, 1301-1318. Google ScholarDigital Library
- Shin’ichi Konomi, Niwat Thepvilojanapong, Ryohei Suzuki, Susanna Pirttikangas, Kaoru Sezaki and Yoshito Tobe. 2009. Askus: Amplifying Mobile Actions. In International Conference on Pervasive Computing, Springer Berlin Heidelberg, 202-219. Google ScholarDigital Library
- Nira Liberman, Yaacov Trope and Elena Stephan. 2007. Psychological distance. Social psychology: Handbook of basic principles 2: 353-383.Google Scholar
- Yong Liu, Denzil Ferreira, Jorge Goncalves, Simo Hosio, Pratyush Pandab and Vassilis Kostakos. 2017. Donating Context Data to Science: The Effects of Social Signals and Perceptions on Action-Taking. Interacting with Computers 29, 2: 132-146.Google Scholar
- Jan Lorenz, Heiko Rauhut, Frank Schweitzer and Dirk Helbing. 2011. How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences 108, 22: 9020-9025.Google ScholarCross Ref
- Thomas Ludwig, Christoph Kotthaus, Christian Reuter, Sören S. v. Dongen and Volkmar Pipek. 2016. Situated crowdsourcing during disasters: Managing the tasks of spontaneous volunteers through public displays. International Journal of Human-Computer Studies. Google ScholarDigital Library
- Thomas W. Malone, Robert Laubacher and Chrysanthos Dellarocas. 2010. The collective intelligence genome. MIT Sloan Management Review 51, 3: 21.Google Scholar
- Jörg Müller, Florian Alt, Daniel Michelis and Albrecht Schmidt. 2010. Requirements and design space for interactive public displays. In Proceedings of the international conference on Multimedia, ACM, 1285-1294. Google ScholarDigital Library
- Prayag Narula, Philipp Gutheim, David Rolnitzky, Anand Kulkarni and Bjoern Hartmann. 2011. MobileWorks: A Mobile Crowdsourcing Platform for Workers at the Bottom of the Pyramid. In AAAI Workshop on Human Computation, 121-123. Google ScholarDigital Library
- Eric Paulos, R. J. Honicky and Ben Hooker. 2008. Citizen science: Enabling participatory urbanism. In Handbook of Research on Urban Informatics: The Practice and Promise of the Real-Time City Markus Foth (eds.). IGI Global, Hershey, PA, 414-436.Google Scholar
- Yvonne Rogers, Helen Sharp and Jenny Preece. 2011. Interaction design: beyond human-computer interaction. John Wiley 8 Sons. Google ScholarDigital Library
- Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, J, Laredo and Maja Vukovic. 2011. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. In International AAAI Conference on Web and Social Media, 321-328.Google Scholar
- Reijo Savolainen. 2006. Time as a context of information seeking. Library 8 Information Science Research 28, 1: 110-127.Google Scholar
- Clay Shirky. 2010. Cognitive surplus: How technology makes consumers into collaborators. Penguin.Google Scholar
- Rannie Teodoro, Pinar Ozturk, Mor Naaman, Winter Mason and Janne Lindqvist. 2014. The motivations and experiences of the on-demand mobile workforce. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work 8 Social Computing, ACM, 236-247. Google ScholarDigital Library
- Jacob Thebault-Spieker, Loren G. Terveen and Brent Hecht. 2015. Avoiding the South Side and the Suburbs: The Geography of Mobile Crowdsourcing Markets. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work 8 Social Computing, ACM, 265-275. Google ScholarDigital Library
- Mirjam A. Tuk, Peeter W.J. Verlegh, A. Smidts and Daniel H.J. Wigboldus. 2009. Sales and sincerity: The role of relational framing in word-of-mouth marketing. Journal of Consumer Psychology 19, 1: 38-47.Google ScholarCross Ref
- Rajan Vaish, Keith Wyngarden, Jingshu Chen, Brandon Cheung and Michael S. Bernstein. 2014. Twitch crowdsourcing: crowd contributions in short bursts of time. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, 3645-3654. Google ScholarDigital Library
- Heli Väätäjä, Teija Vainio, Esa Sirkkunen and Kari Salo. 2011. Crowdsourced news reporting: supporting news content creation with mobile phones. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, ACM, 435-444. Google ScholarDigital Library
- Maja Vukovic, Soundar Kumara and Ohad Greenshpan. 2010. Ubiquitous crowdsourcing. In Proceedings of the 12th ACM international conference adjunct papers on Ubiquitous computing - Adjunct, ACM, 523-526. Google ScholarDigital Library
- Jeroen Vuurens, Arjen A. P. de Vries and Carsten Eickhoff. 2011. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR11), 21-26.Google Scholar
Index Terms
- CrowdPickUp: Crowdsourcing Task Pickup in the Wild
Recommendations
Crowdsourcing GO: Effect of Worker Situation on Mobile Crowdsourcing Performance
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsThe increasing popularity of mobile crowdsourcing platforms has enabled crowd workers to accept jobs wherever/whenever they are, and also provides opportunity for task requesters to order time/location specific tasks to workers. Since workers on mobile ...
Do extra dollars paid-off?: an exploratory study on topcoder
CSI-SE '18: Proceedings of the 5th International Workshop on Crowd Sourcing in Software EngineeringIn general crowdsourcing, different task requesters employ different pricing strategies to balance task cost and expected worker performance. While most existing studies show that increasing incentives tend to benefit crowdsourcing outcomes, i.e. ...
WMSC '16: second workshop on mobile and situated crowdsourcing
UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: AdjunctThe proposed workshop seeks to build upon the success of previous workshops at UbiComp 2010 and 2011 on Ubiquitous Crowdsourcing, and UbiComp 2015 on Mobile and Situated Crowdsourcing. Increasingly, researchers and practitioners alike are turning ...
Comments