Kazushi Ikeda
ABSTRACT
The increasing popularity of mobile crowdsourcing platforms has enabled crowd workers to accept jobs wherever/whenever they are, and also provides opportunity for task requesters to order time/location specific tasks to workers. Since workers on mobile platforms are working on the go, the situation of the workers is expected to influence their performance. However, the effects of mobile worker situations to task performance is an uninvestigated area. In this paper, our research question is, "do worker situations affect task completion, price and quality on mobile crowdsourcing platforms?" We draw on economics and psychology research to examine whether worker situations such as busyness, fatigue and presence of companions affect their performance. Our three-week between-subjects field experiment revealed that worker busyness caused 30.1% relative decrease of task completion rate. Mean accepted task price increased by 7.6% when workers are with companions. Worker fatigue caused 37.4% relative decrease of task quality.
Supplemental Material
- Field Agent, https://www.fieldagent.net/Google Scholar
- Gigwalk, http://www.gigwalk.com/Google Scholar
- Adamczyk, P. D. and Bailey, B. P. If not now, when?: the effects of interruption at different moments within task execution. In Proc. CHI, 2004, 271--278. Google ScholarDigital Library
- Alt, F., Shirazi, A., Schmidt, A., Kramer, U. and Nawaz, Z. Location-based crowdsourcing: extending crowdsourcing to the real world. In Proc. NordiCHI 2010, 13--22. Google ScholarDigital Library
- Antin, J. and Shaw, A. Social desirability bias in reports of motivation for US and India workers on Mechanical Turk. In Proc. CSCW, 2011.Google Scholar
- Armstrong, G. B. and Sopory, P. Effects of background television on phonological and visuo-spatial working memory. Communication Research, 24, 5, 1997, 459--480.Google ScholarCross Ref
- Bailey, B. P. and Konstan, J. A. On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state. Computers in Human Behavior, 22, 4, 2006, 685--708.Google ScholarCross Ref
- Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P. and Kraut, R. Using social psychology to motivate contributions to online communities. In Proc. CSCW, 2004, 212--221. Google ScholarDigital Library
- Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D. and Panovich, K. Soylent: a word processor with a crowd inside. Communications of the ACM, 58, 8, 2015, 85--94.Google ScholarDigital Library
- Boksem, M. A., Meijman, T. F. and Lorist, M. M. Mental fatigue, motivation and action monitoring. Biological psychology, 72, 2, 2006, 123--132.Google Scholar
- Boksem, M. A., Meijman, T. F. and Lorist, M. M. Effects of mental fatigue on attention: an ERP study. Cognitive brain research, 25, 1, 2005, 107--116.Google Scholar
- Cai, C. J., Iqbal, S. T. and Teevan, J. Chain reactions: The impact of order on microtask chains. In Proc. CHI, 2016. Google ScholarDigital Library
- Campagne, A., Pebayle, T. and Muzet, A. Correlation between driving errors and vigilance level: influence of the driver's age. Physiology and Behavior, 80, 2004, 515--524. Google ScholarCross Ref
- Chang Y. J., Paruthi G. and Newman M. W. A field study comparing approaches to collecting annotated activity data in real-world settings. In Proc. UbiComp, 2015, 671--682. Google ScholarDigital Library
- Chatzimilioudis, G., Konstantinidis, A., Laoudias, C. and Zeinalipour-Yazti, D. Crowdsourcing with smartphones. IEEE Internet Computing, 16, 5, 2012, 36--44.Google ScholarDigital Library
- Czerwinski, M., Cutrell, E. and Horvitz, E. Instant messaging: Effects of relevance and timing. In Proc. HCI, 2000, 71--76.Google Scholar
- Dow, S., Kulkarni, A., Klemmer, S. and Hartmann, B. Shepherding the crowd yields better work. In Proc. CSCW, 2012, 1013--1022. Google ScholarDigital Library
- Faber, L. G., Maurits, N. M. and Lorist, M. M. Mental fatigue affects visual selective attention. PLOS one, 7, 10, 2012, e48073.Google Scholar
- Faggiani, A., Gregori, E., Lenzini, L., Luconi, V. and Vecchio, A. Smartphone-based crowdsourcing for network monitoring: Opportunities, challenges, and a case study. IEEE Communications Magazine, 52, 1, 2014, 106--113.Google ScholarCross Ref
- Goldman, A. and Johansson, J. K. Determinants of search for lower prices: An empirical assessment of the economics of information theory. Journal of Consumer Research, 5, 3, 1978, 176--186.Google ScholarCross Ref
- Gould, S., Brumby, D., Cox, A., Gonzélez, V., Salvucci, D. and Taatgen, N. Multitasking and interruptions: a sig on bridging the gap between research on the micro and macro worlds. In Proc. CHI, 2012, 1189--1192. Google ScholarDigital Library
- Gronau, R. The intrafamily allocation of time: The value of the housewives' time. The American Economic Review, 63, 4, 1973, 634--651.Google Scholar
- Gupta, A., Thies, W., Cutrell, E. and Balakrishnan, R. mClerk: enabling mobile crowdsourcing in developing regions. In Proc. CHI, 2012, 1843--1852. Google ScholarDigital Library
- Hembrooke, H., and Geri G. The laptop and the lecture: The effects of multitasking in learning environments. Journal of computing in higher education, 15, 1, 2003, 46--64.Google Scholar
- Ho, J. and Intille, S. S. Using context-aware computing to reduce the perceived burden of interruptions from mobile devices. In. Proc. CHI, 2005, 909--918. Google ScholarDigital Library
- Horton, J. J., and Chilton, L. B. The labor economics of paid crowdsourcing. In Proc. of the 11th ACM conference on Electronic commerce, 2010, 209--218. Google ScholarDigital Library
- Hosio, S., Goncalves, J., Lehdonvirta, V., Ferreira, D. and Kostakos, V. Situated crowdsourcing using a market model. In Proc. UIST, 2014, 55--64. Google ScholarDigital Library
- Hoßfeld, T., Seufert, M., Hirth, M., Zinner, T., TranGia, P. and Schatz, R. Quantification of YouTube QoE via crowdsourcing. In Multimedia, In Proc. ISM, 2011, 494--499.Google Scholar
- Ikeda, K. and Bernstein, M. S. Pay It Backward: PerTask Payments on Crowdsourcing Platforms Reduce Productivity. In Proc. CHI, 2016, 4111--4121. Google ScholarDigital Library
- Iqbal, S. T. and Horvitz, E. Notifications and awareness: A field study of alert usage and preferences. In Proc. CSCW, 2010, 27--30. Google ScholarDigital Library
- Ipeirotis, P. G. Analyzing the amazon mechanical turk marketplace. XRDS: Crossroads, the ACM Magazine for Students, 17, 2, 2010, 16--21.Google Scholar
- Johnson, M. B. Travel time and the price of leisure. Economic Inquiry, 4, 2. 1966, 135--145. Google ScholarCross Ref
- Kaup, F., Jomrich, F. and Hausheer, D. Demonstration of Network Coverage--A Mobile Network Performance Measurement App. In Proc NetSys, 2015.Google Scholar
- Kreifeldt, J. G. and McCarthy, M. E. Interruption as a test of the user-computer interface. In Proc. the 17th Annual Conference on Manual Control, 1981, 655--667.Google Scholar
- Krishna, R., Hata, K., Chen, S., Kravitz, J., Shamma, D. A., Fei-Fei, L. and Bernstein, M. S. Embracing error to enable rapid crowdsourcing. In Proc. CHI, 2016. Google ScholarDigital Library
- Law, E., Yin, M., Joslin Goh, K. C., Terry, M. and Gajos, K. Z. Curiosity killed the cat, but makes crowd work better. In Proc. CHI, 2016.Google Scholar
- Le, J., Edmonds, A., Hester, V. and Biewald, L. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proc. CSE, 2010, 21--26.Google Scholar
- Lippman, S. A. and McCall, J. The economics of job search: A survey. Economic inquiry, 14, 2, 1976, 155--189.Google Scholar
- Liu, Y., Lehdonvirta, V., Alexandrova, T. and Nakajima, T. Drawing on Mobile Crowds via Social Media. Case UbiAsk: Image Based Mobile Social Search Across Languages. Multimedia Systems, 18, 1, 2012, 53--56.Google ScholarCross Ref
- Mason, W. and Watts, D. J. Financial incentives and the "performance of crowds". In Proc. HCOMP, 2009, 77--85. Google ScholarDigital Library
- Marmorstein, H., Grewal, D. and Fishe, R. P. The value of time spent in price-comparison shopping: Survey and experimental evidence. Consumer Research, 19, 1, 1992, 52--61.Google ScholarCross Ref
- Misra, S., Cheng, L., Genevie, J. and Yuan, M. The iPhone effect: the quality of in-person social interactions in the presence of mobile devices. Environment and Behavior, 48, 2, 2016, 275--298.Google ScholarCross Ref
- Musthag, M. and Ganesan, D. Labor dynamics in a mobile micro-task market. In Proc. CHI, 2013, 641--650. Google ScholarDigital Library
- Okoshi, T., Ramos, J., Nozaki, H., Nakazawa, J., Dey, A. K. and Tokuda, H. Attelia: Reducing user's cognitive load due to interruptive notifications on smart phones. In Proc. PerCom, 2015, 96--104.Google Scholar
- Pool, M. M., Koolstra, C. M. and Van der Voort, T. H. Distraction effects of background soap operas on homework performance: An experimental study enriched with observational data. Educational Psychology, 23, 4, 2003, 361--380.Google ScholarCross Ref
- Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T. Valentine, M. and Bernstein, M. S. Expert crowdsourcing with flash teams. In Proc. UIST, 2014, 75--85.Google Scholar
- Roberts, R. Getting the most out of life: The concept of opportunity cost. The Library of Economics and Liberty, 5, 2007.Google Scholar
- Schulze, F. and Groh, G. Conversational context helps improve mobile notification management. In Proc. Mobile HCI, 2016, 518--528. Google ScholarDigital Library
- Shaw, A. D., Horton, J. J. and Chen, D. L. Designing incentives for inexpert human raters. In Proc. CSCW 2011, 275--284. Google ScholarDigital Library
- Speier, C., Valacich, J. S. and Vessey, I. The influence of task interruption on individual decision making: An information overload perspective. Decision Sciences, 30, 2, 1999, 337--360.Google ScholarCross Ref
- Teodoro, R., Ozturk, P., Naaman, M., Mason, W. and Lindqvist, J. The motivations and experiences of the on-demand mobile workforce. In Proc. CSCW, 2014, 236--247. Google ScholarDigital Library
- Vaataja, H., Vainio, T., Sirkkunen, E. and Salo, K. Crowdsourced news reporting: supporting news content creation with mobile phones. In Proc. MobileHCI, 2011, 435--444.Google Scholar
- Vaish, R., Wyngarden, K., Chen, J., Cheung, B. and Bernstein, M. S. Twitch crowdsourcing: crowd contributions in short bursts of time. In Proc. CHI, 2014, 3645--3654. Google ScholarDigital Library
- Yu, L., André, P., Kittur, A. and Kraut, R. A. Comparison of social, learning, and financial strategies on crowd engagement and output quality. In Proc. CSCW 2014, 967--978. Google ScholarDigital Library
- Zick, C. D. and Bryant, W. K. Alternative strategies for pricing home work time. Home Economics Research Journal, 12, 2, 1983, 133--144.Google ScholarCross Ref
- Zijlstra, F. R., Roe, R. A., Leonora, A. B. and Krediet, I. Temporal factors in mental work: Effects of interrupted activities. Occupational and Organizational Psychology, 72, 2, 1999, 163--185.Google ScholarCross Ref
Index Terms
- Crowdsourcing GO: Effect of Worker Situation on Mobile Crowdsourcing Performance
Recommendations
CrowdPickUp: Crowdsourcing Task Pickup in the Wild
We develop and evaluate a new ubiquitous crowdsourcing platform called CrowdPickUp, that combines the advantages of mobile and situated crowdsourcing to overcome their respective limitations. In a 19-day long field study with 70 participants, we ...
Do extra dollars paid-off?: an exploratory study on topcoder
CSI-SE '18: Proceedings of the 5th International Workshop on Crowd Sourcing in Software EngineeringIn general crowdsourcing, different task requesters employ different pricing strategies to balance task cost and expected worker performance. While most existing studies show that increasing incentives tend to benefit crowdsourcing outcomes, i.e. ...
A Server-Assigned Spatial Crowdsourcing Framework
Inaugural IssueWith the popularity of mobile devices, spatial crowdsourcing is rising as a new framework that enables human workers to solve tasks in the physical world. With spatial crowdsourcing, the goal is to crowdsource a set of spatiotemporal tasks (i.e., tasks ...
Comments