Carlos Toxtli
Siddharth Suri
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these markets, which ultimately reduces workers' hourly wages. Invisible labor includes finding good tasks, messaging requesters, or managing payments. However, we currently know little about how much time crowd workers actually spend on invisible labor or how much it costs them economically. To ensure a fair and equitable future for crowd work, we need to be certain that workers are being paid fairly for all of the work they do. In this paper, we conduct a field study to quantify the invisible labor in crowd work. We build a plugin to record the amount of time that 100 workers on Amazon Mechanical Turk dedicate to invisible labor while completing 40,903 tasks. If we ignore the time workers spent on invisible labor, workers' median hourly wage was $3.76. But, we estimated that crowd workers in our study spent 33% of their time daily on invisible labor, dropping their median hourly wage to $2.83. We found that the invisible labor differentially impacts workers depending on their skill level and workers' demographics. The invisible labor category that took the most time and that was also the most common revolved around workers having to manage their payments. The second most time-consuming invisible labor category involved hyper-vigilance, where workers vigilantly watched over requesters' profiles for newly posted work or vigilantly searched for labor. We hope that through our paper, the invisible labor in crowdsourcing becomes more visible, and our results help to reveal the larger implications of the continuing invisibility of labor in crowdsourcing.
Supplemental Material
- Ali Alkhatib, Michael S Bernstein, and Margaret Levi. 2017. Examining crowd work and gig work through the historical lens of piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4599--4616.Google Scholar
Digital Library
- Jeffrey Bardzell, Shaowen Bardzell, and Mark A Blythe. 2018. Critical theory and interaction design. MIT Press.Google Scholar
- Veronica Beechey. 1977. Some notes on female wage labour in capitalist production. Capital & Class 1, 3 (1977), 45--66.Google ScholarCross Ref
- Michael S Bernstein, Eytan Bakshy, Moira Burke, and Brian Karrer. 2013. Quantifying the invisible audience in social networks. In Proceedings of the SIGCHI conference on human factors in computing systems. 21--30.Google ScholarDigital Library
- Michael S Bernstein, Joel Brandt, Robert C Miller, and David R Karger. 2011. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 33--42.Google ScholarDigital Library
- Jeffrey P Bigham, Raja Kushalnagar, Ting-Hao Kenneth Huang, Juan Pablo Flores, and Saiph Savage. 2017. On how deaf people might use speech to control devices. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. 383--384.Google ScholarDigital Library
- Monica Blagescu and John Young. 2006. Capacity development for policy advocacy: current thinking and approaches among agencies supporting civil society organisations. Overseas Development Institute London.Google Scholar
- Encyclopaedia Britannica et al. 2008. Britannica concise encyclopedia. Encyclopaedia Britannica, Inc.Google Scholar
- Mayra Buvinic and Elizabeth M King. 2018. Invisible No More? A Methodology and Policy Review of How Time Use Surveys Measure Unpaid Work. United Nations Foundation, Data2X. Available at https://data2x. org/wp-content/uploads/2019/05/Data2X-Invisible-No-More 1 (2018).Google Scholar
- Erran Carmel and Steve Sawyer. 2020. The Five-Dimensional Space of the Futures of Work: A View to 2030. In IFIP Joint Working Conference on the Future of Digital Work: The Challenge of Inequality. Springer, 295--309.Google ScholarCross Ref
- NU CEPAL. 2015. Classification of time-use activities for Latin America (CAUTAL). (2015).Google Scholar
- Le Chen, Alan Mislove, and Christo Wilson. 2015. Peeking beneath the hood of uber. In Proceedings of the 2015 internet measurement conference. 495--508.Google ScholarDigital Library
- Justin Cheng and Michael S Bernstein. 2015. Flock: Hybrid crowd-machine learning classifiers. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. 600--611.Google ScholarDigital Library
- Lucia Ciciolla and Suniya S Luthar. 2019. Invisible household labor and ramifications for adjustment: Mothers as captains of households. Sex Roles 81, 7--8 (2019), 467--486.Google ScholarCross Ref
- Marion Crain, Winifred Poster, and Miriam Cherry. 2016. Invisible labor: Hidden work in the contemporary world. Univ of California Press.Google Scholar
- Emma Crewe and Mr John Young. 2002. Bridging research and policy: context, evidence and links. (2002).Google Scholar
- Brandon Dang, Martin J Riedl, and Matthew Lease. 2018. But who protects the moderators? the case of crowdsourced image moderation. arXiv preprint arXiv:1804.10999 (2018).Google Scholar
- Arlene Kaplan Daniels. 1987. Invisible work. Social Problems 34, 5 (1987), 403--415.Google ScholarCross Ref
- Angela Davis. 1983. The approaching obsolescence of housework: A working-class perspective. Women, Race, and Class (1983), 222--44.Google Scholar
- Valerio De Stefano. 2015. The rise of the just-in-time workforce: On-demand work, crowdwork, and labor protection in the gig-economy. Comp. Lab. L. & Pol'y J. 37 (2015), 471.Google Scholar
- Xuefei Nancy Deng and KD Joshi. 2013. Is crowdsourcing a source of worker empowerment or exploitation? Understanding crowd workers' perceptions of crowdsourcing career. (2013).Google Scholar
- Xuefei Nancy Deng and KD Joshi. 2016. Why individuals participate in micro-task crowdsourcing work environment: Revealing crowdworkers' perceptions. Journal of the Association for Information Systems 17, 10 (2016), 3.Google ScholarCross Ref
- Iqbal Dhaliwal and Caitlin Tulloch. 2012. From research to policy: using evidence from impact evaluations to inform development policy. Journal of Development Effectiveness 4, 4 (2012), 515--536.Google ScholarCross Ref
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In Proceedings of the 24th international conference on world wide web. 238--247.Google ScholarDigital Library
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G. Ipeirotis, and Philippe Cudré-Mauroux. 2015. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. In Proceedings of the 24th International Conference on World Wide Web (Florence, Italy) (WWW '15 Companion). Association for Computing Machinery, New York, NY, USA, 617. https://doi.org/10.1145/2740908.2744109Google ScholarDigital Library
- Catherine D'Ignazio and Lauren F Klein. 2020. Data feminism. Mit Press.Google Scholar
- Carl DiSalvo, Tom Jenkins, and Thomas Lodato. 2016. Designing speculative civics. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 4979--4990.Google ScholarDigital Library
- Arindrajit Dube, Jeff Jacobs, Suresh Naidu, and Siddharth Suri. 2020. Monopsony in Online Labor Markets. American Economic Review: Insights 2, 1 (March 2020), 33--46. https://doi.org/10.1257/aeri.20180150Google ScholarCross Ref
- Benjamin Duke and UK April. 2020. The effects of the COVID-19 crisis on the gig economy and zero hour contracts. Group 24 (2020), 8.Google Scholar
- Margrit Eichler and Patrizia Albanese. 2007. What is household work? A critique of assumptions underlying empirical studies of housework and an alternative approach. The Canadian Journal of Sociology/Cahiers Canadiens de Sociologie (2007), 227--258.Google Scholar
- Fairwork. 2020. The Gig Economy and Covid-19: Fairwork Report on Platform Policies. Fairwork Report on Platform Policies, Oxford,United Kingdom 24 (2020), 8.Google Scholar
- Shaoyang Fan, Ujwal Gadiraju, Alessandro Checco, and Gianluca Demartini. 2020. CrowdCO-OP: Sharing Risks and Rewards in Crowdsourcing. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1--24.Google ScholarDigital Library
- Silvia Federici. 1975. Wages against housework. Falling Wall Press Bristol.Google Scholar
- Casey Fiesler. 2018. Owning the Servers: A Design Fiction Exploring the Transformation of Fandom into ?Our Own'. Transformative Works and Cultures 28 (2018).Google Scholar
- Casey Fiesler. 2021. Innovating Like an Optimist, Preparing Like a Pessimist: Ethical Speculation and the Legal Imagination. Colorado Technology Law Journal 19, 1 (2021).Google Scholar
- Ailbhe Finnerty, Pavel Kucherbaev, Stefano Tranquillini, and Gregorio Convertino. 2013. Keep It Simple: Reward and Task Design in Crowdsourcing. In Proceedings of the Biannual Conference of the Italian Chapter of SIGCHI (Trento, Italy) (CHItaly '13). Association for Computing Machinery, New York, NY, USA, Article 14, 4 pages. https: //doi.org/10.1145/2499149.2499168Google ScholarDigital Library
- Claudia Flores-Saviaga, Yuwen Li, Benjamin Hanrahan, Jeffrey Bigham, and Saiph Savage. 2020. The Challenges of Crowd Workers in Rural and Urban America. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 8. 159--162.Google ScholarCross Ref
- Maria S Floro and Hitomi Komatsu. 2011. Gender and work in South Africa: what can time-use data reveal? Feminist Economics 17, 4 (2011), 33--66.Google ScholarCross Ref
- Maria S Floro and Anant Pichetpongsa. 2010. Gender, work intensity, and well-being of Thai home-based workers. Feminist Economics 16, 3 (2010), 5--44.Google ScholarCross Ref
- Nancy Folbre and Jayoung Yoon. 2007. What is child care? Lessons from time-use surveys of major English-speaking countries. Review of Economics of the Household 5, 3 (2007), 223--248.Google ScholarCross Ref
- Harley Frazis and Jay Stewart. 2012. How to think about time-use data: What inferences can we make about long-and short-run time use from time diaries? Annals of Economics and Statistics/Annales d'économie et de statistique (2012), 231--245.Google Scholar
- Ujwal Gadiraju, Alessandro Checco, Neha Gupta, and Gianluca Demartini. 2017. Modus operandi of crowd workers: The invisible role of microtask work environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1--29.Google ScholarDigital Library
- Ujwal Gadiraju, Jie Yang, and Alessandro Bozzon. 2017. Clarity is a worthwhile quality: On the role of task clarity in microtask crowdsourcing. In Proceedings of the 28th ACM Conference on Hypertext and Social Media. 5--14.Google ScholarDigital Library
- Ruijiang Gao, Maytal Saar-Tsechansky, Maria De-Arteaga, Ligong Han, Min Kyung Lee, and Matthew Lease. 2021. Human-AI Collaboration with Bandit Feedback. arXiv preprint arXiv:2105.10614 (2021).Google Scholar
- Martha Garcia-Murillo and Ian MacInnes. 2019. The impact of AI on employment: a historical account of its evolution. (2019).Google Scholar
- Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.Google Scholar
- Thomas Gillier, Cédric Chaffois, Mustapha Belkhouja, Yannig Roth, and Barry L Bayus. 2018. The effects of task instructions in crowdsourcing innovative ideas. Technological Forecasting and Social Change 134 (2018), 35--44.Google ScholarCross Ref
- Miriam Glucksmann. 2016. Completing and complementing: The work of consumers in the division of labour. Sociology 50, 5 (2016), 878--895.Google ScholarCross Ref
- Elham Shafiei Gol, Mari-Klara Stein, and Michel Avital. 2018. Why Take the Risk?: Motivations of Highly SkilledWorkers to Participate in Crowdworking Platforms. In Proceedings of the 39th International Conference on Information Systems (ICIS). Association for Information Systems. AIS Electronic Library (AISeL).Google Scholar
- Patricia A Gouthro. 2009. Neoliberalism, lifelong learning, and the homeplace: problematizing the boundaries of ?public'and ?private'to explore women's learning experiences. Studies in Continuing Education 31, 2 (2009), 157--172.Google ScholarCross Ref
- Mary L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Eamon Dolan Books.Google Scholar
- Harish Guda and Upender Subramanian. 2019. Your uber is arriving: Managing on-demand workers through surge pricing, forecast communication, and worker incentives. Management Science 65, 5 (2019), 1995--2014.Google ScholarDigital Library
- Jonathan Hall, Cory Kendrick, and Chris Nosko. 2015. The effects of Uber's surge pricing: A case study. The University of Chicago Booth School of Business (2015).Google Scholar
- Lei Han, Eddy Maddalena, Alessandro Checco, Cristina Sarasua, Ujwal Gadiraju, Kevin Roitero, and Gianluca Demartini. 2020. Crowd worker strategies in relevance judgment tasks. In Proceedings of the 13th International Conference on Web Search and Data Mining. 241--249.Google ScholarDigital Library
- Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019. All those wasted hours: On task abandonment in crowdsourcing. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 321--329.Google ScholarDigital Library
- Lingxin Hao, Daniel Q Naiman, and Daniel Q Naiman. 2007. Quantile regression. Number 149. Sage.Google Scholar
- Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P Bigham. 2018. A data-driven analysis of workers' earnings on Amazon Mechanical Turk. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--14.Google ScholarDigital Library
- Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Benjamin V Hanrahan, Jeffrey P Bigham, and Chris Callison-Burch. 2019. Worker demographics and earnings on amazon mechanical turk: An exploratory analysis. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1--6.Google ScholarDigital Library
- Erin Hatton. 2017. Mechanisms of invisibility: rethinking the concept of invisible work. Work, employment and society 31, 2 (2017), 336--351.Google Scholar
- Danula Hettiachchi, Niels Van Berkel, Vassilis Kostakos, and Jorge Goncalves. 2020. CrowdCog: A Cognitive skill based system for heterogeneous task assignment and recommendation in crowdsourcing. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1--22.Google ScholarDigital Library
- Paul Hitlin. 2016. Research in the Crowdsourcing Age, a Case Study: How Scholars, Companies and Workers are Using Mechanical Turk, a" Gig Economy" Platform, for Tasks Computers Can't Handle. Pew Research Center.Google Scholar
- Sara Horowitz. 2015. The Costs of Nonpayment. Retrieved June 26, 2019 from https://blog.freelancersunion.org/ 2015/12/10/costs-nonpayment/Google Scholar
- Ting-Hao Kenneth Huang and Jeffrey P Bigham. 2017. A 10-month-long deployment study of on-demand recruiting for low-latency crowdsourcing. In Fifth AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
- Mohammad Hossein Jarrahi. 2018. Artificial intelligence and the future of work: Human-AI symbiosis in organizational decision making. Business Horizons 61, 4 (2018), 577--586.Google ScholarCross Ref
- Mohammad Hossein Jarrahi, Gemma Newlands, Brian Butler, Saiph Savage, Christoph Lutz, Michael Dunn, and Steve Sawyer. 2021. Flexible work and personal digital infrastructures. Commun. ACM 64, 7 (2021), 72--79.Google ScholarDigital Library
- Martin Jay. 1996. The dialectical imagination: A history of the Frankfurt School and the Institute of Social Research, 1923--1950. Vol. 10. Univ of California Press.Google Scholar
- Toni Kaplan, Susumu Saito, Kotaro Hara, and Jeffrey P Bigham. 2018. Striving to earn more: a survey of work strategies and tool use among crowd workers. In Sixth AAAI Conference on Human Computation and Crowdsourcing.Google ScholarCross Ref
- Patrick Gage Kelley. 2010. Conducting usable privacy & security studies with amazon's mechanical turk. In Symposium on Usable Privacy and Security (SOUPS)(Redmond, WA.Google Scholar
- Sara Constance Kingsley, Mary L. Gray, and Siddharth Suri. 2015. Accounting for Market Frictions and Power Asymmetries in Online Labor Markets. Policy & Internet 7, 4 (2015), 383--400. https://doi.org/10.1002/poi3.111Google ScholarCross Ref
- Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems. 453--456.Google ScholarDigital Library
- Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012. CrowdWeaver: visually managing complex crowd work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work. 1033--1036.Google ScholarDigital Library
- Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. 1301--1318.Google ScholarDigital Library
- Siou Chew Kuek, Cecilia Paradi-Guilford, Toks Fayomi, Saori Imaizumi, Panos Ipeirotis, Patricia Pina, and Manpreet Singh. 2015. The global opportunity in online outsourcing. (2015).Google Scholar
- Laura Lascau, Sandy JJ Gould, Anna L Cox, Elizaveta Karmannaya, and Duncan P Brumby. 2019. Monotasking or Multitasking: Designing for Crowdworkers' Preferences. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--14.Google ScholarDigital Library
- Walter S Lasecki, Christopher D Miller, and Jeffrey P Bigham. 2013. Warping time for more effective real-time crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2033--2036.Google ScholarDigital Library
- Walter S Lasecki, Jeffrey M Rzeszotarski, Adam Marcus, and Jeffrey P Bigham. 2015. The effects of sequence and delay on crowd work. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1375--1378.Google ScholarDigital Library
- Zhifang Liao, Dayu He, Zhijie Chen, Xiaoping Fan, Yan Zhang, and Shengzong Liu. 2018. Exploring the characteristics of issue-related behaviors in github using visualization techniques. IEEE Access 6 (2018), 24003--24015.Google ScholarCross Ref
- Linda Lim. 1983. Capitalism, imperialism, and patriarchy: the dilemma of third-world women workers in multinational factories. Women, men and the international division of labor (1983), 70--91.Google Scholar
- Joseph Lindley and Paul Coulton. 2015. Back to the future: 10 years of design fiction. In Proceedings of the 2015 British HCI Conference. 210--211.Google ScholarDigital Library
- Conor Linehan, Ben J Kirman, Stuart Reeves, Mark A Blythe, Theresa Jean Tanenbaum, Audrey Desjardins, and Ron Wakkary. 2014. Alternate endings: using fiction to explore design futures. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. 45--48.Google Scholar
- Anna Lioznova, Alexey Drutsa, Vladimir Kukushkin, and Anastasia Bezzubtseva. 2020. Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3172--3182.Google ScholarDigital Library
- Caitlin Lustig, Sean Rintel, Liane Scult, and Siddharth Suri. 2020. Stuck in the middle with you: The Transaction Costs of Corporate Employees Hiring Freelancers. Proceedings of the ACM on Human-Computer Interaction 4, CSCW1 (2020), 1--28.Google ScholarDigital Library
- VK Chaithanya Manam and Alexander J Quinn. 2018. Wingit: Efficient refinement of unclear task instructions. In Sixth AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
- Herbert Marcuse. 1964. One-dimensional man: Studies in the ideology of advanced industrial society. Beacon Press.Google Scholar
- Herbert Marcuse. 1966. The individual in the great society. Collected Papers of Herbert Marcuse Vol. 2: Towards a Critical Theory of Society (1966), 59--80.Google Scholar
- Anoush Margaryan. 2016. Understanding crowdworkers' learning practices. (2016).Google Scholar
- David Martin, Benjamin V Hanrahan, Jacki O'Neill, and Neha Gupta. 2014. Being a turker. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 224--235.Google ScholarDigital Library
- Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016. Taking a HIT: Designing around rejection, mistrust, risk, and workers' experiences in Amazon Mechanical Turk. In Proceedings of the 2016 CHI conference on human factors in computing systems. 2271--2282.Google ScholarDigital Library
- Yann Moulier-Boutang. 2011. Cognitive capitalism. Polity.Google Scholar
- Gemma Newlands and Christoph Lutz. 2020. Crowdwork and the mobile underclass: Barriers to participation in India and the United States. new media & society (2020), 1461444820901847.Google Scholar
- Sihang Qiu, Ujwal Gadiraju, and Alessandro Bozzon. 2020. Improving worker engagement through conversational microtask crowdsourcing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--12.Google ScholarDigital Library
- Alex Rosenblat. 2018. Uberland: How algorithms are rewriting the rules of work. Univ of California Press.Google ScholarCross Ref
- Jeffrey Rzeszotarski and Aniket Kittur. 2012. CrowdScape: interactively visualizing user behavior and output. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 55--62.Google ScholarDigital Library
- Jeffrey M Rzeszotarski, Ed H Chi, Praveen Paritosh, and Peng Dai. 2013. Inserting micro-breaks into crowdsourcing workflows. (2013).Google Scholar
- JeffreyMRzeszotarski and Aniket Kittur. 2011. Instrumenting the crowd: using implicit behavioral measures to predict task performance. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 13--22.Google Scholar
- Susumu Saito, Chun-Wei Chiang, Saiph Savage, Teppei Nakano, Tetsunori Kobayashi, and Jeffrey Bigham. 2019. Turkscanner: Predicting the hourly wage of microtasks. In The World Wide Web Conference. 3187--3193.Google ScholarDigital Library
- Shruti Sannon and Dan Cosley. 2019. Privacy, Power, and Invisible Labor on Amazon Mechanical Turk. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.Google ScholarDigital Library
- Saiph Savage, Chun Wei Chiang, Susumu Saito, Carlos Toxtli, and Jeffrey Bigham. 2020. Becoming the Super Turker: IncreasingWages via a Strategy from High EarningWorkers. In Proceedings of TheWeb Conference 2020. 1241--1252.Google Scholar
- Saiph Savage and Mohammad H Jarrahi. 2020. Solidarity and AI for Transitioning to Crowd Work during COVID-19. In The New Future of Work Symposium 2020.Google Scholar
- Nigel R Shadbolt, Daniel A Smith, Elena Simperl, Max Van Kleek, Yang Yang, and Wendy Hall. 2013. Towards a classification framework for social machines. In Proceedings of the 22nd International Conference on World Wide Web. 905--912.Google ScholarDigital Library
- M Six Silberman, Bill Tomlinson, Rochelle LaPlante, Joel Ross, Lilly Irani, and Andrew Zaldivar. 2018. Responsible research with crowds: pay crowdworkers at least minimum wage. Commun. ACM 61, 3 (2018), 39--41.Google ScholarDigital Library
- RACHEL N SIMONS, DANNA GURARI, and KENNETH R FLEISCHMANN. 2020. " I Hope This Is Helpful": Understanding Crowdworkers' Challenges and Motivations for an Image Description Task. ACM CSCW (2020).Google Scholar
- Aaron Smith. 2016. Labor platforms: Technology-enabled "gig work'. Pew Research Center, Internet and Technology. November 17 (2016).Google Scholar
- Daniel Spurk and Caroline Straub. 2020. Flexible employment relationships and careers in times of the COVID-19 pandemic.Google Scholar
- Daniel Start and Ingie Hovland. 2004. Tools for policy impact: a handbook for researchers. Overseas Development Institute London.Google Scholar
- Hao Su, Jia Deng, and Li Fei-Fei. 2012. Crowdsourcing annotations for visual object detection. In Workshops at the Twenty-Sixth AAAI Conference on Artificial Intelligence.Google Scholar
- Will Sutherland and Mohammad Hossein Jarrahi. 2018. The sharing economy and digital platforms: A review and research agenda. International Journal of Information Management 43 (2018), 328--341.Google ScholarCross Ref
- Tiziana Terranova. 2000. Free labor: Producing culture for the digital economy. Social text 18, 2 (2000), 33--58.Google Scholar
- Carlos Toxtli and Saiph Savage. 2020. Meta-Gig: Empowering anyone to create crowd marketplaces. Avances en Interacción Humano-Computadora 1 (2020), 11--19.Google Scholar
- Hai Wang and Hai Yang. 2019. Ridesourcing systems: A framework and review. Transportation Research Part B: Methodological 129 (2019), 122--155.Google ScholarCross Ref
- Vanesa Weyrauch and Gala Díaz Langou. 2011. Sound expectations: from impact evaluations to policy change. International Initiative for Impact Evaluation Working Paper 12 (2011).Google Scholar
- Mark E Whiting, Grant Hugh, and Michael S Bernstein. 2019. Fair Work: Crowd Work Minimum Wage with One Line of Code. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. 197--206.Google ScholarCross Ref
- Rebecca Whiting and Gillian Symon. 2020. Digi-housekeeping: the invisible work of flexibility. Work, Employment and Society (2020), 0950017020916192.Google Scholar
- Alex C Williams, Gloria Mark, Kristy Milland, Edward Lank, and Edith Law. 2019. The Perpetual Work Life of Crowdworkers: How Tooling Practices Increase Fragmentation in Crowdwork. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1--28.Google ScholarDigital Library
- Richmond Y Wong, Ellen Van Wyk, and James Pierce. 2017. Real-fictional entanglements: Using science fiction and design fiction to interrogate sensing technologies. In Proceedings of the 2017 Conference on Designing Interactive Systems. 567--579.Google ScholarDigital Library
- Meng-Han Wu and Alexander Quinn. 2017. Confusing the crowd: Task instruction quality on amazon mechanical turk. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 5.Google ScholarCross Ref
- Ming Yin, Siddharth Suri, and Mary L. Gray. 2018. Running Out of Time: The Impact and Value of Flexibility in On-Demand Crowdwork. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA.Google Scholar
Index Terms
- Quantifying the Invisible Labor in Crowd Work
Recommendations
Privacy, Power, and Invisible Labor on Amazon Mechanical Turk
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsTasks on crowdsourcing platforms such as Amazon Mechanical Turk often request workers' personal information, raising privacy risks that may be exacerbated by requester-worker power dynamics. We interviewed 14 workers to understand how they navigate ...
Exploring Blue-and White-Collar Employees' Well-Being at Work System: Differences in Indicators of Physical and Psychosocial Conditions of Occupational Groups
The aim of this study was to examine occupational group-related differences in well-being at work WBW indicators ranging from real accidents, absences and retirement to experienced pleasure at work. Occupational group included two categories: blue-and ...
An Analysis of the Use of Qualifications on the Amazon Mechanical Turk Online Labor Market
Several human computation systems use crowdsourcing labor markets to recruit workers. However, it is still a challenge to guarantee that the results produced by workers have a high enough quality. This is particularly difficult in markets based on micro-...
Comments