Durga M. Kandasamy
Armando Fox
David Patterson
ABSTRACT
Though crowdsourcing holds great promise, many struggle with framing tasks and determining which members of the crowd should be recruited to obtain reliable output. In some cases, expert knowledge is desired but, given the time and cost constraints of the problem, may not be available. In this case, it would be beneficial to augment the expert input that is available with input from members of the general population. We believe that reduced reliance on experts will in some cases lead to acceptable performance while reducing cost and latency. In this work, we show that we are able to approach the performance of an expert group for an image labeling task, while reducing our reliance on experts by incorporating non-expert responses.
- Get Better Results with Less Effort with Mechanical Turk Masters! www.mechanicalturk.typepad.com, June 2011.Google Scholar
- J. Heer and M. Bostock. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Virtualization Design. CHI, ACM, 2010. Google ScholarDigital Library
- K. Heimerl and B. Hartmann. Communitysourcing: Engaging Local Crowds to Perform Expert Work Via Physical Kiosks. In Submission to CHI 2012. Google ScholarDigital Library
- L. Hong and S. E. Page. Problem Solving by Heterogeneous Agents. volume, 97, pages 123--163. Journal of Economic Theory, 2000.Google Scholar
- D. Shahaf and E. Horvitz. Generalized Task Markets for Human and Machine Computation. AAAI, July 2010.Google ScholarCross Ref
- A. Sorokin and D. Forsyth. Utility Data Annotation with Amazon Mechanical Turk. Computer Vision and Pattern Recognition Workshops, IEEE, June 2008.Google Scholar
Index Terms
- Diversity within the crowd
Recommendations
Crowd Size, Diversity and Performance
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsCrowds are increasingly being adopted to solve complex problems. Size and diversity are two key characteristics of crowds; however their relationship to performance is often paradoxical. To better understand the effects of crowd size and diversity on ...
The Communication Network Within the Crowd
WWW '16: Proceedings of the 25th International Conference on World Wide WebSince its inception, crowdsourcing has been considered a black-box approach to solicit labor from a crowd of workers. Furthermore, the "crowd" has been viewed as a group of independent workers dispersed all over the world. Recent studies based on in-...
Understanding the Microtask Crowdsourcing Experience for Workers with Disabilities: A Comparative View
CSCWMicrotask crowdsourcing holds great potential as an employment opportunity with the flexibility and anonymity that individuals with disability may require. Though prior research has explored the accessibility of crowd work, the lived crowd work ...
Comments