Michael S. Bernstein
Greg Little
Björn Hartmann
Mark S. Ackerman
David R. Karger
Abstract
This paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. Authoring tools offer help with pragmatics, but for higher-level help, writers commonly turn to other people. We thus present Soylent, a word processing interface that enables writers to call on Mechanical Turk workers to shorten, proofread, and otherwise edit parts of their documents on demand. To improve worker quality, we introduce the Find-Fix-Verify crowd programming pattern, which splits tasks into a series of generation and review stages. Evaluation studies demonstrate the feasibility of crowdsourced editing and investigate questions of reliability, cost, wait time, and work time for edits.
- Andersen, D.G., Franklin, J., Kaminsky, M., Phanishayee, A., Tan, L., Vasudevan, V. FAWN: A fast array of wimpy nodes. In Proc. SOSP '09 (2009). Google Scholar
Digital Library
- Bigham, J.P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R.C., Miller, R., Tatrowicz, A., White, B., White, S., Yeh, T. VizWiz: Nearly real-time answers to visual questions. In Proc. UIST '10 (2010). Google ScholarDigital Library
- Clarke, J., Lapata, M. Models for sentence compression: A comparison across domains, training requirements and evaluation measures. In Proc. ACL '06 (2006). Google ScholarDigital Library
- Cypher, A. Watch What I Do. MIT Press, Cambridge, MA, 1993.Google Scholar
- Dourish, P., Bellotti, V. Awareness and coordination in shared workspaces. In Proc. CSCW '92 (1992). Google ScholarDigital Library
- Heer, J., Bostock, M. Crowdsourcing graphical perception: Using Mechanical Turk to assess visualization design. In Proc. CHI '10 (2010). Google ScholarDigital Library
- Ishii, H., Ullmer, B. Tangible bits: Towards seamless interfaces between people, bits and atoms. In Proc. UIST '97 (1997).Google Scholar
- Kittur, A., Chi, E.H., Suh, B. Crowdsourcing user studies with Mechanical Turk. In Proc. CHI '08 (2008). Google ScholarDigital Library
- Kukich, K. Techniques for automatically correcting words in text. ACM Comput. Surv. (CSUR) 24, 4 (1992), 377--439. Google ScholarDigital Library
- Le, J., Edmonds, A., Hester, V., Biewald, L. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proc. SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation (2010), 21--26.Google Scholar
- Little, G., Chilton, L., Goldman, M., Miller, R.C. TurKit: Human computation algorithms on Mechanical Turk. In Proc. UIST '10 (2010) ACM Press. Google ScholarDigital Library
- Marcu, D. The Theory and Practice of Discourse Parsing and Summarization. MIT Press, Cambridge, MA, 2000. Google ScholarDigital Library
- Mason, W., Watts, D.J. Financial incentives and the performance of crowds. In Proc. HCOMP '09 (2009) ACM Press. Google ScholarDigital Library
- Quinn, A.J., Bederson, B.B. Human computation: A survey and taxonomy of a growing field. In Proc. CHI '11 (2011) ACM. Google ScholarDigital Library
- Ross, J., Irani, L., Silberman, M.S., Zaldivar, A., Tomlinson, B. Who are the crowdworkers? Shifting demographics in Amazon Mechanical Turk. In alt.chi '10 (2010) ACM Press. Google ScholarDigital Library
- Sala, M., Partridge, K., Jacobson, L., Begole, J. An exploration into activity-informed physical advertising using PEST. In Pervasive '07, volume 4480 of Lecture Notes in Computer Science (Berlin, Heidelberg, 2007), Springer, Berlin, Heidelberg. Google ScholarDigital Library
- Snow, R., O'Connor, B., Jurafsky, D., Ng, A.Y. Cheap and fast---But is it good? Evaluating non-expert annotations for natural language tasks. In Proc. ACL '08 (2008). Google ScholarDigital Library
- Sorokin, A., Forsyth, D. Utility data annotation with Amazon Mechanical Turk. Proc. CVPR '08 (2008).Google ScholarCross Ref
- von Ahn, L., Dabbish, L. Labeling images with a computer game. In CHI '04 (2004). Google ScholarDigital Library
Index Terms
- Soylent: a word processor with a crowd inside
Recommendations
Soylent: a word processor with a crowd inside
UIST '10: Proceedings of the 23nd annual ACM symposium on User interface software and technologyThis paper introduces architectural and interaction patterns for integrating crowdsourced human contributions directly into user interfaces. We focus on writing and editing, complex endeavors that span many levels of conceptual and pragmatic activity. ...
Character and numeral recognition for non-Indic and Indic scripts: a survey
AbstractA collection of different scripts is employed in writing languages throughout the world. Character and numeral recognition of a particular script is a key area in the field of pattern recognition. In this paper, we have presented a comprehensive ...
A survey on Arabic character segmentation
Arabic character segmentation is a necessary step in Arabic Optical Character Recognition (OCR). The cursive nature of Arabic script poses challenging problems in Arabic character recognition; however, incorrectly segmented characters will cause ...
Comments