Alexander J. Quinn
Benjamin B. Bederson
ABSTRACT
The rapid growth of human computation within research and industry has produced many novel ideas aimed at organizing web users to do great things. However, the growth is not adequately supported by a framework with which to understand each new system in the context of the old. We classify human computation systems to help identify parallels between different systems and reveal "holes" in the existing work as opportunities for new research. Since human computation is often confused with "crowdsourcing" and other terms, we explore the position of human computation with respect to these related topics.
- Ask 500 People. http://www.ask500people.com.Google Scholar
- Bederson, B. B., Hu, C., & Resnik, P. Translation by iterative collaboration between monolingual users.GI'10. Google ScholarDigital Library
- Bernstein, M., Miller, R. C., Little, G., Ackerman, M., Hartmann, B., Karger, D. R., & Panovich, K. Soylent: A Word Processor with a Crowd Inside. Proc. UIST 2010. Google ScholarDigital Library
- Biewald, L. Massive multiplayer human computation for fun, money, and survival. XRDS (Dec. 2010), 10--15. Google ScholarDigital Library
- Bigham, J. P., Jayant, C., Ji, H., Little, G., Miller, A., Miller, R. C., Miller, R., Tatrowicz, A., White, B., White, S., & Yeh, T. VizWiz: nearly real-time answers to visual questions. UIST 2010, ACM (2010), 333--342. Google ScholarDigital Library
- Callison-Burch, C., & Dredze, M. Creating Speech and Language Data with Amazon's Mechanical Turk. Workshop on Creating Speech And Language Data With Amazon's Mechanical Turk NAACL HLT, (2010), 1--12. Google ScholarDigital Library
- ChaCha search engine. http://www.chacha.com.Google Scholar
- Chan, K. T., King, I., & Yuen, M. Mathematical modeling of social games. Proc CSE 2009, 1205--1210. Google ScholarDigital Library
- Chandrasekar, R., Chi, E., Chickering, M., Ipeirotis, P. G., Mason, W., Provost, F., Tam, J., von Ahn, Front matter. Proc. SIGKDD HCOMP 2010.Google Scholar
- Chen, K., Wu, C., Chang, Y., & Lei, C. A crowdsourceable QoE evaluation framework for multimedia content. Proc MM 2009, ACM, 491--500. Google ScholarDigital Library
- Chklovski, T. Learner: a system for acquiring commonsense knowledge by analogy. Proc. K-CAP '03. Google ScholarDigital Library
- Chklovski, T. Collecting paraphrase corpora from volunteer contributors. Proc. K-CAP'05, ACM (2005). Google ScholarDigital Library
- Chklovski, T. & Gil, Y. Towards Managing Knowledge Collection from Volunteer Contributors, KCVC 2005.Google Scholar
- Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., & Popovic, Z. Predicting protein structures with a multiplayer online game. Nature, 466:7307, (Aug. 2010), 756--760.Google ScholarCross Ref
- Dryer, D. C., Eisbach, C., & Ark, W. S. At what cost pervasive? a social computing view of mobile computing systems. IBM Systems Journal, (Dec 1999). Google ScholarDigital Library
- Edmunds, P. SwarmSketch. http://swarmsketch.com/.Google Scholar
- Fayyad, U., Piatetsky-Shapiro, P., & Smyth, P. Knowledge Discovery and Data Mining: Towards a Unifying Framework. Proc. KDD 1996.Google Scholar
- Gentry, C., Ramzan, Z., & Stubblebine, S. Secure distributed human computation. Proc EC 2005, 155--164. Google ScholarDigital Library
- Gonçalves, D., Jesus, R., & Correia, N. A gesture based game for image tagging. CHI 2008 EA.Google Scholar
- Help Find Jim. http://www.helpfindjim.com.Google Scholar
- Horvitz, E. & Paek, T. Complementary computing: policies for transferring callers from dialog systems to human receptionists. User Modeling and User-Adapted Interaction 17, 1-2 (Mar. 2007), 159--182. Google ScholarDigital Library
- Howe, Jeff. The Rise of Crowdsourcing. Wired. Jun'06.Google Scholar
- Howe, Jeff. Crowdsourcing: Why the Power of the Crowd is Driving the Future of Business. Crown (2008). Google ScholarDigital Library
- Howe, Jeff. Crowdsourcing: A Definition. http://crowdsourcing.typepad.com.Google Scholar
- Hu, C., Bederson, B. B., & Resnik, P. MonoTrans2: A New Human Computation System to Support Monolingual Translation, Proc. CHI 2011. Google ScholarDigital Library
- Iowa Electronic Markets. http://www.biz.uiowa.edu/iem.Google Scholar
- Ipeirotis, P. G. Analyzing the Amazon Mechanical Turk Marketplace. XRDS 17:2, ACM (Dec. 2010), 16--21. Google ScholarDigital Library
- Ipeirotis, P. G., Chandrasekar, R., & Bennett, P. Report on the human computation workshop. HCOMP 2010.Google Scholar
- Ipeirotis, P. G., Provost, F., & Wang, J. Quality management on Amazon Mechanical Turk. HCOMP'10. Google ScholarDigital Library
- Kittur, A., Suh, B., Pendleton, B. A., & Chi, E. He says, she says:conflict and coordination in Wikipedia. CHI'07. Google ScholarDigital Library
- Koblin, A. Sheep Market. http://thesheepmarket.com.Google Scholar
- Kosorukoff, A. Human based genetic algorithm. Trans. on Systems, Man, and Cybernetics, 5:3464, IEEE (2001).Google Scholar
- Law, E. & von Ahn, L. Input-agreement: a new mechanism for collecting data using human computation games. Proc. CHI 2009, 1197--1206. Google ScholarDigital Library
- Law, E., West, K., Mandel, M., Bay, M., Downie, J. S. Evaluation of algorithms using games: the case of music tagging. Proc. ISMIR 2009, 387--392.Google Scholar
- Lenat, D. B. CYC: a large-scale investment in knowledge infrastructure, CACM, (Nov. 1995), 33--38. Google ScholarDigital Library
- Levy, P. Collective intelligence: Mankind's emerging world in cyberspace. Perseus, Cambridge, MA, 1997. Google ScholarDigital Library
- Licklider, J. C. R. Man-computer symbiosis. IRE Transactions on Human Factors in Electronics, vol. HFE-1 IEEE (Mar. 1960), 4--11.Google ScholarCross Ref
- Lieberman, H., Smith, D., & Teeters, A. Common Consensus: a web-based game for collecting common-sense goals. Proc. IUI 2007.Google Scholar
- Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. Exploring iterative and parallel human computation processes. Proc. HCOMP 2010, 68--76. Google ScholarDigital Library
- LiveOps. http://www.liveops.com.Google Scholar
- Malone, T. W., Laubacher, R., & Dellarocas, C. N. Harnessing crowds: Mapping the genome of collective intelligence. MIT Sloan Research Paper 4732-09, (2009).Google Scholar
- Mataric, M. J. Designing emergent behaviors: from local interactions to collective intelligence. Proc of Animals To Animats 2, MIT Press (1993), 432--441. Google ScholarDigital Library
- Mason, W. & Watts, D. J. Financial incentives and the "performance of crowds". SIGKDD Explorations Newsletter. 11:2, ACM (May 2010), 100--108. Google ScholarDigital Library
- Mechanical Turk. http://mturk.com.Google Scholar
- Nalimov, E. V., Wirth, C., & Haworth, G. M. KQQKQQ and the Kasparov-World Game. ICGA J., (1999)195--212.Google Scholar
- News Futures. http://www.newsfutures.com.Google Scholar
- Page, L., Brin, S., Motwani, R., & Winograd, T. The PageRank citation ranking: Bringing order to the Web. Tech. report, Stanford Digital Libraries Project (1998).Google Scholar
- Parameswaran, M. & Whinston, A. B. Social Computing: An Overview. CAIS 19:37, (2007), 762--780.Google Scholar
- Peha, J. M. & Khamitov, I. M. PayCash: a secure efficient Internet payment system. Proc. ICEC 2003, 125--130. Google ScholarDigital Library
- Quinn, A. & Bederson, B. B. A Taxonomy of Distributed Human Computation. Tech. Rep. HCIL-2009-23, University of Maryland, (2009).Google Scholar
- Quinn, A., Bederson, B. B., Yeh, T., & Lin, J. CrowdFlow: Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility. Tech. Rep HCIL-2010-09, Univ. of Maryland, (2009).Google Scholar
- reCAPTCHA. http://recaptcha.net.Google Scholar
- Resnik, P., Buzek, O., Hu, C., Kronrod, Y., Quinn, A., & Bederson, B. B. Improving Translation via Targeted Paraphrasing, EMNLP 2010, ACL (2010), 127--137. Google ScholarDigital Library
- Schall, D., Truong, H., & Dustdar, S. The Human-Provided Services Framework. CECANDEEE 2008. Google ScholarDigital Library
- Secretan, J., Beato, N., Ambrosio, D. D. B., & Rodriguez, A., Campbell, A., Stanley, K. O. Picbreeder: evolving pictures collaboratively online. Proc CHI 2008. Google ScholarDigital Library
- Sheng, V., Provost, F., & Ipeirotis, P. Get Another Label? improving data quality and data mining using multiple nosy labelers. KDD 2008, 614--622. Google ScholarDigital Library
- Silberman, S. Inside the High Tech Hunt for a Missing Silicon Valley Legend. Wired Magazine. (Jul. 24, 2007). http://wired.com/techbiz/people/magazine/15-08/ff_jimgray.Google Scholar
- Singh, P., Lin, T., Mueller, E. T., Lim, G., Perkins, T., & Zhu, W. L. Open Mind Common Sense: Knowledge Acquisition from the General Public. Lecture Notes in Computer Science, v. 2519, Springer (2002), 1223--1237. Google ScholarDigital Library
- Surowiecki, J. The Wisdom of Crowds. Anchor (2005). Google ScholarDigital Library
- The FACTory. http://game.cyc.com.Google Scholar
- Tong, S. & Koller, D. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2 (Ma. 2002), 45--66. Google ScholarDigital Library
- Turing, A. M. Computing Machinery and Intelligence. Mind, 59:236, Oxford University Press (1950), 433--460.Google Scholar
- von Ahn, L. & Dabbish, L. Labeling images with a computer game. Proc CHI 2004. Google ScholarDigital Library
- von Ahn, L. Human Computation. Doctoral Thesis. UMI Order Number: AAI3205378, Carnegie Mellon University, (2005). Google ScholarDigital Library
- von Ahn, L. Games with a Purpose. Computer, 39:6, IEEE Computer Society (Jun 2006), 92--96. Google ScholarDigital Library
- von Ahn, L. & Dabbish, L. Designing Games with a Purpose. CACM, 51:8, (Aug. 2008), 58--67. Google ScholarDigital Library
- von Ahn, L., Kedia, M, & Blum, M. Verbosity: a game for collecting common-sense facts. Proc CHI 2006. Google ScholarDigital Library
- von Ahn, L., Maurer, B., McMillen, C., Abraham, D., & Blum, M. ReCAPTCHA: human-based character recognition via web security measures. Science, 321:5895, (Sept. 12, 2008), 1465--1468.Google ScholarCross Ref
- Wayland, F. The Limitations Of Human Responsibility. Applewood Books, Bedford, Massachusetts, (1838).Google Scholar
- Westphal, A. J., Butterworth, A. L., Snead, C. J., Craig, N., Anderson, D., Jones, S. M., Brownlee, D. E., Farnsworth, R., Zolensky, M. E. Stardust@ home: A Massively Distributed Public Search for Interstellar Dust in the Stardust Interstellar Dust Collector. Thirty-Sixth Lunar and Planetary Science Conference (2005).Google Scholar
- Wikipedia. http://en.wikipedia.org/wiki/Wikipedia:5P.Google Scholar
- Wolfers, J. & Zitzewitz, E. Prediction markets. Journal of Economic Perspectives, 18:2, (2004), 107--126.Google ScholarCross Ref
- Yang, Y., Zhu, B. B., Guo, R., Yang, L., Li, S., & Yu, N. A comprehensive human computation framework: with application to image labeling. Proc. MM 2008. Google ScholarDigital Library
- Yuen, M., Chen, L., & King, I. A Survey of Human Computation Systems. Proc CSE 2009. Google ScholarDigital Library
Index Terms
- Human computation: a survey and taxonomy of a growing field
Index terms have been assigned to the content through auto-classification.
Recommendations
Emerging theories and models of human computation systems: a brief survey
UbiCrowd '11: Proceedings of the 2nd international workshop on Ubiquitous crowdsouringThe ubiquitous access to computing systems has spurred the design of a variety of human computation systems, each aiming to harness the power of human computation to tackle problems that cannot be solved by today's computers alone. However, the rapid ...
Five design challenges for human computation
NordiCHI '10: Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending BoundariesHuman computation systems, which draw upon human competencies in order to solve hard computational problems, represent a growing interest within HCI. Despite the numerous technical demonstrations of human computation systems, however, there are few ...
An introduction to human computation and games with a purpose
ICWE'13: Proceedings of the 13th international conference on Web EngineeringCrowdsourcing and human computation are novel disciplines that enable the design of computation processes that include humans as actors for task execution. In such a context, Games With a Purpose are an effective mean to channel, in a constructive ...
Comments