Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision

ABSTRACT

Crowdsourcing systems accomplish large tasks with scale and speed by breaking work down into independent parts. However, many types of complex creative work, such as fiction writing, have remained out of reach for crowds because work is tightly interdependent: changing one part of a story may trigger changes to the overall plot and vice versa. Taking inspiration from how expert authors write, we propose a technique for achieving interdependent complex goals with crowds. With this technique, the crowd loops between reflection, to select a high-level goal, and revision, to decompose that goal into low-level, actionable tasks. We embody this approach in Mechanical Novel, a system that crowdsources short fiction stories on Amazon Mechanical Turk. In a field experiment, Mechanical Novel resulted in higher-quality stories than an iterative crowdsourcing workflow. Our findings suggest that orienting crowd work around high-level goals may enable workers to coordinate their effort to accomplish complex work.

References

1. Foldingstory. http://foldingstory.com/.
2. Alexander, C., Ishikawa, S., and Silverstein, M. Pattern languages. Center for Environmental Structure 2 (1977).
3. Andre, P., Kittur, A., and Dow, S. P. Crowd synthesis: Extracting categories and clusters from complex data. In Proc. CSCW, CSCW '14, ACM (New York, NY, USA, 2014), 989--998.
4. Bayles, D., Orland, T., and Morey, A. Art & fear. Tantor Media, Incorporated, 2012.
5. Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., and Panovich, K. Soylent: A word processor with a crowd inside. In Proc. UIST, UIST '10, ACM (New York, NY, USA, 2010), 313--322.
6. Bigham, J. P., Bernstein, M. S., and Adar, E. Human-computer interaction and collective intelligence. In Collective Intelligence Handbook. MIT Press, 2015.
7. Burroway, J. Imaginative writing: The elements of craft. Longman, 2003.
8. Chan, J., Dang, S., Kremer, P., Guo, L., and Dow, S. Ideagens: A social ideation system for guided crowd brainstorming. In Second AAAI Conference on Human Computation and Crowdsourcing (2014).
9. Chilton, L. B., Little, G., Edge, D., Weld, D. S., and Landay, J. A. Cascade: Crowdsourcing taxonomy creation. In Proc. CHI, CHI '13, ACM (New York, NY, USA, 2013), 1999--2008.
10. d.school, S. Design method: I like, i wish, what if. http://dschool.stanford.edu/wp-content/themes/dschool/method-cards/i-like-i-wish-what-if.pdf.
11. Flower, L., and Hayes, J. R. A cognitive process theory of writing. College composition and communication (1981), 365--387.
12. Fraser, R., and Wickes, G. Aldous huxley: The art of fiction no. 24. The Paris Review 23 (1960).
13. Hahn, N., Chang, J., Kim, J. E., and Kittur, A. The knowledge accelerator: Big picture thinking in small pieces. In Proc. CHI, CHI '16, ACM (New York, NY, USA, 2016), 2258--2270.
14. Hill, B. M., and Monroy-Hernández, A. The cost of collaboration for code and art: Evidence from a remixing community. In Proc. CSCW, CSCW '13, ACM (New York, NY, USA, 2013), 1035--1046.
15. Jansson, D. G., and Smith, S. M. Design fixation. Design studies 12, 1 (1991), 3--11.
16. Kim, H.-C. E., and Eklundh, K. S. Reviewing practices in collaborative writing. Comput. Supported Coop. Work 10, 2 (Jan. 2001), 247--259.
17. Kim, J., Cheng, J., and Bernstein, M. S. Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proc. CSCW, ACM (New York, NY, USA, 2014), 745--755.
18. Kim, J., and Monroy-Hernandez, A. Storia: Summarizing social media content based on narrative theory using crowdsourcing. In Proc. CSCW, CSCW '16 (2016), 1018--1027.
19. Kittur, A., Smus, B., Khamkar, S., and Kraut, R. E. Crowdforge: Crowdsourcing complex work. In Proc. UIST, ACM (New York, NY, USA, 2011), 43--52.
20. Koblin, A. The Johnny Cash Project, 2010. http://www.thejohnnycashproject.com/.
21. Kulkarni, A., Can, M., and Hartmann, B. Collaboratively crowdsourcing workflows with turkomatic. In Proc. CSCW, CSCW '12, ACM (New York, NY, USA, 2012), 1003--1012.
22. Lasecki, W., Miller, C., Sadilek, A., Abumoussa, A., Borrello, D., Kushalnagar, R., and Bigham, J. Real-time captioning by groups of non-experts. In Proc. UIST, UIST '12, ACM (New York, NY, USA, 2012), 23--34.
23. Lasecki, W. S., Kim, J., Rafter, N., Sen, O., Bigham, J. P., and Bernstein, M. S. Apparition: Crowdsourced user interfaces that come to life as you sketch them. In Proc. CHI, CHI '15, ACM (New York, NY, USA, 2015), 1925--1934.
24. Lasecki, W. S., Wesley, R., Nichols, J., Kulkarni, A., Allen, J. F., and Bigham, J. P. Chorus: A crowd-powered conversational assistant. In Proc. UIST, UIST '13, ACM (New York, NY, USA, 2013), 151--162.
25. Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. Exploring iterative and parallel human computation processes. In Proc. HCOMP, HCOMP '10, ACM (New York, NY, USA, 2010), 68--76.
26. Little, G., Chilton, L. B., Goldman, M., and Miller, R. C. Turkit: Human computation algorithms on mechanical turk. In Proc. UIST, UIST '10, ACM (New York, NY, USA, 2010), 57--66.
27. Luther, K., Fiesler, C., and Bruckman, A. Redistributing leadership in online creative collaboration. In Proc. CSCW, CSCW '13, ACM (New York, NY, USA, 2013), 1007--1022.
28. Mason, B., and Thomas, S. A million penguins research report. Institute of Creative Technologies, De Montfort University, Leicester, United Kingdom (2008).
29. Nebeling, M., To, A., Guo, A., de Freitas, A. A., Teevan, J., Dow, S. P., and Bigham, J. P. Wearwrite: Crowd-assisted writing from smartwatches. In Proc. CHI, CHI '16, ACM (New York, NY, USA, 2016), 3834--3846.
30. Noel, S., and Robert, J.-M. Empirical study on collaborative writing: What do co-authors do, use, and like? Comput. Supported Coop. Work 13, 1 (Jan. 2004), 63--89.
31. Pecher, D., and Zwaan, R. A. Grounding cognition: The role of perception and action in memory, language, and thinking. Cambridge University Press, 2005.
32. Retelny, D., Robaszkiewicz, S., To, A., Lasecki, W. S., Patel, J., Rahmati, N., Doshi, T., Valentine, M., and Bernstein, M. S. Expert crowdsourcing with flash teams. In Proc. UIST, UIST '14, ACM (New York, NY, USA, 2014), 75--85.
33. Schon, D. A. The reflective practioner. London: Temple Smith (1983).
34. Sharples, M. An account of writing as creative design. Psychology Press, 1999.
35. Simpson, R., Page, K. R., and De Roure, D. Zooniverse: Observing the world's largest citizen science platform. In Proc. WWW, WWW '14 Companion (2014), 1049--1054.
36. Teevan, J., Iqbal, S. T., and von Veh, C. Supporting collaborative writing with microtasks. In Proc. CHI, CHI '16, ACM (New York, NY, USA, 2016), 2657--2668.
37. Thompson, J. D. Organizations in action: Social science bases of administrative theory. Transaction publishers, 1967.
38. Verroios, V., and Bernstein, M. S. Context trees: Crowdsourcing global understanding from local views. In HCOMP 2014 (November 2014).
39. WeAreDynamo. Guidelines for academic requesters. http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters. {Online; accessed 12-May-2015}.
40. Xu, A., Huang, S.-W., and Bailey, B. Voyant: Generating structured feedback on visual designs using a crowd of non-experts. In Proc. CSCW, ACM (New York, NY, USA, 2014), 1433--1444.
41. Yu, L., and Nickerson, J. V. Cooks or cobblers?: Crowd creativity through combination. In Proc. CHI, ACM (New York, NY, USA, 2011), 1393--1402.
42. Zhang, H., Law, E., Miller, R., Gajos, K., Parkes, D., and Horvitz, E. Human computation tasks with global constraints. In Proc. CHI, CHI '12, ACM (New York, NY, USA, 2012), 217--226.

Index Terms

• Human-centered computing

  • Collaborative and social computing