Evaluating the impact of the Wikipedia Teahouse on newcomer socialization and retention

ABSTRACT

Effective socialization of new contributors is vital for the long-term sustainability of open collaboration projects. Previous research has identified many common barriers to participation. However, few interventions employed to increase newcomer retention over the long term by improving aspects of the onboarding experience have demonstrated success. This study presents an evaluation of the impact of one such intervention, the Wikipedia Teahouse, on new editor survival. In a controlled experiment, we find that new editors invited to the Teahouse are retained at a higher rate than editors who do not receive an invite. The effect is observed for both low-and high-activity newcomers, and for both short- and long-term survival.

References

1. Mark S. Ackerman and David W. McDonald. 1996. Answer Garden 2: Merging Organizational Memory with Collaborative Help. In Proceedings of the 1996 ACM Conference on Computer Supported Cooperative Work (CSCW '96). ACM, New York, NY, USA, 97--105.
2. Ivan Beschastnikh, Travis Kriplean, and David W McDonald. 2008. Wikipedian Self-Governance in Action: Motivating the Policy Lens.. In ICWSM.
3. Moira Burke, Robert Kraut, and Elisabeth Joyce. 2010. Membership Claims and Requests: Conversation-Level Newcomer Socialization Strategies in Online Groups. Small Group Research 41, 1 (2010), 4--40. arXiv:https://doi.org/10.1177/1046496409351936
4. Boreum Choi, Kira Alexander, Robert E. Kraut, and John M. Levine. 2010. Socialization Tactics in Wikipedia and Their Effects. In Proceedings of the 2010 ACM Conference on Computer Supported Cooperative Work (CSCW '10). ACM, New York, NY, USA, 107--116.
5. Giovanni Luca Ciampaglia and Dario Taraborelli. 2015. MoodBar: Increasing New User Retention in Wikipedia Through Lightweight Socialization. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 734--742.
6. Nicolas Ducheneaut. 2005. Socialization in an open source software community: A socio-technical analysis. Computer Supported Cooperative Work (CSCW) 14, 4 (2005), 323--368.
7. Andrea Forte and Cliff Lampe. 2013. Defining, Understanding, and Supporting Open Collaboration: Lessons From the Literature. American Behavioral Scientist 57, 5 (2013), 535--547. arXiv:https://doi.org/10.1177/0002764212469362
8. Lauren Gardener, Panthea Lee, Abigail Ripstra, and Neil Patel Quinn. 2017. New Editor Experiences. https://www.mediawiki.org/wiki/New_Editor_Experiences
9. R Stuart Geiger, Aaron Halfaker, Maryana Pinchuk, and Steven Walling. 2012. Defense Mechanism or Socialization Tactic? Improving. In Proceedings of the Sixth International AAAI Conference on Weblogs and Social Media.
10. Aaron Halfaker. 2013. Wikimedia Foundation Growth Team experiments: Onboarding new Wikipedians. https://meta.wikimedia.org/wiki/Research:Onboarding_new_Wikipedians R@5https://www.mediawiki.org/wiki/ORES
11. Aaron Halfaker, R. Stuart Geiger, Jonathan T. Morgan, and John Riedl. 2013. The Rise and Decline of an Open Collaboration System: How WikipediaâĂŹs Reaction to Popularity Is Causing Its Decline. American Behavioral Scientist 57, 5 (2013), 664--688. arXiv:https://doi.org/10.1177/0002764212469365
12. Aaron Halfaker, R. Stuart Geiger, and Loren G. Terveen. 2014. Snuggle: Designing for Efficient Socialization and Ideological Critique. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 311--320.
13. John M Lachin. 2000. Statistical considerations in the intent-to-treat principle. Contemporary Clinical Trials 21, 3 (2000), 167--189.
14. Cliff Lampe and Erik Johnston. 2005. Follow the (Slash) Dot: Effects of Feedback on New Members in an Online Community. In Proceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work (GROUP '05). ACM, New York, NY, USA, 11--20.
15. Cliff Lampe, Rick Wash, Alcides Velasquez, and Elif Ozkaya. 2010. Motivations to Participate in Online Communities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1927--1936.
16. Jonathan T Morgan. 2012. Teahouse phase two report. https://meta.wikimedia.org/wiki/Research:Teahouse/Phase_2_report
17. Jonathan T. Morgan, Siko Bouterse, Heather Walls, and Sarah Stierch. 2013. Tea and Sympathy: Crafting Positive New User Experiences on Wikipedia. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 839--848.
18. Jonathan T. Morgan and Mark Zachry. 2010. Negotiating with Angry Mastodons: The Wikipedia Policy Environment As Genre Ecology. In Proceedings of the 16th ACM International Conference on Supporting Group Work (GROUP '10). ACM, New York, NY, USA, 165--168.
19. David R. Musicant, Yuqing Ren, James A. Johnson, and John Riedl. 2011. Mentoring in Wikipedia: A Clash of Cultures. In Proceedings of the 7th International Symposium on Wikis and Open Collaboration (WikiSym '11). ACM, New York, NY, USA, 173--182.
20. Sneha Narayan, Jake Orlowitz, Jonathan Morgan, Benjamin Mako Hill, and Aaron Shaw. 2017. The Wikipedia Adventure: Field Evaluation of an Interactive Tutorial for New Users. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1785--1799.
21. Katherine Panciera, Aaron Halfaker, and Loren Terveen. 2009. Wikipedians Are Born, Not Made: A Study of Power Editors on Wikipedia. In Proceedings of the ACM 2009 International Conference on Supporting Group Work (GROUP '09). ACM, New York, NY, USA, 51--60.
22. Maryana Pinchuk. 2012. Wikimedia Foundation Growth Team experiments: Post-edit feedback. https://meta.wikimedia.org/wiki/Research:Post-edit_feedback
23. Jenny Preece, Blair Nonnecke, and Dorine Andrews. 2004. The top five reasons for lurking: improving community experiences for everyone. Computers in human behavior 20, 2 (2004), 201--223.
24. Yuqing Ren, Robert Kraut, Sara Kiesler, and Paul Resnick. 2012. Encouraging commitment in online communities. Building successful online communities: Evidence-based social design (2012), 77--124.
25. Wikimedia Foundation Research. 2014. Research Metric standardization: Surviving new editor. https://meta.wikimedia.org/wiki/Research:Surviving_new_editor
26. Ellen L Rubenstein. 2015. Rituals of introduction and revolving roles: socialization in an online breast cancer community. Library & Information Science Research 37, 4 (2015), 353--362.
27. Igor Steinmacher, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles. 2015. Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 1379--1392.
28. I. Steinmacher, I. Wiese, A. P. Chaves, and M. A. Gerosa. 2013. Why do newcomers abandon open source software projects?. In 2013 6th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE). 25--32.
29. Igor Steinmacher, Igor Scaliante Wiese, Tayana Conte, Marco Aurélio Gerosa, and David Redmiles. 2014. The Hard Life of Open Source Software Project Newcomers. In Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE 2014). ACM, New York, NY, USA, 72--78.
30. Bongwon Suh, Gregorio Convertino, Ed H. Chi, and Peter Pirolli. 2009. The Singularity is Not Near: Slowing Growth of Wikipedia. In Proceedings of the 5th International Symposium on Wikis and Open Collaboration (WikiSym '09). ACM, New York, NY, USA, Article 8, 10 pages.
31. Georg Von Krogh, Sebastian Spaeth, and Karim R Lakhani. 2003. Community, joining, and specialization in open source software innovation: a case study. Research Policy 32, 7 (2003), 1217--1241.
32. Steven Walling and Dario Taraborelli. 2012. Wikimedia Foundation Growth Team experiments: Account creation UX. https://meta.wikimedia.org/wiki/Research:Account_creation_UX
33. Dennis M. Wilkinson. 2008. Strong Regularities in Online Peer Production. In Proceedings of the 9th ACM Conference on Electronic Commerce (EC '08). ACM, New York, NY, USA, 302--309.
34. Haiyi Zhu, Amy Zhang, Jiping He, Robert E. Kraut, and Aniket Kittur. 2013. Effects of Peer Feedback on Contribution: A Field Experiment in Wikipedia. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2253--2262.

Index Terms

• Human-centered computing

  • Collaborative and social computing

    • Empirical studies in collaborative and social computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI