From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks

ABSTRACT

A/B testing, also known as bucket testing, split testing, or controlled experiment, is a standard way to evaluate user engagement or satisfaction from a new service, feature, or product. It is widely used among online websites, including social network sites such as Facebook, LinkedIn, and Twitter to make data-driven decisions. At LinkedIn, we have seen tremendous growth of controlled experiments over time, with now over 400 concurrent experiments running per day. General A/B testing frameworks and methodologies, including challenges and pitfalls, have been discussed extensively in several previous KDD work [7, 8, 9, 10]. In this paper, we describe in depth the experimentation platform we have built at LinkedIn and the challenges that arise particularly when running A/B tests at large scale in a social network setting. We start with an introduction of the experimentation platform and how it is built to handle each step of the A/B testing process at LinkedIn, from designing and deploying experiments to analyzing them. It is then followed by discussions on several more sophisticated A/B testing scenarios, such as running offline experiments and addressing the network effect, where one user's action can influence that of another. Lastly, we talk about features and processes that are crucial for building a strong experimentation culture.

References

1. Rubin, Donald B. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
2. Ugander, Johan, Karrer, Brian, Backstrom, Lars and Kleinberg, Jon. Graph cluster randomization: network exposure to multiple universes. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 329--337. ACM, 2013.
3. Katzir, Liran, Liberty, Edo and Somekh Oren. Framework and algorithms for network bucket testing. Proceedings of the 21st international conference on World Wide Web, pages 1029--1036. ACM, 2012.
4. Toulis, Panos and Kao, Edward. Estimation of causal peer influence effects. Proceedings of The 30th International Conference on Machine Learning, pages 1489--1497, 2013.
5. Eckles, Dean, Karrer, Brian and Ugander, Johan. Design and analysis of experiments in networks: Reducing bias from interference. arXiv preprint arXiv:1404.7530, 2014.
6. Aronow, Peter M, and Samii, Cyrus. Estimating average causal effects under general interference. arXiv preprint arXiv:1305.6156, 2013
7. Kohavi, Ron, et al. Trustworthy online controlled experiments: Five puzzling outcomes explained. Proceedings of the 18th Conference on Knowledge Discovery and Data Mining. 2012, www.exp-platform.com/Pages/PuzzingOutcomesExplained.aspx.
8. Tang, Diane, et al. Overlapping Experiment Infrastructure: More, Better, Faster Experimentation. Proceedings 16th Conference on Knowledge Discovery and Data Mining. 2010.
9. Kohavi, Ron, et al. Online Controlled Experiments at Large Scale. KDD 2013: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 2013. http://bit.ly/ExPScale.
10. Kohavi, Ron, et al. Seven Rules of Thumb for Web Site Experimenters. KDD 2014: Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. 2014.
11. Yates, Frank, Sir Ronald Fisher and the Design of Experiments. Biometrics, 20(2):307--321, 1964.
12. Bakshy, Eytan, Echles, Dean and Bernstein, Michael S. Designing and Deploying Online Field Experiments. Proceedings of the 23rd international conference on World Wide Web, pages 283--292, ACM, 2014
13. Kohavi, Ron. et al. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery. February 2009, Vol. 18, 1, pp. 140--181. http://www.exp-platform.com/Pages/hippo_long.aspx.
14. Crook, Thomas, et al. Seven Pitfalls to Avoid when Running Controlled Experiments on the Web. {ed.} Peter Flach and Mohammed Zaki. KDD '09: Proceedings of the 15th ACM SIGKDD international conference on knowledge discovery and data mining. 2009, pp. 1105--1114. http://www.exp-platform.com/Pages/ExPpitfalls.aspx.
15. Ioannidis, John PA. "Why most published research findings are false." PLoS medicine 2.8 (2005): e124.
16. Wacholder, Sholom, et al. "Assessing the probability that a positive report is false: an approach for molecular epidemiology studies." Journal of the National Cancer Institute 96.6 (2004): 434--442.
17. Benjamini, Yoav, and Yosef Hochberg. "Controlling the false discovery rate: a practical and powerful approach to multiple testing." Journal of the Royal Statistical Society. Series B (Methodological) (1995): 289--300.
18. Saaty, Thomas L. "How to make a decision: the analytic hierarchy process."European journal of operational research 48.1 (1990): 9--26.
19. Gui, Huan, Xu, Ya, Bhasin, Anmol, Han Jiawei. Network A/B Testing: From Sampling to Estimation. Proceedings of the 24rd international conference on World Wide Web, ACM, 2015
20. Box, George EP, J. Stuart Hunter, and William G. Hunter. "Statistics for experimenters: design, innovation, and discovery." AMC 10 (2005): 12.
21. Gerber, A. S., and Green, D. P. Field Experiments: Design, Analysis, and Interpretation. WW Norton, 2012
22. Sumbaly, Roshan, et al. "Serving large-scale batch computed data with project voldemort." Proceedings of the 10th USENIX conference on File and Storage Technologies. USENIX Association, 2012.
23. Tate, Ryan. The Software Revolution Behind LinkedIn's Gushing Profits. {Online} http://www.wired.com/2013/04/linkedin-software-revolution
24. Auradkar, Aditya, et al. "Data infrastructure at LinkedIn." Data Engineering (ICDE), 2012 IEEE 28th International Conference on. IEEE, 2012.
25. Kreps, Jay, Neha Narkhede, and Jun Rao. "Kafka: A distributed messaging system for log processing." Proceedings of 6th International Workshop on Networking Meets Databases (NetDB), Athens, Greece. 2011.
26. Naga, Praveen Neppalli, Real-time Analytics at Massive Scale with Pinot. {Online} September 29, 2014 http://engineering.linkedin.com/analytics/real-time-analytics-massive-scale-pinot
27. Fisher, Ronald A. Presidential Address. Sankhya: The Indian Journal of Statistics. 1938, Vol. 4, 1. http://www.jstor.org/stable/40383882.
28. Montgomery, Douglas C. Design and analysis of experiments. John Wiley & Sons, 2008.
29. Betz, Joe, Tagle, Moira. Rest.li:RESTful Service Architecture at Scale. {Online} February, 19, 2013 https://engineering.linkedin.com/architecture/restli-restful-service-architecture-scale
30. Romano, Joseph P. Azeem M. Shaikh and Michael Wolf. 2010b Multiple Testing. New Palgrave Dictionary of Economics. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.418.4975&rep=rep1&type=pdf
31. Wikipedia. Simpson's Paradox. {Online} http://en.wikipedia.org/wiki/Simpson%27s_paradox
32. McFarland, Colin. Experiment!: Website conversion rate optimization with A/B and multivariate testing. s.1. : New Riders, 2012.978-0321834607
33. Eisenberg, Bryan. How to Improve A/B Testing. ClickZ Network. {Online} April 29, 2005. www.clickz.com/clickz/column/1717234/how-improvem-a-b-testing.
34. Vemuri, Srinivas, Varshney, Maneesh, Puttaswamy, Krishna and Liu, Rui. Execution Primitives for Scalable Joins and Aggregations in Map Reduce. Proceedings of the VLDB Endowment, Vol. 7, No. 13
35. Varshney, Maneesh, Vemuri, Srinivas. Open Sourcing Cubert: A High Performance Computation Engine for Complex Big Data Analytics {Online} November 11, 2014 https://engineering.linkedin.com/big-data/open-sourcing-cubert-high-performance-computation-engine-complex-big-data-analytics

Index Terms

• Mathematics of computing

  • Probability and statistics

    • Statistical paradigms

      • Exploratory data analysis