Lucas Bernardi
ABSTRACT
Booking.com is the world's largest online travel agent where millions of guests find their accommodation and millions of accommodation providers list their properties including hotels, apartments, bed and breakfasts, guest houses, and more. During the last years we have applied Machine Learning to improve the experience of our customers and our business. While most of the Machine Learning literature focuses on the algorithmic or mathematical aspects of the field, not much has been published about how Machine Learning can deliver meaningful impact in an industrial environment where commercial gains are paramount. We conducted an analysis on about 150 successful customer facing applications of Machine Learning, developed by dozens of teams in Booking.com, exposed to hundreds of millions of users worldwide and validated through rigorous Randomized Controlled Trials. Following the phases of a Machine Learning project we describe our approach, the many challenges we found, and the lessons we learned while scaling up such a complex technology across our organization. Our main conclusion is that an iterative, hypothesis driven process, integrated with other disciplines was fundamental to build 150 successful products enabled by Machine Learning.
- Ioannis Arapakis, Xiao Bai, and B. Barla Cambazoglu. 2014. Impact of Response Latency on User Behavior in Web Search. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR '14). ACM, New York, NY, USA, 103--112. Google Scholar
Digital Library
- L Bernardi, J Kamps, Y Kiseleva, MJI Mueller, T Bogers, and M Koolen. 2015. The continuous cold start problem in e-commerce recommender systems. In CEUR Workshop Proceedings, Vol. 1448. CEUR-WS. org.Google Scholar
- Alex Deng and Victor Hu. 2015. Diluted treatment effect estimation for trigger analysis in online controlled experiments. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining. ACM, 349--358. Google ScholarDigital Library
- Miroslav Dud'ik, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. In Proceedings of the 28th International Conference on International Conference on Machine Learning (ICML'11). Omnipress, USA, 1097--1104. http://dl.acm.org/citation.cfm?id=3104482.3104620 Google ScholarDigital Library
- Dietmar Jannach, Paul Resnick, Alexander Tuzhilin, and Markus Zanker. 2016. Recommender systems--beyond matrix completion. Commun. ACM , Vol. 59, 11 (2016), 94--102. Google ScholarDigital Library
- Raphael Lopez Kaufman, Jegar Pitchforth, and Lukas Vermeer. 2017. Democratizing online controlled experiments at Booking. com. arXiv preprint arXiv:1710.08217 (2017).Google Scholar
- Timo Kluck. 2016. Using multivariant tests to determine performance impact. (2016). Retrieved Dec 04, 2017 from https://booking.ai/using-multivariant-tests-to-determine-performance-impact-c249ab9bfc16Google Scholar
- Pavel Levin, Nishikant Dhanuka, Talaat Khalil, Fedor Kovalev, and Maxim Khalilov. 2017. Toward a full-scale neural machine translation in production: the Booking. com use case. arXiv preprint arXiv:1709.05820 (2017).Google Scholar
- Xiao-Li Li and Bing Liu. 2005. Learning from positive and unlabeled examples with different data distributions. Machine Learning: ECML 2005 (2005), 218--229. Google ScholarDigital Library
- Athanasios Noulas and M Stafseng Einarsen. 2014. User engagement through topic modelling in travel. Proceeding of the Second Workshop on User Engagement Optimization. 2--7.Google Scholar
- James M Robins, Andrea Rotnitzky, and Lue Ping Zhao. 1994. Estimation of regression coefficients when some regressors are not always observed. Journal of the American statistical Association , Vol. 89, 427 (1994), 846--866.Google ScholarCross Ref
- Kagan Tumer and Joydeep Ghosh. 2003. Bayes error rate estimation using classifier ensembles. International Journal of Smart Engineering System Design , Vol. 5, 2 (2003), 95--109.Google ScholarCross Ref
- Kiri Wagstaff. 2012. Machine learning that matters. arXiv preprint arXiv:1206.4656 (2012). Google ScholarDigital Library
Index Terms
- 150 Successful Machine Learning Models: 6 Lessons Learned at Booking.com
Recommendations
Machine Learning: The State of the Art
The two fundamental problems in machine learning (ML) are statistical analysis and algorithm design. The former tells us the principles of the mathematical models that we establish from the observation data. The latter defines the conditions on which ...
Customer Lifetime Value Analysis Based on Machine Learning
ICISDM '22: Proceedings of the 6th International Conference on Information System and Data MiningCustomer lifetime value (CLV) is a powerful tool to determine the value of customers and filter customers most likely to attrite or most likely to make their first purchase, especially for e-commerce companies. This article reviewed machine learning ...
Machine Learning: Algorithms, Real-World Applications and Research Directions
AbstractIn the current age of the Fourth Industrial Revolution (4IR or Industry 4.0), the digital world has a wealth of data, such as Internet of Things (IoT) data, cybersecurity data, mobile data, business data, social media data, health data, etc. To ...
Comments