Enric Pujol
Georgios Smaragdakis
ABSTRACT
Large content providers, known as hyper-giants, are responsible for sending the majority of the content traffic to consumers. These hyper-giants operate highly distributed infrastructures to cope with the ever-increasing demand for online content. To achieve commercial-grade performance of Web applications, enhanced end-user experience, improved reliability, and scaled network capacity, hyper-giants are increasingly interconnecting with eyeball networks at multiple locations. This poses new challenges for both (1) the eyeball networks having to perform complex inbound traffic engineering, and (2) hyper-giants having to map end-user requests to appropriate servers.
We report on our multi-year experience in designing, building, rolling-out, and operating the first-ever large scale system, the Flow Director, which enables automated cooperation between one of the largest eyeball networks and a leading hyper-giant. We use empirical data collected at the eyeball network to evaluate its impact over two years of operation. We find very high compliance of the hyper-giant to the Flow Director's recommendations, resulting in (1) close to optimal user-server mapping, and (2) 15% reduction of the hyper-giant's traffic overhead on the ISP's long-haul links, i.e., benefits for both parties and end-users alike.
- B. Ager, W. Mühlbauer, G. Smaragdakis, and S. Uhlig. 2010. Comparing DNS resolvers in the wild. In ACM IMC 2010. 15--21.Google Scholar
- Akamai. 2019. https://www.akamai.com/us/en/resources/cdn-solutions.jsp. (2019).Google Scholar
- Akamai. 2019. Akamai Facts and Figures. https://www.akamai.com/us/en/about/facts-figures.jsp. (2019).Google Scholar
- R. Alimi, R. Penno, and Y. Yang. 2011. ALTO Protocol. IETF RFC 7285. (2011).Google Scholar
- Amazon. 2019. https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html. (2019).Google Scholar
- T. Böttger, F. Cuadrado, and S. Uhlig. 2018. Looking for Hypergiants in PeeringDB. ACM CCR 48, 3 (2018).Google Scholar
- C. Labovitz. 2019. Internet Traffic 2009-2019. APRICOT 2019. (2019).Google Scholar
- M. Caesar and J. Rexford. 2005. BGP Routing Policies in ISP networks. IEEE network 19, 6 (2005), 5--11.Google Scholar
- M. Calder, X. Fan, Z. Hu, E. Katz-Bassett, J. Heidemann, and R. Govindan. 2013. Mapping the Expansion of Google's Serving Infrastructure. In ACM IMC. 313--326.Google Scholar
- M. Calder, M. Schroder, R. Gao, R. Stewart, J. Padhye, R. Mahajan, G. Ananthanarayanan, and E. Katz-Bassett. 2018. Odin: Microsoft's Scalable Fault-Tolerant CDN Measurement System. In Proc. USENIX NSDI 2018. 501--517.Google Scholar
- Amazon CloudFront CDN. 2019. https://aws.amazon.com/cloudfront/. (2019).Google Scholar
- F. Chen, R. K. Sitaraman, and M. Torres. 2015. End-User Mapping: Next Generation Request Routing for Content Delivery. In Proc. ACM SIGCOMM 2015.Google Scholar
- Y. C. Chiu, B. Schlinker, A. B. Radhakrishnan, E. Katz-Bassett, and R. Govindan. 2015. Are We One Hop Away from a Better Internet?. In ACM IMC.Google Scholar
- Cisco. 2012. Introduction to Cisco IOS NetFlow - A Technical Overview. https://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-netflow/prod_white_paper0900aecd80406232.html. (2012).Google Scholar
- Cisco. 2019. Cisco Visual Networking Index: Forecast and Trends, 2017--2022 White Paper. https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-741490.html. (2019).Google Scholar
- B. Claise, B. Trammell, and P. Aitken. 2013. RFC 7011: Specification of the IPFIX Protocol for the Exchange of Flow Information. (2013).Google Scholar
- D. D. Clark, J. Wroclawski, K. Sollins, and R. Braden. 2002. Tussle in Cyberspace: Defining Tomorrow's Internet. In Proc. ACM SIGCOMM 2002. 347--356.Google ScholarDigital Library
- C. Contavalli, W. van der Gaast, D. Lawrence, and W. Kumari. 2016. Client Subnet in DNS Queries. IETF RFC 7871. (2016).Google Scholar
- CZ.NIC Labs. 2019. The BIRD Internet Routing Daemon. http://bird.network.cz. (2019).Google Scholar
- A. Dhamdhere, D. D. Clark, A. Gamero-Garrido, M. Luckie, R. K. P. Mok, G. Akiwate, K. Gogia, V. Bajpai, A. C. Snoeren, and kc claffy. 2018. Inferring Persistent Interdomain Congestion. In Proc. ACM SIGCOMM.Google Scholar
- F. Dobrian, A. Awan, D. Joseph, A. Ganjam, J. Zhan, V. Sekar, I. Stoica, and H. Zhang. 2011. Understanding the Impact of Video Quality on User Engagement. In Proc. ACM SIGCOMM 2011. 362--373.Google Scholar
- X. Fan, E. Katz-Bassett, and J. Heidemann. 2015. Assessing affinity between users and CDN sites. In TMA.Google Scholar
- P. Faratin, D. D. Clark, S. Bauer, W. Lehr, P. Gilmore, and A. Berger. 2008. The Growing Complexity of Internet Interconnection. Communications and Strategies 72 (2008), 51.Google Scholar
- T. Flach, N. Dukkipati, A. Terzis, B. Raghavan, N. Cardwell, Y. Cheng, A. Jain, S. Hao, E. Katz-Bassett, and R. Govindan. 2013. Reducing Web Latency: the Virtue of Gentle Aggression. In Proc. ACM SIGCOMM.Google Scholar
- B. Frank, I. Poese, Y. Lin, G. Smaragdakis, A. Feldmann, B. Maggs, J. Rake, S. Uhlig, and R. Weber. 2013. Pushing CDN-ISP Collaboration to the Limit. ACM CCR 43, 3 (2013), 34--44.Google ScholarDigital Library
- M. J. Freedman. 2010. Experiences with CoralCDN: A Five-Year Operational View. In Proc. USENIX NSDI 2010. 95--110.Google Scholar
- M. J. Freedman, M. Vutukuru, N. Feamster, and H. Balakrishnan. 2005. Geographic Locality of IP Prefixes. In ACM IMC 2005. 13--13.Google Scholar
- V. Giotsas, M. Luckie, B. Huffaker, and k. claffy. 2014. Inferring Complex AS Relationships. In ACM IMC 2014. 23--30.Google Scholar
- V. Giotsas, G. Smaragdakis, B. Huffaker, M. Luckie, and k. claffy. 2015. Mapping Peering Interconnections at the Facility Level. In Proc. ACM CoNEXT 2015. 37.Google Scholar
- Benocs GmbH. 2019. https://github.com/Benocs/utee. (2019).Google Scholar
- Benocs GmbH. 2019. https://github.com/Benocs/bftee. (2019).Google Scholar
- Google. 2019. Google Global Cache. http://ggcadmin.google.com/ggc. (2019).Google Scholar
- Google. 2019. Google Peering. https://peering.google.com. (2019).Google Scholar
- A. Gupta, L. Vanbever, M. Shahbaz, S. P. Donovan, B. Schlinker, N. Feamster, J. Rexford, S. Shenker, R. Clark, and E. Katz-Bassett. 2014. SDX: A Software Defined Internet Exchange. In Proc. ACM SIGCOMM 2014.Google Scholar
- O. Hohlfeld, J. Rüth, K. Wolsing, and T. Zimmermann. 2018. Characterizing a Meta-CDN. In PAM.Google Scholar
- S. Jain, A. Kumar, S. Mandal, J. Ong, L. Poutievski, A. Singh, S. Venkata, J. Wanderer, J. Zhou, M. Zhu, J. Zolla, U. Holzle, S. Stuart, and A. Vahdat. 2013. B4: Experience with a Globally-Deployed Software Defined WAN. ACM CCR 43 (2013), 3--14.Google ScholarDigital Library
- J. Jung, B. Krishnamurthy, and M. Rabinovich. 2002. Flash Crowds and Denial of Service Attacks. In Proc. ACM WWW 2002. 293--304.Google Scholar
- E. Katz-Bassett, J. P. John, A. Krishnamurthy, D. Wetherall, T. Anderson, and Y. Chawathe. 2006. Towards IP geolocation using delay and topology measurements. In ACM IMC 2006. 71--84.Google Scholar
- kc claffy, D. D. Clark, S. Bauer, and A. Dhamdhere. 2016. Policy challenges in mapping Internet interdomain congestion. In TPRC.Google Scholar
- S. Kiesel, W. Roome, R. Woundy, S. Previdi, S. Shalunov, R. Alimi, R. Penno, and Y. R. Yang. 2014. Application-Layer Traffic Optimization (ALTO) Protocol. IETF RFC 7285. (2014).Google Scholar
- R. Kohavi, R. M. Henne, and D. Sommerfield. 2007. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. In Proc. ACM KDD 2007. 959--967.Google Scholar
- R. Krishnan, H. Madhyastha, S. Srinivasan, S. Jain, A. Krishnamurthy, T. Anderson, and J. Gao. 2009. Moving Beyond End-to-end Path Information to Optimize CDN Performance. In ACM IMC 2009. 190--201.Google Scholar
- S. S. Krishnan and R. K. Sitaraman. 2012. Video Stream Quality Impacts Viewer Behavior: Inferring Causality using Quasi-Experimental Designs. In ACM IMC 2012. 2001--2014.Google Scholar
- C. Labovitz, S. Lekel-Johnson, D. McPherson, J. Oberheide, and F. Jahanian. 2010. Internet Inter-Domain Traffic. In Proc. ACM SIGCOMM.Google Scholar
- T. Leighton. 2009. Improving Performance on the Internet. Comm. of the ACM 52, 2 (2009), 44--51.Google ScholarDigital Library
- A. Lodhi, N. Larson, A. Dhamdhere, C. Dovrolis, and K. Claffy. 2014. Using PeeringDB to Understand the Peering Ecosystem. ACM CCR 44, 2 (2014).Google Scholar
- M. Luckie, A. Dhamdhere, D. Clark, B. Huffaker, and kc claffy. 2014. Challenges in Inferring Internet Interdomain Congestion. In ACM IMC 2014. 15--22.Google Scholar
- B. M. Maggs and R. K. Sitaraman. 2015. Algorithmic Nuggets in Content Delivery. ACM CCR 45, 3 (2015), 52--66.Google ScholarDigital Library
- Netflix. 2019. https://openconnect.netflix.com/en/deployment-guide/network-configuration/#routing-and-content-steering-via-bgp. (2019).Google Scholar
- Netflix. 2019. Netflix Open Connect. https://signup.netflix.com/openconnect. (2019).Google Scholar
- E. Nygren, R. K. Sitaraman, and J. Sun. 2010. The Akamai Network: A Platform for High-performance Internet Applications. SIGOPS Oper. Syst. Rev. 44, 3 (2010), 2--19.Google ScholarDigital Library
- P. Phaal and M. Lavine. 2004. sFlow version 5. https://sflow.org/sflow_version_5.txt. (2004).Google Scholar
- R. Padmanabhan, A. Dhamdhere, E. Aben, kc. Claffy, and N. Spring. 2016. Reasons Dynamic Addresses Change. In ACM IMC 2016. 183--198.Google Scholar
- PeeringDB. 2019. PeeringDB. https://www.peeringdb.com. (2019).Google Scholar
- pmacct. 2019. http://www.pmacct.net/. (2019).Google Scholar
- I. Poese, B. Frank, B. Ager, G. Smaragdakis, and A. Feldmann. 2010. Improving Content Delivery using Provider-aided Distance Information. In ACM IMC 2010. 22--34.Google Scholar
- I. Poese, B. Frank, G. Smaragdakis, S. Uhlig, A. Feldmann, and B. Maggs. 2012. Enabling Content-aware Traffic Engineering. ACM CCR 42, 5 (2012), 21--28.Google ScholarDigital Library
- I. Poese, S. Uhlig, M. A. Kaafar, B. Donnet, and B. Gueye. 2011. IP Geolocation Databases: Unreliable? ACM CCR 41, 2 (2011), 53--56.Google ScholarDigital Library
- L. Popa, A. Ghodsi, and I. Stoica. 2010. HTTP as the Narrow Waist of the Future Internet. In Proc. ACM SIGCOMM HotNets. 1--6.Google Scholar
- E. Pujol, P. Richter, B. Chandrasekaran, G. Smaragdakis, A. Feldmann, B. Maggs, and K. C. Ng. 2014. Back-Office Web Traffic on The Internet. In ACM IMC 2014. 257--270.Google Scholar
- Quagga community. 2019. Quagga Routing Suite. http://www.nongnu.org/quagga/. (2019).Google Scholar
- P. Richter, G. Smaragdakis, D. Plonka, and A. Berger. 2016. Beyond Counting: New Perspectives on the Active IPv4 Address Space. In ACM IMC.Google ScholarDigital Library
- J. P. Rula and F. E. Bustamante. 2014. Behind the Curtain - Cellular DNS and Content Replica Selection. In ACM IMC 2014. 59--72.Google Scholar
- M A. Sánchez, J. S. Otto, Z. S. Bischof, D. R. Choffnes, F. E. Bustamante, B. Krishnamurthy, and W. Willinger. 2013. Dasu: Pushing Experiments to the Internet's Edge. In Proc. USENIX NSDI 2013. 487--499.Google Scholar
- B. Schlinker, H. Kim, T. Cui, E. Katz-Bassett, H. V. Madhyastha, I. Cunha, J. Quinn, S. Hasan, P. Lapukhov, and H. Zeng. 2017. Engineering Egress with Edge Fabric: Steering Oceans of Content to the World. In Proc. ACM SIGCOMM 2017. 418--431.Google Scholar
- J. Scudder, R. Fernando, and S. Stuart. 2016. BGP Monitoring Protocol (BMP). IETF RFC 7854. (2016).Google Scholar
- J. Seedorf and E. Burger. 2009. Application-Layer Traffic Optimization (ALTO) Problem Statement. IETF RFC 5693. (2009).Google Scholar
- R. Singh, M. Ghobadi, K. T. Foerster, M. Filer, and P. Gill. 2018. RADWAN: Rate Adaptive Wide Area Network. In Proc. ACM SIGCOMM 2018. 547--560.Google Scholar
- R. K. Sitaraman, M. Kasbekar, W. Lichtenstein, and M. Jain. 2014. Overlay Networks: An Akamai Perspective. John Wiley & Sons.Google Scholar
- V. Stocker, G. Smaragdakis, W. Lehr, and S. Bauer. 2017. The Growing Complexity of Content Delivery Networks: Challenges and Implications for the Internet Ecosystem. Telecommunications Policy 41, 10 (2017), 1003--1016.Google ScholarCross Ref
- F. Streibelt, J. Boettger, N. Chatzis, G. Smaragdakis, and A. Feldmann. 2013. Exploring EDNS-Client-Subnet Adopters in your Free Time. In ACM IMC 2013. 305--312.Google Scholar
- V. Valancius, N. Laoutaris, L. Massoulie, C. Diot, and P. Rodriguez. 2009. Greening the Internet with Nano Data Centers. In Proc. ACM CoNEXT 2009. 37--48.Google Scholar
- D. Walton, A. Retana, E. Chen, and J. Scudder. 2016. Advertisement of Multiple Paths in BGP. IETF RFC 7911. (2016).Google Scholar
- P. Wendell and M. J. Freedman. 2011. Going Viral: Flash Crowds in an Open CDN. In ACM IMC 2011. 549--558.Google Scholar
- F. Wohlfart, N. Chatzis, C. Dabanoglu, G. Carle, and W. Willinger. 2018. Leveraging Interconnections for Performance: The Serving Infrastructure of a Large CDN. In Proc. ACM SIGCOMM 2018. 206--220.Google Scholar
- H. Xie, Y. R. Yang, A. Krishnamurthy, Y. G. Liu, and A. Silberschatz. 2008. P4P: Provider Portal for Applications. In Proc. ACM SIGCOMM 2008.Google Scholar
- K-K. Yap, M. Motiwala, J. Rahe, S. Padgett, M. Holliman, G. Baldus, M. Hines, T. Kim, A. Narayanan, A. Jain, V. Lin, C. Rice, B. Rogan, A. Singh, B. Tanaka, M. Verma, P. Sood, M. Tariq, M. Tierney, D. Trumic, V. Valancius, C. Ying, M. Kallahalla, B. Koley, and A. Vahdat. 2017. Taking the Edge off with Espresso: Scale, Reliability and Programmability for Global Internet Peering. In Proc. ACM SIGCOMM 2017. 432--445.Google Scholar
Index Terms
- Steering hyper-giants' traffic at scale
Recommendations
End-to-end quality of service provisioning through inter-provider traffic engineering
This paper addresses the issue of delivering solutions that will enable the incremental implementation of inter-domain quality of service (QoS) in the multi-provider commercial Internet. The paper first introduces a holistic architecture that describes ...
Invited A new traffic engineering manager for DiffServ/MPLS networks: design and implementation on an IP QoS Testbed
In a multi-service network, different applications have varying QoS requirements. The IETF has proposed the DiffServ architecture as a scalable solution to provide Quality of Service (QoS) in IP Networks. In order to provide quantitative guarantees and ...
Comments