Toke Høiland-Jørgensen
ABSTRACT
Programmable packet processing is increasingly implemented using kernel bypass techniques, where a userspace application takes complete control of the networking hardware to avoid expensive context switches between kernel and userspace. However, as the operating system is bypassed, so are its application isolation and security mechanisms; and well-tested configuration, deployment and management tools cease to function.
To overcome this limitation, we present the design of a novel approach to programmable packet processing, called the eXpress Data Path (XDP). In XDP, the operating system kernel itself provides a safe execution environment for custom packet processing applications, executed in device driver context. XDP is part of the mainline Linux kernel and provides a fully integrated solution working in concert with the kernel's networking stack. Applications are written in higher level languages such as C and compiled into custom byte code which the kernel statically analyses for safety, and translates into native instructions.
We show that XDP achieves single-core packet processing performance as high as 24 million packets per second, and illustrate the flexibility of the programming model through three example use cases: layer-3 routing, inline DDoS protection and layer-4 load balancing.
Supplemental Material
- David Ahern. 2018. XDP forwarding example. https://elixir.bootlin.com/linux/v4.18-rc1/source/samples/bpf/xdp_fwd_kern.cGoogle Scholar
- Cilium Authors. 2018. BPF and XDP Reference Guide. https://cilium.readthedocs.io/en/latest/bpf/Google Scholar
- Cilium Authors. 2018. Cilium software. https://github.com/cilium/ciliumGoogle Scholar
- Suricata authors. 2018. Suricata - eBPF and XDP. https://suricata.readthedocs.io/en/latest/capture-hardware/ebpf-xdp.htmlGoogle Scholar
- Adam Belay, George Prekas, Ana Klimovic, Samuel Grossman, Christos Kozyrakis, and Edouard Bugnion. 2014. IX: A protected dataplane operating system for high throughput and low latency. In Proceedings of the 11th USENIX Symposium on Operating System Design and Implementation (OSDI '14). USENIX. Google ScholarDigital Library
- Gilberto Bertin. 2017. XDP in practice: integrating XDP in our DDoS mitigation pipeline. In NetDev 2.1 - The Technical Conference on Linux Networking.Google Scholar
- Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, Dan Talayco, Amin Vahdat, George Varghese, et al. 2014. P4: Programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review 44, 3 (2014). Google ScholarDigital Library
- bpftool authors. 2018. bpftool manual. https://elixir.bootlin.com/linux/v4.18-rc1/source/tools/bpf/bpftool/Documentation/bpftool.rstGoogle Scholar
- Cisco. 2018. TRex Traffic Generator. https://trex-tgn.cisco.com/Google Scholar
- CZ.nic. 2018. BIRD Internet Routing Daemon. https://bird.network.cz/Google Scholar
- Luca Deri. 2009. Modern packet capture and analysis: Multi-core, multi-gigabit, and beyond. In the 11th IFIP/IEEE International Symposium on Integrated Network Management (IM).Google Scholar
- Mihai Dobrescu, Norbert Egi, Katerina Argyraki, Byung-Gon Chun, Kevin Fall, Gianluca Iannaccone, Allan Knies, Maziar Manesh, and Sylvia Ratnasamy. 2009. RouteBricks: exploiting parallelism to scale software routers. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. ACM. Google ScholarDigital Library
- Linux documentation authors. 2018. Linux Tracing Technologies. https://www.kernel.org/doc/html/latest/trace/index.htmlGoogle Scholar
- Paul Emmerich, Sebastian Gallenmüller, Daniel Raumer, Florian Wohlfart, and Georg Carle. 2015. Moongen: A scriptable high-speed packet generator. In Proceedings of the 2015 Internet Measurement Conference. ACM. Google ScholarDigital Library
- Facebook. 2018. Katran source code repository. https://github.com/facebookincubator/katranGoogle Scholar
- Linux Foundation. 2018. Data Plane Development Kit. https://www.dpdk.org/Google Scholar
- The Linux Foundation. 2018. FRRouting. https://frrouting.org/Google Scholar
- Sebastian Gallenmüller, Paul Emmerich, Florian Wohlfart, Daniel Raumer, and Georg Carle. 2015. Comparison of Frameworks for High-Performance Packet IO. In Proceedings of the Eleventh ACM/IEEE Symposium on Architectures for Networking and Communications Systems (ANCS '15). IEEE Computer Society, 29--38. Google ScholarDigital Library
- Sangjin Han, Keon Jang, KyoungSoo Park, and Sue Moon. 2010. PacketShader: a GPU-accelerated software router. In ACM SIGCOMM Computer Communication Review, Vol. 40. ACM. Google ScholarDigital Library
- Sangjin Han, Scott Marshall, Byung-Gon Chun, and Sylvia Ratnasamy. 2012. MegaPipe: A New Programming Interface for Scalable Network I/O. In Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI '12). Google ScholarDigital Library
- Tom Herbert. 2016. Initial thoughts on TXDP. https://www.spinics.net/lists/netdev/msg407537.htmlGoogle Scholar
- Toke Høiland-Jørgensen, Jesper Dangaard Brouer, Daniel Borkmann, John Fastabend, Tom Herbert, David Ahern, and David Miller. 2018. XDP-paper online appendix. https://github.com/tohojo/xdp-paperGoogle Scholar
- Toke Høiland-Jørgensen, Per Hurtig, and Anna Brunstrom. 2015. The Good, the Bad and the WiFi: Modern AQMs in a residential setting. Computer Networks 89 (Oct. 2015). Google ScholarDigital Library
- Solarflare Communications Inc. 2018. OpenOnload. https://www.openonload.org/Google Scholar
- EunYoung Jeong, Shinae Woo, Muhammad Asim Jamshed, Haewon Jeong, Sunghwan Ihm, Dongsu Han, and KyoungSoo Park. 2014. mTCP: a Highly Scalable User-level TCP Stack for Multicore Systems.. In Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI '14), Vol. 14. 489--502. Google ScholarDigital Library
- Rick Jones. 2018. Netperf. Open source benchmarking software. http://www.netperf.org/Google Scholar
- Jakub Kicinski and Nic Viljoen. 2016. eBPF/XDP hardware offload to SmartNICs. In NetDev 1.2 - The Technical Conference on Linux Networking.Google Scholar
- Davide Kirchner, Raihana Ferdous, Renato Lo Cigno, Leonardo Maccari, Massimo Gallo, Diego Perino, and Lorenzo Saino. 2016. Augustus: a CCN router for programmable networks. In Proceedings of the 3rd ACM Conference on Information-Centric Networking. ACM. Google ScholarDigital Library
- Chris Lattner and Vikram Adve. 2004. LLVM: A compilation framework for lifelong program analysis & transformation. In Proceedings of the international symposium on Code generation and optimization: feedback-directed and runtime optimization. IEEE Computer Society. Google ScholarDigital Library
- libbpf authors. 2018. libbpf source code. https://elixir.bootlin.com/linux/v4.18-rc1/source/tools/lib/bpfGoogle Scholar
- Leonardo Linguaglossa, Dario Rossi, Salvatore Pontarelli, Dave Barach, Damjan Marjon, and Pierre Pfister. 2017. High-speed software data plane via vectorized packet processing. Technical Report. Telecom ParisTech.Google Scholar
- John W Lockwood, Nick McKeown, Greg Watson, Glen Gibb, Paul Hartke, Jad Naous, Ramanan Raghuraman, and Jianying Luo. 2007. NetFPGA-an open platform for gigabit-rate network switching and routing. In IEEE International Conference on Microelectronic Systems Education. IEEE. Google ScholarDigital Library
- Rodrigo B Mansilha, Lorenzo Saino, Marinho P Barcellos, Massimo Gallo, Emilio Leonardi, Diego Perino, and Dario Rossi. 2015. Hierarchical content stores in high-speed ICN routers: Emulation and prototype implementation. In Proceedings of the 2nd ACM Conference on Information-Centric Networking. ACM. Google ScholarDigital Library
- Tudor Marian, Ki Suh Lee, and Hakim Weatherspoon. 2012. NetSlices: scalable multi-core packet processing in user-space. In Proceedings of the eighth ACM/IEEE symposium on Architectures for networking and communications systems. ACM. Google ScholarDigital Library
- Ilias Marinos, Robert NM Watson, and Mark Handley. 2014. Network stack specialization for performance. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 175--186. Google ScholarDigital Library
- Joao Martins, Mohamed Ahmed, Costin Raiciu, Vladimir Olteanu, Michio Honda, Roberto Bifulco, and Felipe Huici. 2014. ClickOS and the art of network function virtualization. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation. USENIX Association. Google ScholarDigital Library
- Steven McCanne and Van Jacobson. 1993. The BSD Packet Filter: A New Architecture for User-level Packet Capture. In USENIX winter, Vol. 93. Google ScholarDigital Library
- Sebastiano Miano, Matteo Bertrone, Fulvio Risso, Massimo Tumolo, and Mauricio Vásquez Bernal. 2018. Creating Complex Network Service with eBPF: Experience and Lessons Learned. In IEEE International Conference on High Performance Switching and Routing.Google ScholarCross Ref
- David S. Miller. 2017. Generic XDP. https://git.kernel.org/torvalds/c/b5cdae3291f7Google Scholar
- Robert Morris, Eddie Kohler, John Jannotti, and M Frans Kaashoek. 1999. The Click modular router. ACM SIGOPS Operating Systems Review 33, 5 (1999). Google ScholarDigital Library
- Juniper Networks. 2018. Juniper Contrail Virtual Router. https://github.com/Juniper/contrail-vrouterGoogle Scholar
- perf authors. 2018. perf: Linux profiling with performance counters. https://perf.wiki.kernel.org/index.php/Main_PageGoogle Scholar
- Simon Peter, Jialin Li, Irene Zhang, Dan RK Ports, Doug Woos, Arvind Krishnamurthy, Thomas Anderson, and Timothy Roscoe. 2016. Arrakis: The operating system is the control plane. ACM Transactions on Computer Systems (TOCS) 33, 4 (2016). Google ScholarDigital Library
- Ben Pfaff, Justin Pettit, Teemu Koponen, Ethan J Jackson, Andy Zhou, Jarno Rajahalme, Jesse Gross, Alex Wang, Joe Stringer, Pravin Shelar, et al. 2015. The Design and Implementation of Open vSwitch. In Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI '15). Google ScholarDigital Library
- Ntop project. 2018. PF_RING ZC (Zero Copy). https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/Google Scholar
- Luigi Rizzo. 2012. Netmap: a novel framework for fast packet I/O. In 21st USENIX Security Symposium (USENIX Security 12).Google ScholarDigital Library
- Luigi Rizzo and Giuseppe Lettieri. 2012. Vale, a switched ethernet for virtual machines. In Proceedings of the 8th international conference on Emerging networking experiments and technologies. ACM. Google ScholarDigital Library
- Pedro M Santiago del Rio, Dario Rossi, Francesco Gringoli, Lorenzo Nava, Luca Salgarelli, and Javier Aracil. 2012. Wire-speed statistical classification of network traffic on commodity hardware. In Proceedings of the 2012 Internet Measurement Conference. ACM. Google ScholarDigital Library
- William Tu. 2018. {ovs-dev} AF_XDPsupport for OVS. https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/351295.htmlGoogle Scholar
- IO Visor. 2018. BCC BPF Compiler Collection. https://www.iovisor.org/technology/bccGoogle Scholar
- VMWare. 2018. p4c-xdp. https://github.com/vmware/p4c-xdpGoogle Scholar
- Kenichi Yasukata, Michio Honda, Douglas Santry, and Lars Eggert. 2016. StackMap: Low-Latency Networking with the OS Stack and Dedicated NICs. In 2016 USENIX Annual Technical Conference (USENIX ATC 16). USENIX Association, 43--56. Google ScholarDigital Library
- Qi Zhang. 2018. {dpdk-dev} PMD driver for AF_XDP. http://mails.dpdk.org/archives/dev/2018-February/091502.htmlGoogle Scholar
Index Terms
- The eXpress data path: fast programmable packet processing in the operating system kernel
Recommendations
Accelerating Linux Security with eBPF iptables
SIGCOMM '18: Proceedings of the ACM SIGCOMM 2018 Conference on Posters and DemosDomain specific run time optimization for software data planes
ASPLOS '22: Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating SystemsState-of-the-art approaches to design, develop and optimize software packet-processing programs are based on static compilation: the compiler's input is a description of the forwarding plane semantics and the output is a binary that can accommodate any ...
MoonGen: A Scriptable High-Speed Packet Generator
IMC '15: Proceedings of the 2015 Internet Measurement ConferenceWe present MoonGen, a flexible high-speed packet generator. It can saturate 10 GbE links with minimum-sized packets while using only a single CPU core by running on top of the packet processing framework DPDK. Linear multi-core scaling allows for even ...
Comments