My VM is Lighter (and Safer) than your Container

ABSTRACT

Containers are in great demand because they are lightweight when compared to virtual machines. On the downside, containers offer weaker isolation than VMs, to the point where people run containers in virtual machines to achieve proper isolation. In this paper, we examine whether there is indeed a strict tradeoff between isolation (VMs) and efficiency (containers). We find that VMs can be as nimble as containers, as long as they are small and the toolstack is fast enough.

We achieve lightweight VMs by using unikernels for specialized applications and with Tinyx, a tool that enables creating tailor-made, trimmed-down Linux virtual machines. By themselves, lightweight virtual machines are not enough to ensure good performance since the virtualization control plane (the toolstack) becomes the performance bottleneck. We present LightVM, a new virtualization solution based on Xen that is optimized to offer fast boot-times regardless of the number of active VMs. LightVM features a complete redesign of Xen's control plane, transforming its centralized operation to a distributed one where interactions with the hypervisor are reduced to a minimum. LightVM can boot a VM in 2.3ms, comparable to fork/exec on Linux (1ms), and two orders of magnitude faster than Docker. LightVM can pack thousands of LightVM guests on modest hardware with memory and CPU usage comparable to that of processes.

References

1. Amazon Web Services {n. d.}. Amazon EC2 Container Service. https://aws.amazon.com/ecs/. ({n. d.}).
2. Amazon Web Services {n. d.}. AWS Lambda - Serverless Compute. https://aws.amazon.com/lambda. ({n. d.}).
3. Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. 2003. Xen and the Art of Virtualization. SIGOPS Open Syst. Rev. 37, 5 (Oct. 2003), 164--177.
4. J. Clark. {n. d.}. Google: "EVERYTHING at Google runs in a container". http//:www.theregister.co.uk/2014/05/23/google_containerizationtwobillion/. ({n. d.}).
5. Patrick Colp, Mihir Nanavati, Jun Zhu, William Aiello, George Coker, Tim Deegan, Peter Loscocco, and Andrew Warfield. 2011. Breaking Up is Hard to Do: Security and Functionality in a Commodity Hypervisor. In Proceedings of the Twenty-Third ACM Symposium on Operating Systems Principles (SOSP '11). ACM, New York, NY, USA, 189--202.
6. Docker {n. d.}. The Docker Containerization Platform. https://www.docker.com/. ({n. d.}).
7. John R. Douceur, Jeremy Elson, Jon Howell, and Jacob R. Lorch. 2008. Leveraging Legacy Code to Deploy Desktop Applications on the Web. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI'08). USENIX Association, Berkeley, CA, USA, 339--354. http://dl.acm.org/citation.cfm?id=1855741.1855765
8. D. R. Engler, M. F. Kaashoek, and J. O'Toole, Jr. 1995. Exokernel: An Operating System Architecture for Application-level Resource Management. In Proceedings of the Fifteenth ACM Symposium on Operating Systems Principles (SOSP '95). ACM, New York, NY, USA, 251--266.
9. Erlang on Xen 2012. Erlang on Xen. http://erlangonxen.org/. (July 2012).
10. Google Cloud Platform {n. d.}. The Google Cloud Platform Container Engine. https://cloud.google.com/container-engine. ({n. d.}).
11. A. Grattafiori. {n. d.}. Understanding and Hardening Linux Containers. https://www.nccgroup.trust/us/our-research/understanding-and-hardening-linux-containers/. ({n. d.}).
12. Cameron Hamilton-Rich. {n. d.}. axTLS Embedded SSL. http://axtls.sourceforge.net. ({n. d.}).
13. Poul henning Kamp and Robert N. M. Watson. 2000. Jails: Confining the omnipotent root. In In Proc. 2nd Intl. SANE Conference.
14. J. Hertz. {n. d.}. Abusing Privileged and Unprivileged Linux Containers. https://www.nccgroup.tmst/uk/our-research/abusing-privileged-and-unprivileged-linux-containers/, ({n. d.}).
15. Jon Howell, Bryan Parno, and John R. Douceur. 2013. Embassies: Radically Refactoring the Web. In Presented as part of the 10th USENTX Symposium on Networked Systems Design and Implementation (NSDI13). USENIX, Lombard, IL, 529--545. https://www.usenix.org/conference/nsdil3/technical-sessions/presentation/howell
16. Yun Chao Hu, Milan Patel, Dario Sabella, Nurit Sprecher, and Valerie Young. 2015. Mobile Edge Computing - A key technology towards 5G. ETSI White Paper No. 11, First edition (2015).
17. IBM. {n. d.}. Docker at insane scale on IBM Power Systems. https://www.ibm.com/blogs/bluemix/2015/ll/docker-insane-scale-on-ibm-power-systems. ({n. d.}).
18. IBM developerWorks Open {n. d.}. Solo5 Unikernel. https://developer.ibm.com/open/openprojects/solo5-unikernel/. ({n. d.}).
19. Intel. {n. d.}. Intel Clear Containers: A Breakthrough Combination of Speed and Workload Isolation. https://clearlinux.org/sites/default/files/vmscontainers_wp_v5.pdf. ({n. d.}).
20. Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. 2007. KVM: the Linux Virtual Machine Monitor. In In Proc. 2007 Ottawa Linux Symposium (OLS '07).
21. Avi Kivity, Dor Laor, Glauber Costa, Pekka Enberg, Nadav Har'El, Don Marti, and Vlad Zolotarov. 2014. OSv---Optimizing the Operating System for Virtual Machines. In Proceedings of the 2014 USENTX Annual Technical Conference (USENIX ATC '14). USENIX Association, Philadelphia, PA, 61--72. https://www.usenix.org/conference/atcl4/technical-sessions/presentation/kivity
22. E. Kovacs. {n. d.}. Docker Fixes Vulnerabilities, Shares Plans For Making Platform Safer. http//:www.securityweek.com/docker-fixes-vulnerabilities-shares-plans-making-platform-safer. ({n. d.}).
23. Simon Kuenzer, Anton Ivanov, Filipe Manco, Jose Mendes, Yuri Volchkov, Florian Schmidt, Kenichi Yasukata, Michio Honda, and Felipe Huici. 2017. Unikernels Everywhere: The Case for Elastic CDNs. In Proceedings of the 13th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE '17). ACM, New York, NY, USA, 15--29.
24. Horacio Andrés Lagar-Cavilla, Joseph Andrew Whitney, Adin Matthew Scannell, Philip Patchin, Stephen M. Rumble, Eyal de Lara, Michael Brudno, and Mahadev Satyanarayanan. 2009. SnowFlock: Rapid Virtual Machine Cloning for Cloud Computing. In Proceedings of the 4th ACM European Conference on Computer Systems (EuroSys '09). ACM, New York, NY, USA, 1--12.
25. LinuxContainers.org {n. d.}. LinuxContainers.org. https://linuxcontainers.org. ({n. d.}).
26. Anil Madhavapeddy, Thomas Leonard, Magnus Skjegstad, Thomas Gazagnaire, David Sheets, Dave Scott, Richard Mortier, Amir Chaudhry, Balraj Singh, Jon Ludlam, Jon Crowcroft, and Ian Leslie. 2015. Jitsu: Just-In-Time Summoning of Unikernels. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI '15). USENIX Association, Oakland, CA, 559--573. https://www.usenix.org/conference/nsdil5/technical-sessions/presentation/madhavapeddy
27. Anil Madhavapeddy and David J. Scott. 2013. Unikernels: Rise of the Virtual Library Operating System. Queue 11, 11, Article 30 (Dec. 2013), 15 pages.
28. Y. Mao, J. Zhang, and K. B. Letaief. 2016. Dynamic Computation Offloading for Mobile-Edge Computing With Energy Harvesting Devices. IEEE Journal on Selected Areas in Communications 34, 12 (Dec 2016), 3590--3605.
29. Joao Martins, Mohamed Ahmed, Costin Raiciu, Vladimir Olteanu, Michio Honda, Roberto Bifulco, and Felipe Huici. 2014. ClickOS and the Art of Network Function Virtualization. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI '14). USENIX Association, Seattle, WA, 459--473. https://www.usenix.org/conference/nsdil4/technical-sessions/presentation/martins
30. McAffee. 2016. Mobile Threat Report. https://www.mcafee.com/us/resources/reports/rp-mobile-threat-report-2016.pdf. (2016).
31. MicroPython {n. d.}. MicroPython. https://micropython.org/. ({n. d.}).
32. Microsoft. {n. d.}. Azure Container Service. https://azure.microsoft.com/en-us/services/container-service/. ({n. d.}).
33. Microsoft Research. {n. d.}. Drawbridge. https://www.microsoft.com/en-us/research/project/drawbridge/. ({n. d.}).
34. minios {n. d.}. Mini-OS. https://wiki.xenproject.org/wiki/Mini-OS. ({n. d.}).
35. A. Mourat. {n. d.}. 5 security concerns when using Docker. https://www.oreilly.com/ideas/five-security-concerns-when-using-docker. ({n. d.}).
36. Vlad Nitu, Pierre Olivier, Alain Tchana, Daniel Chiba, Antonio Barbalace, Daniel Hagimont, and Binoy Ravindran. 2017. Swift Birth and Quick Death: Enabling Fast Parallel Guest Boot and Destruction in the Xen Hypervisor. In Proceedings of the 13th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE '17). ACM, New York, NY, USA, 1--14.
37. MAN page. {n. d.}. Linux system calls list. http://man7.org/linux/man-pages/man2/syscalls.2.html. ({n. d.}).
38. Rumpkernel.org {n. d.}. Rump Kernels. http://rumpkernel.org/. ({n. d.}).
39. Sandvine. {n. d.}. Internet traffic encryption. https://www.sandvine.com/trends/encryption.html. ({n. d.}).
40. Mahadev Satyanarayanan, Paramvir Bahl, Ramón Caceres, and Nigel Davies. 2009. The Case for VM-Based Cloudlets in Mobile Computing. IEEE Pervasive Computing 8, 4 (Oct. 2009), 14--23.
41. Justine Sherry, Shaddi Hasan, Colin Scott, Arvind Krishnamurthy, Sylvia Ratnasamy, and Vyas Sekar. 2012. Making Middleboxes Someone Else's Problem: Network Processing As a Cloud Service. In Proceedings of the ACM SIGCOMM 2012 Conference on Computer Communication (SIGCOMM '12). ACM, New York, NY, USA, 13--24.
42. Stephen Soltesz, Herbert Pötzl, Marc E. Fiuczynski, Andy Bavier, and Larry Peterson. 2007. Container-based Operating System Virtualization: A Scalable, High-performance Alternative to Hypervisors. SIGOPS Oper. Syst. Rev. 41, 3 (March 2007), 275--287.
43. S. Stabellini. {n. d.}. Xen on ARM. http//:www.slideshare.net/xen_com_mgr/alsf13-stabellini. ({n. d.}).
44. Udo Steinberg and Bernhard Kauer. 2010. NOVA: A Microhypervisorbased Secure Virtualization Architecture. In Proceedings of the 5th European Conference on Computer Systems (EuroSys '10). ACM, New York, NY, USA, 209--222.
45. A. van de Ven. {n. d.}. An introduction to Clear Containers. https://lwn.net/Articles/644675/. ({n. d.}).
46. Akshat Verma, Gargi Dasgupta, Tapan Kumar Nayak, Pradipta De, and Ravi Kothari. 2009. Server Workload Analysis for Power Minimization Using Consolidation. In Proceedings of the 2009 USENIX Annual Technical Conference (USENIX ATC '09). USENIX Association, Berkeley, CA, USA, 28--28. http://dl.acm.org/citation.cfm?id=1855807.1855835
47. VMWare. {n. d.}. vSphere ESXi Bare-Metal Hypervisor. http//:www.vmware.com/products/esxi-and-esx.html. ({n. d.}).
48. Michael Vrable, Justin Ma, Jay Chen, David Moore, Erik Vandekieft, Alex C. Snoeren, Geoffrey M. Voelker, and Stefan Savage. 2005. Scalability, Fidelity, and Containment in the Potemkin Virtual Honey-farm. SIGOPS Oper. Syst. Rev. 39, 5 (Oct. 2005), 148--162.
49. Andrew Whitaker, Marianne Shaw, and Steven D. Gribble. 2002. Scale and Performance in the Denali Isolation Kernel. SIGOPS Oper. Syst. Rev. 36, SI (Dec. 2002), 195--209.
50. Dan Williams and Ricardo Koller. 2016. Unikernel Monitors: Extending Minimalism Outside of the Box. In 8th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud '16). USENIX Association, Denver, CO. https://www.usenix.org/conference/hotcloud16/workshop-program/presentation/williams
51. Wei Zhang, Jinho Hwang, Shriram Rajagopalan, K.K. Ramakrishnan, and Timothy Wood. 2016. Flurries: Countless Fine-Grained NFs for Flexible Per-Flow Customization. In Proceedings of the 12th International on Conference on Emerging Networking EXperiments and Technologies (CoNEXT '16). ACM, New York, NY, USA, 3--17.

Index Terms

• Software and its engineering

  • Software organization and properties

    • Contextual software domains

      • Operating systems

      • Software infrastructure

        • Virtual machines