Xin Xu
ABSTRACT
Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned to a virtual machine (VM) in passthrough mode to significantly improve the network performance. However, VF passthrough mode is not compatible with live migration, which is an essential capability that enables many advanced virtualization features such as high availability and resource provisioning.
To solve this problem, we design SRVM which provides hypervisor support to ensure the VF device can be correctly used by the migrated VM and the applications. SRVM is implemented in the hypervisor without modification in guest operating systems or guest VM drivers. Our experimental results show that SRVM can effectively migrate all memory state, and there is no data loss or corruption in applications after live migration. SRVM does not increase VM downtime. It only costs limited resources (an extra CPU core), and there is no significant runtime overhead in VM network performance. In fact, since the VF can continue to be used during the pre-copy ase, it offers network throughput which is 9.6 times and network latency which is 98% lower compared to other solutions that switch to para-virtualization mode during live migration.
- "InfiniBand Trade Association," http://www.infinibandta.org.Google Scholar
- "RDMA Consortium," http://www.rdmaconsortium.org.Google Scholar
- "HIGH PERFORMANCE, OPEN STANDARD VIRTUALIZATION WITH NFV AND SDN,"http://www.windriver.com/whitepapers/ovp/ovp_whitepaper.pdf, [Online;accessed 12-Oct-2015].Google Scholar
- "WIND RIVER TITANIUM SERVER,"http://www.windriver.com/without-compromise/resources/pdf/CGCS_Product_Overview.pdf,[Online; accessed 12-Oct-2015].Google Scholar
- A. Kadav and M. M. Swift, "Live migration of direct-access devices," SIGOPS Oper. Syst. Rev., vol. 43, no. 3, pp. 95--104, Jul. 2009.Google ScholarDigital Library
- Z. Pan, Y. Dong, Y. Chen, L. Zhang, and Z. Zhang, "Compsc: Live migration withpass-through devices," in Proceedings of the 8th ACM SIGPLAN/SIGOPS Conference on Virtual Execution Environments, ser. VEE'12. New York, NY, USA: ACM, 2012, pp. 109--120.Google ScholarDigital Library
- E. Zhai, G. D. Cummings, and Y. Dong, "Live migration with pass-through devicefor linux vm," in OLS'08: The 2008 Ottawa Linux Symposium, 2008,pp. 261--268.Google Scholar
- "Microsoft Hyper-V," http://www.microsoft.com/en-us/server-cloud/solutions/virtualization.aspx,[Online; accessed 20-Aug-2015].Google Scholar
- "Linux ethernet bonding driver howto," https://www.kernel.org/doc/Documentation/networking/bonding.txt, 2011,[Online; accessed 20-Aug-2015].Google Scholar
- Intel, "Intel Virtualization Technology for Direct I/O,"[Oct-2014].Google Scholar
- C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. Jul, C. Limpach, I. Pratt, andA. Warfield, "Live migration of virtual machines," in Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation-Volume 2. USENIX Association, 2005, pp. 273--286.Google ScholarDigital Library
- VMware, "VMware vSere vMotion Architecture, Performance and Best Practicesin VMware vSere 5," [Technical White Paper].Google Scholar
- Intel, "Intel 64 and IA-32 Architectures Software Developer's Manual,"[May-2011].Google Scholar
- ----, "Intel 82599 10 GbE Controller Datasheet," [Feb-2015 Revision 3.1331520-002].Google Scholar
- ----, "Intel 82599 SR-IOV Driver Companion Guide," [May-2010 Revision1.00].Google Scholar
- "Data Plane Development Kit," http://dpdk.org, [Accessed on Nov 17, 2015].Google Scholar
- P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer,I. Pratt, and A. Warfield, "Xen and the Art of Virtualization," inProceedings of the Nineteenth ACM Symposium on Operating Systems Principles, ser. SOSP'03. New York,NY, USA: ACM, 2003.Google Scholar
- "Kernel-based Virtual Machine," http://www.linux-kvm.org, [Online;accessed 15-Oct-2015].Google Scholar
- "VMware," http://www.vmware.com, [Online; accessed 15-Oct-2015].Google Scholar
- H. Z. Weidong Han,"Live Migration with SR-IOV Pass-through," [Presentation on KVM Forum 2015, Aug 2015].Google Scholar
- PCI-SIG, "Address Translation Services," [Jan-26-2009 Revision 1.1].Google Scholar
- ----, "PCI Express Base Specification Revision 3.0," [Nov-10--2010 Revision 3.0].Google Scholar
- "Pktgen-dpdk: Traffic generator powered by DPDK,"http://dpdk.org/browse/apps/pktgen-dpdk/refs/, [Accessed on Nov 17, 2015].Google Scholar
Index Terms
- SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
Recommendations
A Hypervisor Approach to Enable Live Migration with Passthrough SR-IOV Network Devices
Special TopicsSingle-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (physical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly ...
SRVM: Hypervisor Support for Live Migration with Passthrough SR-IOV Network Devices
VEE '16Single-Root I/O Virtualization (SR-IOV) is a specification that allows a single PCI Express (PCIe) device (ysical function or PF) to be used as multiple PCIe devices (virtual functions or VF). In a virtualization system, each VF can be directly assigned ...
Crane: fast and migratable GPU passthrough for OpenCL applications
SYSTOR '17: Proceedings of the 10th ACM International Systems and Storage ConferenceGeneral purpose GPU (GPGPU) computing in virtualized environments leverages PCI passthrough to achieve GPU performance comparable to bare-metal execution. However, GPU passthrough prevents service administrators from performing virtual machine migration ...
Comments