Traps, events, emulation, and enforcement: managing the yin and yang of virtualization-based security

ABSTRACT

We question current trends that attempt to leverage virtualization techniques to achieve security goals. We suggest that the security role of a virtual machine centers on being a policy interpreter rather than a resource provider. These two roles (security reference monitor and resource emulator) are currently conflated within the context of virtual machines and VMMs. We believe that this ``double-duty'' leads to both a significant performance impact as well as a bloated virtualization layer. Increased complexity reduces confidence that the code is elementary enough to verify or trust from a security perspective. Ironically, as more security-related functionality is shoved into a VM platform, the system becomes less trustworthy as it becomes increasingly trusted.

We argue that a principle reason for such an unfortunate situation is the lack of efficient hardware trapping mechanisms. We propose an architecture to help ameliorate this problem by transferring the security enforcement and program analysis roles from the virtualization component to a policy-directed FPGA.

References

1. M. Abadi, M. Budiu, U. Erlingsson, and J. Ligatti. Control-Flow Integrity: Principles, Implementations, and Applications. In Proceedings of the ACM Conference on Computer and Communications Security (CCS), 2005.
2. H. Agrawal. Towards Automatic Debugging of Computer Programs, August 1991.
3. T. Beauchamp and D. Weston. Re:Trace - Applied Reverse Engineering on OS X.
4. T. Beauchamp and D. Weston. Re:Trace - Applied Reverse Engineering on OS X. RECON 2008, 2008. Montreal, Quebec.
5. F. Bellard. QEMU, a Fast and Portable Dynamic Translator. In Proceedings of the 2005 USENIX Annual Technical Conference, FREENIX Track, pages 41--46, April 2005.
6. S. M. Bellovin. Virtual Machines, Virtual Security. Communications of the ACM, 49(10), October 2006.
7. F. Brooks. The Mythical Man Month. Addison-Wesley Professional, 2 edition, 1995.
8. D. Bruening, T. Garnett, and S. Amarasinghe. An infrastructure for adaptive dynamic optimization. In Proceedings of the International Symposium on Code Generation and Optimization, pages 265--275, 2003.
9. B. Buck and J. K. Hollingsworth. An API for Runtime Code Patching. The International Journal of High Performance Computing Applications, 14(4):317--329, Winter 2000.
10. B. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic instrumentation of production systems. In USENIX Annual Technical Conference, General Track, pages 15--28, 2004.
11. P. A. Karger and D. R. Safford. Security and Performance Trade-Offs in I/O Operations for Virtual Machine Monitors. In IBM Research Technical Report RC24500 (W0802--069), February 2008.
12. S. T. King, J. Tucek, A. Cozzie, C. Grier, W. Jiang, and Y. Zhou. Designing and Implementing Malicious Hardware. In Proceedings of the 1st USENIX Workshop on Large-Scale Exploits and Emergent Threats, 2008.
13. V. Kiriansky, D. Bruening, and S. Amarasinghe. Secure Execution Via Program Shepherding. In Proceedings of the 11th USENIX Security Symposium, August 2002.
14. C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation. In Proceedings of Programming Language Design and Implementation (PLDI), June 2005.
15. mayhem. The Cerberus ELF Interface. Phrack, 2003.
16. N. Nethercote and J. Seward. Valgrind: A Framework for Heavyweight Dynamic Binary Instrumentation. In Proceedings of ACM SIGPLAN 2007 Conference on Programming Language Design and Implementation (PLDI 2007), June 2007.
17. E. B. Nightingale, D. Peek, P. M. Chen, and J. Flinn. Parallelizing Security Checks on Commodity Hardware. In Proceedings of the International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2008.
18. J. Oplinger and M. S. Lam. Enhancing Software Reliability with Speculative Threads. In Proceedings of the 10th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS X), October 2002.
19. V. Prasad, W. Cohen, F. C. Eigler, M. Hunt, J. Keniston, and B. Chen. Locating system problems using dynamic instrumentation. 2005.
20. T. Roscoe, K. Elphinstone, and G. Heiser. Hype and Virtue. In Proceedings of the $11^th$ Workshop on Hot Topics in Operating Systems (HOTOS XI), May 2007.
21. E. C. Sezer, P. Ning, C. Kil, and J. Xu. MemSherlock: an Automated Debugger for Unknown Memory Corruption Vulnerabilities. In Proceedings of the 14th ACM conference on Computer and communications security (CCS 2007), pages 562--572, New York, NY, USA, 2007. ACM.
22. T. E. shell crew. Embedded ELF Debugging: the middle head of Cerberus. Phrack, 2003.
23. R. M. Stallman, R. H. Pesch, and S. Shebs. Debugging with GDB: The GNU Source-Level Debugger. Free Software Foundation, 2003.
24. H. Yin, Z. Liang, and D. Song. HookFinder: Identifying and Understanding Malware Hooking Behaviors. In Proceedings of the 15th Annual Network and Distributed System Security Symposium (NDSS), February 2008.

Index Terms

• Computer systems organization

  • Architectures

    • Other architectures

      • Data flow architectures

• Hardware

  • Very large scale integration design

    • Application-specific VLSI designs