Detecting ROP with Statistical Learning of Program Characteristics

ABSTRACT

Return-Oriented Programming (ROP) has emerged as one of the most widely used techniques to exploit software vulnerabilities. Unfortunately, existing ROP protections suffer from a number of shortcomings: they require access to source code and compiler support, focus on specific types of gadgets, depend on accurate disassembly and construction of Control Flow Graphs, or use hardware-dependent (microarchitectural) characteristics. In this paper, we propose EigenROP, a novel system to detect ROP payloads based on unsupervised statistical learning of program characteristics. We study, for the first time, the feasibility and effectiveness of using microarchitecture-independent program characteristics -- namely, memory locality, register traffic, and memory reuse distance -- for detecting ROP. We propose a novel directional statistics based algorithm to identify deviations from the expected program characteristics during execution. EigenROP works transparently to the protected program, without requiring debug information, source code or disassembly. We implemented a dynamic instrumentation prototype of EigenROP using Intel Pin and measured it against in-the-wild ROP exploits and on payloads generated by the ROP compiler ROPC. Overall, EigenROP achieved significantly higher accuracy than prior anomaly-based solutions. It detected the execution of the ROP gadget chains with 81% accuracy, 80% true positive rate, only 0.8% false positive rate, and incurred comparable overhead to similar Pin-based solutions.

References

1. Ropc. https://github.com/pakt/ropc.
2. Scikit. http://scikit-learn.org/stable/.
3. Applications, tools and techniques on the road to exascale computing. In K. de Bosschere et al., editors, Advances in Parallel Computing, volume 22. 2012.
4. T. Bletsch, X. Jiang, V. W. Freeh, and Z. Liang. Jump-oriented programming: a new class of code-reuse attack. In 6th ASIACCS. ACM, 2011.
5. N. Carlini and D. Wagner. Rop is still dangerous: Breaking modern defenses. In USENIX Security, 2014.
6. Y. Cheng et al. Ropecker: A generic and practical approach for defending against rop attack. In NDSS, 2014.
7. L. Davi, A.-R. Sadeghi, and M. Winandy. Ropdefender: A detection tool to defend against return-oriented programming attacks. In 6th ASIACCS. ACM, 2011.
8. J. Demme and others. On the feasibility of online malware detection with performance counters. Computer Architecture News, 41(3), 2013.
9. I. Fratric. Ropguard: Runtime prevention of return-oriented programming attacks, 2012.
10. K. Hoste and L. Eeckhout. Comparing benchmarks using key microarch.-independent characteristics. In Workload Characterization. IEEE, 2006.
11. K. Hoste and L. Eeckhout. Microarchitecture-independent workload characterization. IEEE Micro, 3, 2007.
12. E. R. Jacobson and others. Detecting code reuse attacks with a model of conformant program execution. In Engineering Secure Software and Systems. Springer, 2014.
13. C.-K. Luk et al. Building customized program analysis tools with dynamic instrumentation. In PLDI, 2005.
14. C. Malone, M. Zahran, and R. Karri. Are hardware performance counters a cost effective way for integrity checking of programs. In 6th ACM workshop on Scalable Trusted Computing, 2011.
15. V. Pappas, M. Polychronakis, and A. D. Keromytis. Transparent rop exploit mitigation using indirect branch tracing. In USENIX Security, 2013.
16. D. Pfaff et al. Learning how to prevent return-oriented programming efficiently. In Engineering Secure Software and Systems. Springer, 2015.
17. B. Scholkopf, A. Smola, and K.-R. Muller. Kernel principal component analysis. In Artificial Neural Networks - ICANN, pages 583--588. Springer, 1997.
18. F. Schuster et al. Counterfeit object-oriented programming: On the difficulty of preventing code reuse attacks in c++ applications. In Security & Privacy. IEEE, 2015.
19. H. Shacham. The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In 14th CCS. ACM, 2007.
20. C. Smutz and A. Stavrou. When a tree falls: Using diversity in ensemble classifiers to identify evasion in malware detectors. In NDSS, 2016.
21. P. Stewin. A primitive for revealing stealthy peripheral-based attacks on the computing platform's main memory. In RAID. Springer, 2013.
22. A. Tang, S. Sethumadhavan, and S. J. Stolfo. Unsupervised anomaly-based malware detection using hardware features. In RAID. Springer, 2014.
23. M. Tran et al. On the expressiveness of return-into-libc attacks. In RAID. Springer, 2011.
24. K. Wang, J. J. Parekh, and S. J. Stolfo. Anagram: A content anomaly detector resistant to mimicry attack. In RAID. Springer, 2006.

Index Terms

• Computing methodologies

  • Machine learning

    • Machine learning approaches

      • Kernel methods

• Security and privacy

  • Intrusion/anomaly detection and malware mitigation

  • Software and application security