Tomasz Kuchta
Hristina Palikareva
Cristian Cadar
Skip Abstract Section
Abstract
While developers are aware of the importance of comprehensively testing patches, the large effort involved in coming up with relevant test cases means that such testing rarely happens in practice. Furthermore, even when test cases are written to cover the patch, they often exercise the same behaviour in the old and the new version of the code. In this article, we present a symbolic execution-based technique that is designed to generate test inputs that cover the new program behaviours introduced by a patch. The technique works by executing both the old and the new version in the same symbolic execution instance, with the old version shadowing the new one. During this combined shadow execution, whenever a branch point is reached where the old and the new version diverge, we generate a test input exercising the divergence and comprehensively test the new behaviours of the new version. We evaluate our technique on the Coreutils patches from the CoREBench suite of regression bugs, and show that it is able to generate test inputs that exercise newly added behaviours and expose some of the regression bugs.
- Domagoj Babić, Lorenzo Martignoni, Stephen McCamant, and Dawn Song. 2011. Statically directed dynamic automated test generation. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’11). Google Scholar
Digital Library
- Marcel Böhme, Bruno C. D. S. Oliveira, and Abhik Roychoudhury. 2013. Partition-based regression verification. In Proceedings of the 35th International Conference on Software Engineering (ICSE’13).Google ScholarCross Ref
- Marcel Böhme, Bruno C. D. S. Oliveira, and Abhik Roychoudhury. 2013. Regression tests to expose change interaction errors. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM Symposium on the Foundations of Software Engineering (ESEC/FSE’13). Google ScholarDigital Library
- Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury. 2017. Directed greybox fuzzing. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS’17). Google ScholarDigital Library
- Marcel Böhme and Abhik Roychoudhury. 2014. Corebench: Studying complexity of regression errors. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’14).Google ScholarDigital Library
- Stefan Bucur, Vlad Ureche, Cristian Zamfir, and George Candea. 2011. Parallel symbolic execution for automated real-world software testing. In Proceedings of the 6th European Conference on Computer Systems (EuroSys’11). Google ScholarDigital Library
- Cristian Cadar and Hristina Palikareva. 2014. Shadow symbolic execution for better testing of evolving software. In Proceedings of the 36th International Conference on Software Engineering, New Ideas and Emerging Results (ICSE NIER’14). Google ScholarDigital Library
- Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: Three decades later. Commun. Assoc. Comput. Mach. 56, 2 (2013), 82–90. Google ScholarDigital Library
- Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon, and Mark Harman. 2017. An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption. In Proceedings of the 39th International Conference on Software Engineering (ICSE’17).Google ScholarDigital Library
- Marcelo d’Amorim, Steven Lauterburg, and Darko Marinov. 2007. Delta execution for efficient state-space exploration of object-oriented programs. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’07). Google ScholarDigital Library
- Leonardo De Moura and Nikolaj Bjørner. 2011. Satisfiability modulo theories: Introduction and applications. Commun. Assoc. Comput. Mach. 54, 9 (Sept. 2011), 69–77. Google ScholarDigital Library
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. In Proceedings of the Conference on Programing Language Design and Implementation (PLDI’05). Google ScholarDigital Library
- Shengjian Guo, Markus Kusano, and Chao Wang. 2016. Conc-iSE: Incremental symbolic execution of concurrent software. In Proceedings of the 31th IEEE International Conference on Automated Software Engineering (ASE’16). Google ScholarDigital Library
- Rajiv Gupta, Mary Jean Harrold, and Mary Lou Soffa. 1996. Program slicing-based regression testing techniques. Softw. Test. Verificat. Reliabil. 6 (1996), 83–112.Google ScholarCross Ref
- Kelly J. Hayhurst, Dan S. Veerhusen, John J. Chilenski, and Leanna K. Rierson. 2001. A Practical Tutorial on Modified Condition/Decision Coverage. Technical Report NASA/TM-2001-210876. NASA. Google Scholar
- Petr Hosek and Cristian Cadar. 2013. Safe software updates via multi-version execution. In Proceedings of the 35th International Conference on Software Engineering (ICSE’13). Google ScholarDigital Library
- Petr Hosek and Cristian Cadar. 2015. Varan the unbelievable: An efficient N-version execution framework. In Proceedings of the 20th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’15). Google ScholarDigital Library
- Martin Kellogg, Benjamin Floyd, Stephanie Forrest, and Westley Weimer. 2016. Combining bug detection and test case generation. In Proceedings of the ACM Symposium on the Foundations of Software Engineering (FSE’16). Google ScholarDigital Library
- Chang Hwan Peter Kim, Sarfraz Khurshid, and Don Batory. 2012. Shared execution for efficiently testing product lines. In Proceedings of the 23rd International Symposium on Software Reliability Engineering (ISSRE’12).Google Scholar
- Miryung Kim and David Notkin. 2006. Program element matching for multi-version program analyses. In Proceedings of the 2006 International Workshop on Mining Software Repositories (MSR’06). Google ScholarDigital Library
- Shuvendu K. Lahiri, Kenneth L. McMillan, Rahul Sharma, and Chris Hawblitzel. 2013. Differential assertion checking. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM Symposium on the Foundations of Software Engineering (ESEC/FSE’13). Google ScholarDigital Library
- Wei Le and Shannon D. Pattison. 2014. Patch verification via multiversion interprocedural control flow graphs. In Proceedings of the 36th International Conference on Software Engineering (ICSE’14). Google ScholarDigital Library
- Daniel Liew, Daniel Schemmel, Cristian Cadar, Alastair Donaldson, Rafael Zähl, and Klaus Wehrle. 2017. Floating-point symbolic execution: A case study in N-version programming. In Proceedings of the 32nd IEEE International Conference on Automated Software Engineering (ASE’17). Google ScholarDigital Library
- Kin-Keung Ma, Yit Phang Khoo, Jeffrey S. Foster, and Michael Hicks. 2011. Directed symbolic execution. In Proceedings of the 18th International Static Analysis Symposium (SAS’11). Google ScholarDigital Library
- Paul Dan Marinescu and Cristian Cadar. 2012. High-coverage symbolic patch testing. In Proceedings of the 19th International SPIN Workshop on Model Checking of Software (SPIN’12).Google ScholarDigital Library
- Paul Dan Marinescu and Cristian Cadar. 2012. make test-zesti: A symbolic execution solution for improving regression testing. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12).Google ScholarCross Ref
- Paul Dan Marinescu and Cristian Cadar. 2013. KATCH: High-coverage testing of software patches. In Proceedings of the joint meeting of the European Software Engineering Conference and the ACM Symposium on the Foundations of Software Engineering (ESEC/FSE’13). Google ScholarDigital Library
- Paul Dan Marinescu, Petr Hosek, and Cristian Cadar. 2014. Covrig: A framework for the analysis of code, test, and coverage evolution in real software. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’14).Google ScholarDigital Library
- Matthew Maurer and David Brumley. 2012. TACHYON: Tandem execution for efficient live patch testing. In Proceedings of the 21st USENIX Security Symposium (USENIX Security’12). Google ScholarDigital Library
- Yannic Noller, Hoang Lam Nguyen, Minxing Tang, and Timo Kehrer. 2018. Shadow symbolic execution with Java PathFinder. ACM SIGSOFT Softw. Eng. Notes 42, 4 (2018), 1–5. Google ScholarDigital Library
- Hristina Palikareva, Tomasz Kuchta, and Cristian Cadar. 2016. Shadow of a doubt: Testing for divergences between software versions. In Proceedings of the 38th International Conference on Software Engineering (ICSE’16). Google ScholarDigital Library
- Nimrod Partush and Eran Yahav. 2014. Abstract semantic differencing via speculative correlation. In Proceedings of the 29th Annual Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA’14). Google ScholarDigital Library
- Corina S. Păsăreanu, Willem Visser, David Bushnell, Jaco Geldenhuys, Peter Mehlitz, and Neha Rungta. 2013. Symbolic PathFinder: Integrating symbolic execution with model checking for java bytecode analysis. Auto. Softw. Eng. 20, 3 (Sep 2013), 391–425.Google Scholar
- Suzette Person, Matthew B. Dwyer, Sebastian Elbaum, and Corina S. Pǎsǎreanu. 2008. Differential symbolic execution. In Proceedings of the ACM Symposium on the Foundations of Software Engineering (FSE’08). Google ScholarDigital Library
- Suzette Person, Guowei Yang, Neha Rungta, and Sarfraz Khurshid. 2011. Directed incremental symbolic execution. In Proceedings of the Conference on Programing Language Design and Implementation (PLDI’11). Google ScholarDigital Library
- Dawei Qi, Abhik Roychoudhury, and Zhenkai Liang. 2010. Test generation to expose changes in evolving programs. In Proceedings of the 25th IEEE International Conference on Automated Software Engineering (ASE’10). Google ScholarDigital Library
- R. Santelices, P. K. Chittimalli, T. Apiwattanapong, A. Orso, and M. J. Harrold. 2008. Test-suite augmentation for evolving software. In Proceedings of the 23rd IEEE International Conference on Automated Software Engineering (ASE’08). Google ScholarDigital Library
- Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A concolic unit testing engine for C. In Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM Symposium on the Foundations of Software Engineering (ESEC/FSE’05). Google ScholarDigital Library
- Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitry Vyukov. 2012. AddressSanitizer: A fast address sanity checker. In Proceedings of the 2012 USENIX Annual Technical Conference (USENIX ATC’12). Google ScholarDigital Library
- Kunal Taneja, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. 2011. eXpress: Guided path exploration for efficient regression test generation. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’11). Google ScholarDigital Library
- The 50th CREST Open Workshop—Genetic Improvement 2017. Retrieved from http://crest.cs.ucl.ac.uk/cow/50/.Google Scholar
- Joseph Tucek, Weiwei Xiong, and Yuanyuan Zhou. 2009. Efficient online validation with delta execution. In Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’09). Google ScholarDigital Library
- Zhihong Xu and Gregg Rothermel. 2009. Directed test suite augmentation. In Proceedings of the 16th Asia-Pacific Software Engineering Conference (ASPEC’09). Google ScholarDigital Library
Index Terms
- Shadow Symbolic Execution for Testing Software Patches
Recommendations
Shadow of a doubt: testing for divergences between software versions
ICSE '16: Proceedings of the 38th International Conference on Software EngineeringWhile developers are aware of the importance of comprehensively testing patches, the large effort involved in coming up with relevant test cases means that such testing rarely happens in practice. Furthermore, even when test cases are written to cover ...
Using Metamorphic Testing to Improve Dynamic Symbolic Execution
ASWEC '15: Proceedings of the 2015 24th Australasian Software Engineering Conference (ASWEC)Dynamic symbolic execution (DSE) is an approach for automatically generating test inputs from source code using constraint information. It is used in fuzzing: the execution of tests while monitoring for generic properties such as buffer overflows and ...
Test generation via Dynamic Symbolic Execution for mutation testing
ICSM '10: Proceedings of the 2010 IEEE International Conference on Software MaintenanceMutation testing has been used to assess and improve the quality of test inputs. Generating test inputs to achieve high mutant-killing ratios is important in mutation testing. However, existing test-generation techniques do not provide effective support ...
Comments