Tsong Yueh Chen
Dave Towey
T. H. Tse
Skip Abstract Section
Abstract
Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this article, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research.
- Alexander Alexandrov, Christoph Brücke, and Volker Markl. 2013. Issues in big data testing and benchmarking. In Proceedings of the 6th International Workshop on Testing Database Systems (DBTest’13). ACM, New York, NY, 1:1--1:5.Google Scholar
Digital Library
- Paul E. Ammann and John C. Knight. 1988. Data diversity: An approach to software fault tolerance. IEEE Transactions on Computers 37, 4, 418--425.Google ScholarDigital Library
- Chittineni Aruna and R. Siva Ram Prasad. 2014. Testing approach for dynamic web applications based on automated test strategies. In ICT and Critical Infrastructure: Proceedings of the 48th Annual Convention of Computer Society of India, Vol II, Advances in Intelligent Systems and Computing, Vol. 249. Springer, Berlin, 399--410.Google Scholar
- Earl T. Barr, Mark Harman, Phil McMminn, Muzammil Shahbaz, and Shin Yoo. 2015. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering 41, 5, 507--525.Google ScholarDigital Library
- Arlinta Christy Barus. 2010. An In-Depth Study of Adaptive Random Testing for Testing Program with Complex Input Types. Ph.D. Thesis. Faculty of Information and Communication Technologies, Swinburne University of Technology, Melbourne, Australia.Google Scholar
- Arlinta Christy Barus, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Robert Merkel, and Gregg Rothermel. 2016. A cost-effective random testing method for programs with non-numeric inputs. IEEE Transactions on Computers 65, 12, 3509--3523.Google ScholarDigital Library
- Arlinta Christy Barus, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, and Heinz W. Schmidt. 2016. The impact of source test case selection on the effectiveness of metamorphic testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 5--11.Google Scholar
- Sami Beydeda. 2006. Self-metamorphic-testing components. In Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Vol. 1. IEEE Computer Society, Los Alamitos, CA, 265--272.Google ScholarDigital Library
- Barry W. Boehm. 1984. Verifying and validating software requirements and design specifications. IEEE Software 1, 1, 75--88.Google ScholarDigital Library
- Rajkumar Buyya, Chee Shin Yeo, Srikumar Venugopal, James Broberg, and Ivona Brandic. 2009. Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future Generation Computer Systems 25, 6, 599--616.Google ScholarDigital Library
- Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: Three decades later. Communications of the ACM 56, 2, 82--90.Google ScholarDigital Library
- Yuxiang Cao, Zhi Quan Zhou, and Tsong Yueh Chen. 2013. On the correlation between the effectiveness of metamorphic relations and dissimilarities of test case executions. In Proceedings of the 13th International Conference on Quality Software (QSIC’13). IEEE Computer Society, Los Alamitos, CA, 153--162.Google ScholarDigital Library
- Wing Kwong Chan, Tsong Yueh Chen, Heng Lu, T. H. Tse, and Stephen S. Yau. 2006. Integration testing of context-sensitive middleware-based applications: A metamorphic approach. International Journal of Software Engineering and Knowledge Engineering 16, 5, 677--703.Google ScholarCross Ref
- Wing Kwong Chan, Shing Chi Cheung, and Karl R. P. H. Leung. 2007. A metamorphic testing approach for online testing of service-oriented software applications. International Journal of Web Services Research 4, 2, 60--80.Google ScholarCross Ref
- Tsong Yueh Chen, Shing Chi Cheung, and Siu Ming Yiu. 1998. Metamorphic testing: A new approach for generating next test cases. Technical Report HKUST-CS98-01. Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong.Google Scholar
- Tsong Yueh Chen, Joshua W. K. Ho, Huai Liu, and Xiaoyuan Xie. 2009. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 10, article no. 24.Google Scholar
- Tsong Yueh Chen, De Hao Huang, T. H. Tse, and Zhi Quan Zhou. 2004. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC’04). Polytechnic University of Madrid, Madrid, Spain, 569--583.Google Scholar
- Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, and Shengqiong Wang. 2009. Conformance testing of network simulators based on metamorphic testing technique. In Proceedings of the Joint 11th IFIP WG 6.1 International Conference (FMOODS’09) and 29th IFIP WG 6.1 International Conference on Formal Techniques for Distributed Systems (FORTE’09), Lecture Notes in Computer Science, Vol. 5522. Springer, Berlin, 243--248.Google ScholarDigital Library
- Tsong Yueh Chen, Fei-Ching Kuo, Ying Liu, and Antony Tang. 2004. Metamorphic testing and testing with special values. In Proceedings of the 5th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing (SNPD’04). International Association for Computer and Information Science, Mt. Pleasant, MI, 128--134.Google Scholar
- Tsong Yueh Chen, Fei-Ching Kuo, Wenjuan Ma, Willy Susilo, Dave Towey, Jeffrey Voas, and Zhi Quan Zhou. 2016. Metamorphic testing for cybersecurity. Computer 49, 6, 48--55.Google ScholarDigital Library
- Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive random testing: The ART of test case diversity. Journal of Systems and Software 83, 1, 60--66.Google ScholarDigital Library
- Tsong Yueh Chen, Fei-Ching Kuo, Dave Towey, and Zhi Quan Zhou. 2015. A revisit of three studies related to random testing. Science China Information Sciences 58, 5, 052104:1--052104:9.Google Scholar
- Tsong Yueh Chen, Fei-Ching Kuo, and Zhi Quan Zhou. 2005. An effective testing method for end-user programmers. In Proceedings of the 1st Workshop on End-User Software Engineering (WEUSE’05). ACM, New York, NY, 21--25.Google ScholarDigital Library
- Tsong Yueh Chen, Pak-Lok Poon, and T. H. Tse. 2003. A choice relation framework for supporting category-partition test case generation. IEEE Transactions on Software Engineering 29, 7, 577--593.Google ScholarDigital Library
- Tsong Yueh Chen, Pak-Lok Poon, and Xiaoyuan Xie. 2016. METRIC: METamorphic Relation Identification based on the Category-choice framework. Journal of Systems and Software 116, 177--190.Google ScholarDigital Library
- Tsong Yueh Chen, T. H. Tse, and Yuen Tak Yu. 2001. Proportional sampling strategy: A compendium and some insights. Journal of Systems and Software 58, 1, 65--81.Google ScholarDigital Library
- Tsong Yueh Chen, T. H. Tse, and Zhi Quan Zhou. 2002. Semi-proving: An integrated method based on global symbolic evaluation and metamorphic testing. In Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’02). ACM, New York, NY, 191--195.Google ScholarDigital Library
- Tsong Yueh Chen, T. H. Tse, and Zhi Quan Zhou. 2003. Fault-based testing without the need of oracles. Information and Software Technology 45, 1, 1--9.Google ScholarDigital Library
- Tsong Yueh Chen, T. H. Tse, and Zhi Quan Zhou. 2011. Semi-proving: An integrated method for program proving, testing, and debugging. IEEE Transactions on Software Engineering 37, 1, 109--125.Google ScholarDigital Library
- David Cohen, Mikael Lindvall, and Patricia Costa. 2004. An introduction to agile methods. In Advances in Computers, Vol. 62. Elsevier, Amsterdam, The Netherlands, 1--66.Google Scholar
- Edsger W. Dijkstra. 1972. The humble programmer. Communications of the ACM 15, 10, 859--866.Google ScholarDigital Library
- Junhua Ding, Xin-Hua Hu, and Venkat Gudivada. 2017. A machine learning based framework for verification and validation of massive scale image data. IEEE Transactions on Big Data. DOI:10.1109/TBDATA.2017.2680460.Google Scholar
- Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2008. Case studies on testing with compositional metamorphic relations. Journal of Southeast University (English Edition) 24, 4, 437--443.Google Scholar
- Guowei Dong, Baowen Xu, Lin Chen, Changhai Nie, and Lulu Wang. 2009. Survey of metamorphic testing. Journal of Frontiers of Computer Science and Technology 3, 2, 130--143.Google Scholar
- Sebastian Elbaum, Hui Nee Chin, Matthew B. Dwyer, and Jonathan Dokulil. 2006. Carving differential unit test cases from system test cases. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 253--264.Google ScholarDigital Library
- Matthew Geller. 1978. Test data as an aid in proving program correctness. Communications of the ACM 21, 5, 368--375.Google ScholarDigital Library
- Arnaud Gotlieb and Bernard Botella. 2003. Automated metamorphic testing. In Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC’03). IEEE Computer Society, Los Alamitos, CA, 34--40.Google ScholarCross Ref
- Ralph Guderlei and Johannes Mayer. 2007. Statistical metamorphic testing: Testing programs with random output by means of statistical hypothesis tests and metamorphic testing. In Proceedings of the 7th International Conference on Quality Software (QSIC’07). IEEE Computer Society, Los Alamitos, CA, 404--409.Google ScholarCross Ref
- Richard Hamlet. 2002. Random testing. In Encyclopedia of Software Engineering. John Wiley, New York, NY.Google Scholar
- Mark Harman, Yue Jia, and Yuanyuan Zhang. 2015. Achievements, open problems and challenges for search based software testing. In Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation (ICST’15). IEEE Computer Society, Los Alamitos, CA.Google ScholarCross Ref
- Peifeng Hu, Zhenyu Zhang, Wing Kwong Chan, and T. H. Tse. 2006. An empirical comparison between direct and indirect test result checking approaches. In Proceedings of the 3rd International Workshop on Software Quality Assurance (SOQUA’06) in Conjunction with the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 6--13.Google Scholar
- Zhan-Wei Hui and Song Huang. 2013. Achievements and challenges of metamorphic testing. In Proceedings of the 4th World Congress on Software Engineering (WCSE’13). IEEE Computer Society, Los Alamitos, CA, 73--77.Google ScholarDigital Library
- Monica Hutchins, Herb Foster, Tarak Goradia, and Thomas Ostrand. 1994. Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering (ICSE’94). IEEE Computer Society, Los Alamitos, CA, 191--200.Google ScholarDigital Library
- Mingyue Jiang, Tsong Yueh Chen, Fei-Ching Kuo, and Zuohua Ding. 2013. Testing central processing unit scheduling algorithms using metamorphic testing. In Proceedings of the 4th IEEE International Conference on Software Engineering and Service Science (ICSESS’13). IEEE Computer Society, Los Alamitos, CA, 530--536.Google Scholar
- Mingyue Jiang, Tsong Yueh Chen, Fei-Ching Kuo, Dave Towey, and Zuohua Ding. 2017. A metamorphic testing approach for supporting program repair without the need for a test oracle. Journal of Systems and Software 126, 127--140.Google ScholarCross Ref
- Hao Jin, Yanyan Jiang, Na Liu, Chang Xu, Xiaoxing Ma, and Jian Lu. 2015. Concolic metamorphic debugging. In Proceedings of the IEEE 39th Annual International Computers, Software and Applications Conference (COMPSAC’15), Vol. 2. IEEE Computer Society, Los Alamitos, CA, 232--241.Google ScholarDigital Library
- Upulee Kanewala, James M. Bieman, and Asa Ben-Hur. 2016. Predicting metamorphic relations for testing scientific software: A machine learning approach using graph kernels. Software Testing, Verification and Reliability 26, 3, 245--269.Google ScholarDigital Library
- Andrew J. Ko, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, Henry Lieberman, Brad Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, and Susan Wiedenbeck. 2011. The state of the art in end-user software engineering. ACM Computing Surveys 43, 3, 21:1--21:44.Google Scholar
- Fei-Ching Kuo, Tsong Yueh Chen, and Wing K Tam. 2011. Testing embedded software by metamorphic testing: A wireless metering system case study. In Proceedings of the IEEE 36th Conference on Local Computer Networks (LCN’11). IEEE Computer Society, Los Alamitos, CA, 291--294.Google ScholarDigital Library
- Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’14). ACM, New York, NY, 216--226.Google ScholarDigital Library
- Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core compiler fuzzing. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’15). ACM, New York, NY, 65--76.Google Scholar
- Mikael Lindvall, Dharmalingam Ganesan, Ragnar Árdal, and Robert E. Wiegand. 2015. Metamorphic model-based testing applied on NASA DAT: An experience report. In Proceedings of the 37th International Conference on Software Engineering (ICSE’15), Vol. 2. IEEE, Piscataway, NJ, 129--138.Google ScholarDigital Library
- Mikael Lindvall, Dharmalingam Ganesan, Sigurthor Bjorgvinsson, Kristjan Jonsson, Haukur Steinn Logason, Frederik Dietrich, and Robert E. Wiegand. 2016. Agile metamorphic model-based testing. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 26--32.Google Scholar
- Huai Liu, Fei-Ching Kuo, and Tsong Yueh Chen. 2010. Teaching an end-user testing methodology. In Proceedings of the 23rd IEEE Conference on Software Engineering Education and Training (CSEE&T’’10). IEEE Computer Society, Los Alamitos, CA, 81--88.Google ScholarDigital Library
- Huai Liu, Fei-Ching Kuo, Dave Towey, and Tsong Yueh Chen. 2014. How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering 40, 1, 4--22.Google ScholarDigital Library
- Huai Liu, Xuan Liu, and Tsong Yueh Chen. 2012. A new method for constructing metamorphic relations. In Proceedings of the 12th International Conference on Quality Software (QSIC’12). IEEE Computer Society, Los Alamitos, CA, 59--68.Google ScholarDigital Library
- Huai Liu, Iman I. Yusuf, Heinz W. Schmidt, and Tsong Yueh Chen. 2014. Metamorphic fault tolerance: An automated and systematic methodology for fault tolerance in the absence of test oracle. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion’14). ACM, New York, NY, 420--423.Google ScholarDigital Library
- Heng Lu, Wing Kwong Chan, and T. H. Tse. 2006. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’06/FSE-14). ACM, New York, NY, 242--252.Google Scholar
- Xiaoli Lu, Yunwei Dong, and Chao Luo. 2010. Testing of component-based software: A metamorphic testing methodology. In Proceedings of the 7th International Conference on Ubiquitous Intelligence and Computing and the 7th International Conference on Autonomic and Trusted Computing (UIC/ATC’10). IEEE Computer Society, Los Alamitos, CA, 272--276.Google ScholarDigital Library
- Ieng Kei Mak. 1997. On the Effectiveness of Random Testing. Master’s Thesis. Department of Computer Science, University of Melbourne, Melbourne, Australia.Google Scholar
- L. I. Manolache and D. G. Kourie. 2001. Software testing using model programs. Software: Practice and Experience 31, 13, 1211--1236.Google ScholarDigital Library
- Johannes Mayer and Ralph Guderlei. 2006. An empirical study on the selection of good metamorphic relations. In Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Vol. 1. IEEE Computer Society, Los Alamitos, CA, 475--484.Google ScholarDigital Library
- Kunal Swaroop Mishra and Gail E. Kaiser. 2012. Effectiveness of Teaching Metamorphic Testing. Technical Report CUCS-020-12. Department of Computer Science, Columbia University, New York, NY.Google Scholar
- Kunal Swaroop Mishra, Gail E. Kaiser, and Swapneel Kalpesh Sheth. 2013. Effectiveness of Teaching Metamorphic Testing, Part II. Technical Report CUCS-022-13. Department of Computer Science, Columbia University, New York, NY.Google Scholar
- Christian Murphy, Gail Kaiser, Lifeng Hu, and Leon Wu. 2008. Properties of machine learning applications for use in metamorphic testing. In Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE’08). Knowledge Systems Institute Graduate School, Skokie, IL, 867--872.Google Scholar
- Christian Murphy, M. S. Raunak, Andrew King, Sanjian Chen, Christopher Imbriano, Gail Kaiser, Insup Lee, Oleg Sokolsky, Lori Clarke, and Leon Osterweil. 2011. On effective testing of health care simulation software. In Proceedings of the 3rd Workshop on Software Engineering in Health Care (SEHC’11). ACM, New York, NY, 40--47.Google ScholarDigital Library
- Shin Nakajima and Hai Ngoc Bui. 2016. Dataset coverage for testing machine learning computer programs. In Proceedings of the 2016 Asia-Pacific Software Engineering Conference (APSEC’16). IEEE Computer Society, Los Alamitos, CA, 297--304.Google ScholarCross Ref
- Rafael A. P. Oliveira, Upulee Kanewala, and Paulo A. Nardi. 2015. Automated test oracles: State of the art, taxonomies, and trends. In Advances in Computers, Vol. 95. Elsevier, Amsterdam, The Netherlands, 113--199.Google Scholar
- Alessandro Orso and Gregg Rothermel. 2014. Software testing: A research travelogue (2000--2014). In Proceedings of the Future of Software Engineering (FOSE’14). ACM, New York, NY, 117--132.Google ScholarDigital Library
- Carlos E. Otero and Adrian Peter. 2015. Research directions for engineering big data analytics software. IEEE Intelligent Systems 30, 1, 13--19.Google ScholarDigital Library
- Krishna Patel and Robert M. Hierons. submitted for publication. A systematic literature review on testing and debugging non-testable systems. Retrieved from http://people.brunel.ac.uk/∼csstrmh/Intt/synth.pdf.Google Scholar
- Mauro Pezzè and Cheng Zhang. 2014. Automated test oracles: A survey. In Advances in Computers, Vol. 95. Academic Press, Waltham, MA, 1--48.Google Scholar
- Pak-Lok Poon, Fei-Ching Kuo, Huai Liu, and Tsong Yueh Chen. 2014. How can non-technical end users effectively test their spreadsheets? Information Technology and People 27, 4, 440--462.Google ScholarCross Ref
- Pak-Lok Poon, Huai Liu, and Tsong Yueh Chen. 2017. Error trapping and metamorphic testing for spreadsheet failure detection. Journal of Organizational and End User Computing 29, 2, 25--42.Google ScholarDigital Library
- Laura L. Pullum and Ozgur Ozmen. 2012. Early results from metamorphic testing of epidemiological models. In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12). IEEE Computer Society, Los Alamitos, CA, 62--67.Google Scholar
- Arvind Ramanathan, Chad A. Steed, and Laura L. Pullum. 2012. Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics. In Proceedings of the 2012 ASE/IEEE International Conference on BioMedical Computing (BioMedCom’12). IEEE Computer Society, Los Alamitos, CA, 68--73.Google Scholar
- Peifeng Rao, Zheng Zheng, Tsong Yueh Chen, Nan Wang, and Kai-Yuan Cai. 2013. Impacts of test suite’s class imbalance on spectrum-based fault localization techniques. In Proceedings of the 13th International Conference on Quality Software (QSIC’13). IEEE Computer Society, Los Alamitos, CA, 260--267.Google ScholarDigital Library
- John Regehr. 2014. Finding compiler bugs by removing dead code. Retrieved from http://blog.regehr.org/archives/1161.Google Scholar
- David S. Rosenblum. 1995. A practical approach to programming with assertions. IEEE Transactions on Software Engineering 21, 1, 19--31.Google ScholarDigital Library
- Md. Shaik Sadi, Fei-Ching Kuo, Joshua W. K. Ho, Michael A. Charleston, and Tsong Yueh Chen. 2011. Verification of phylogenetic inference programs using metamorphic testing. Journal of Bioinformatics and Computational Biology 9, 6, 729--747.Google ScholarCross Ref
- Sergio Segura, Gordon Fraser, Ana B. Sánchez, and Antonio Ruiz-Cortés. 2016. A survey on metamorphic testing. IEEE Transactions on Software Engineering 42, 9, 805--824.Google ScholarCross Ref
- Sergio Segura, Robert M. Hierons, David Benavides, and Antonio Ruiz-Cortés. 2010. Automated test data generation on the analyses of feature models: A metamorphic testing approach. In Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST’10). IEEE Computer Society, Los Alamitos, CA, 35--44.Google ScholarDigital Library
- Sergio Segura, Robert M. Hierons, David Benavides, and Antonio Ruiz-Cortés. 2011. Automated metamorphic testing on the analyses of feature models. Information and Software Technology 53, 3, 245--258.Google ScholarDigital Library
- Chang-Ai Sun, Guan Wang, Baohong Mu, Huai Liu, Zhaoshun Wang, and Tsong Yueh Chen. 2011. Metamorphic testing for web services: Framework and a case study. In Proceedings of the 2011 IEEE International Conference on Web Services (ICWS’11). IEEE Computer Society, Los Alamitos, CA, 283--290.Google ScholarDigital Library
- Systems and software engineering: Systems and software Quality Requirements and Evaluation (SQuaRE): System and software quality models. ISO/IEC 25010:2011, ISO.Google Scholar
- Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An automatic testing approach for compiler based on metamorphic testing technique. In Proceedings of the 2010 Asia Pacific Software Engineering Conference (APSEC’10). IEEE Computer Society, Los Alamitos, CA, 270--279.Google ScholarDigital Library
- Dave Towey and Tsong Yueh Chen. 2015. Teaching software testing skills: Metamorphic testing as vehicle for creativity and effectiveness in software testing. In Proceedings of the 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE’15). IEEE Computer Society, Los Alamitos, CA, 161--162.Google ScholarCross Ref
- Dave Towey, Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, and Zhi Quan Zhou. 2016. Metamorphic testing: A new student engagement approach for a new software testing paradigm. In Proceedings of the 2016 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE’16). IEEE Computer Society, Los Alamitos, CA, 228--235.Google ScholarCross Ref
- Michael Troup, Andrian Yang, Amir Hossein Kamali, Eleni Giannoulatou, Tsong Yueh Chen, and Joshua W. K. Ho. 2016. A cloud-based framework for applying metamorphic testing to a bioinformatics pipeline. In Proceedings of the 1st International Workshop on Metamorphic Testing (MET’16). ACM, New York, NY, 33--36.Google Scholar
- T. H. Tse, Stephen S. Yau, Wing Kwong Chan, Heng Lu, and Tsong Yueh Chen. 2004. Testing context-sensitive middleware-based software applications. In Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC’04), Vol. 1. IEEE Computer Society, Los Alamitos, CA, 458--465.Google ScholarCross Ref
- Xiaoyuan Xie, Joshua W. K. Ho, Christian Murphy, Gail E. Kaiser, Baowen Xu, and Tsong Yueh Chen. 2011. Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software 84, 4, 544--558.Google ScholarDigital Library
- Xiaoyuan Xie, W. Eric Wong, Tsong Yueh Chen, and Baowen Xu. 2011. Spectrum-based fault localization: Testing oracles are no longer mandatory. In Proceedings of the 11th International Conference on Quality Software (QSIC’11). IEEE Computer Society, Los Alamitos, CA, 1--10.Google ScholarDigital Library
- Xiaoyuan Xie, W. Eric Wong, Tsong Yueh Chen, and Baowen Xu. 2013. Metamorphic slice: An application in spectrum-based fault localization. Information and Software Technology 55, 5, 866--879.Google ScholarDigital Library
- Baowen Xu, Ju Qian, Xiaofang Zhang, Zhongqiang Wu, and Lin Chen. 2005. A brief survey of program slicing. ACM SIGSOFT Software Engineering Notes 30, 2, 1--36.Google ScholarDigital Library
- Iman I. Yusuf, Ian E. Thomas, Maria Spichkova, Steve Androulakis, Grischa R. Meyer, Daniel W. Drumm, George Opletal, Salvy P. Russo, Ashley M. Buckle, and Heinz W. Schmidt. 2015. Chiminey: Reliable computing and data management platform in the cloud. In Proceedings of the 37th International Conference on Software Engineering (ICSE’15), Vol. 2. IEEE, Piscataway, NJ, 677--680.Google Scholar
- Jie Zhang, Junjie Chen, Dan Hao, Yingfei Xiong, Bing Xie, Lu Zhang, and Hong Mei. 2014. Search-based inference of polynomial metamorphic relations. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE’14). ACM, New York, NY, 701--712.Google ScholarDigital Library
- Zhenyu Zhang, Wing Kwong Chan, T. H. Tse, and Peifeng Hu. 2009. Experimental study to compare the use of metamorphic testing and assertion checking. Journal of Software 20, 10, 2637--2654.Google ScholarCross Ref
- Zhi Quan Zhou, Shaowen Xiang, and Tsong Yueh Chen. 2016. Metamorphic testing for software quality assessment: A study of search engines. IEEE Transactions on Software Engineering 42, 3, 264--284.Google ScholarDigital Library
- Zhi Quan Zhou, ShuJia Zhang, Markus Hagenbuchner, T. H. Tse, Fei-Ching Kuo, and Tsong Yueh Chen. 2012. Automated functional testing of online search services. Software Testing, Verification and Reliability 22, 4, 221--243.Google ScholarDigital Library
- Hong Zhu. 2015. JFuzz: A tool for automated Java unit testing based on data mutation and metamorphic testing methods. In Proceedings of the 2nd International Conference on Trustworthy Systems and Their Applications (TSA’15). IEEE Computer Society, Los Alamitos, CA, 8--15.Google ScholarDigital Library
- Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software unit test coverage and adequacy. ACM Computing Surveys 29, 4, 366--427.Google ScholarDigital Library
Index Terms
- Metamorphic Testing: A Review of Challenges and Opportunities
Recommendations
Feedback-Directed Metamorphic Testing
Over the past decade, metamorphic testing has gained rapidly increasing attention from both academia and industry, particularly thanks to its high efficacy on revealing real-life software faults in a wide variety of application domains. On the basis of a ...
An application of metamorphic testing for testing scientific software
MET '16: Proceedings of the 1st International Workshop on Metamorphic TestingOne of the grand challenges for adequately testing scientific software is due to the oracle problem. Metamorphic testing is the promise technique for addressing the problem through checking the satisfaction of a metamorphic relation that defines the ...
Application of metamorphic testing monitored by test adequacy in a Monte Carlo simulation program
One of the grand challenges for adequately testing complex software is due to the oracle problem. Metamorphic Testing (MT) is a promising technique to alleviate the oracle problem through using one or multiple Metamorphic Relations (MRs) as test ...
Reviews
Jacques Carette
Testing software remains a complex business, so finding different means for doing so is important. Metamorphic testing belongs in one's toolkit, along with unit tests and property-based testing. Thus, overviews are particularly welcome. Here, many of the most prolific authors on the subject have gotten together to review the topic, its challenges, and the opportunities still open to be tackled. Unfortunately, the resulting paper is deeply flawed. For example, a proper definition of metamorphic testing (MT) doesn't occur until page 4, but an understanding of MT is used throughout pages 2 and 3. Some of the obvious context behind MT, such as there being no specification for a program available, are not stated explicitly. Concretely, the oracle problem would be quite different if a specification were available. The example after Definition 3 (p. 5) is confusing because the notation uses comma-separated tuples for two different things; it is not incorrect, just a poor choice of notation. Definition 4 is vague about the roles of f and P . In other words, the authors assume too much implicit knowledge on the part of the reader. Surprising for a paper in ACM Computing Surveys , the English is quite rough. And the tone is odd. For example, Section 3, "Frequently Misunderstood Concepts in MT," comes off as being very defensive, while some of the challenges in Section 5 are so vague as to be quite hard to understand. There is also a fair amount of repetition (for example, the Siemens programs are mentioned four separate times). Worse still: Example 3 (p. 10) represents bad numerical analysis that is taught as incorrect to undergraduate students. Someone interested in a survey of metamorphic testing, including some of the more speculative material covered here, would be better advised to read [1].
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments