R. Krishnan
Abstract
Combinatorial testing methods address generation of test cases for problems involving multiple parameters and combinations. The Orthogonal Array Based Testing Strategy (OATS) is one such combinatorial testing method, a systematic, statistical way of testing pair-wise interactions. It provides representative (uniformly distributed) coverage of all variable pair combinations. This makes the technique particularly useful for testing of software, wherever there is combinatorial explosion: a. In system testing for handling feature interactions b. In integration testing components c. It is also quite useful for testing products with a large number of configuration possibilities.
One of the fundamental assumptions behind OATS approach is that a subset covering all pair-wise combinations will be more effective than a randomly selected subset. OATS provides a means to select a minimal test set that guarantees testing the pair-wise combinations of all the selected variables. Covering pair-wise combinations has been reported to be very effective in the literature. Successful use of this technique, with 50% effort saving and improved testing with a factor of 2.6 is reported in the literature.
In this paper, we report on the in-house web-based application that we designed and implemented to support customized version of OATS and our experience in piloting and getting this method used in projects. In the in-house tool we have introduced a number of additional features, that help in generation and post processing of test-cases. We have also designed a supporting process for using this method, and we discuss the steps in this process in the paper. We share details on application in feature testing of a mobile phone application. This method has also been successfully used in designing feature interaction test cases and for augmenting the regression suite to increase coverage.
- Madhav S Phadke, Phadke Associates Inc: Planning Efficient Software Tests. In Crosstalk, October 97, Vol. 10, No. 10, pp.11-15Google Scholar
- David M. Cohen (1997) :The AETG System: An approach to Testing based on Combinatorial Desig. In IEEE Transactions on Software Engineering, Vol 23, No 7, 1997. Google ScholarDigital Library
- D. Richard Kuhn et.al (2002): An Investigation of the Applicability of Design of Experiments to Software Testing. In Proceedings, 27th NASA Goddard Space Flight Center, 4-6 December, 2002. Google ScholarDigital Library
- Library of Orthogonal Arrays. In http://www.research.att.com/~njas/oadir/Google Scholar
- Myra B. Cohen et.al (2003): Constructing test suites for interaction testing. In Proc. of the Intl. Conf. on Software Engineering (ICSE 2003), Portland, Oregon, May 2003, pp. 38-48 . Google ScholarDigital Library
Index Terms
- Combinatorial testing: learnings from our experience
Recommendations
Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied ComputingThe behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
A combinatorial testing strategy for concurrent programs
One approach to testing concurrent programs is called reachability testing, which derives test sequences automatically and on-the-fly, without constructing a static model. Existing reachability testing algorithms are exhaustive in that they are intended ...
A Combinatorial Test Suite Generator for Gray-Box Testing
SMC-IT '09: Proceedings of the Third IEEE International Conference on Space Mission Challenges for Information TechnologyIn black-box testing, the system being tested is typically characterized as a number of inputs, where each input can take one of a number of values. Thus each test is a vector of input settings, and the set of possible tests is an N dimensional space, ...
Comments