Browse Books

Reviewer: Robert L. Glass

The good news about this book is that: it provides an extremely thorough description of a vitally important way of optimizing the number of test cases needed to test a piece of software; it comes straight from the horse's mouth (the people who did the research that led to the approach); and it includes the experimental findings on the applications and value of the approach. The bad news is that applying the approach to the usual piece of application software is difficult and very application-dependent. There is no magic formula here! First, consider what is usually meant by combinatorial testing. There are at least three possible meanings here. The one featured in this book involves the necessity of testing combinations of input data to detect faults that arise when certain combinations interact. Another possible meaning, also discussed in the book, accounts for the possible combinations of configurations the software may run on (the book notes that this is probably the most common present use of combinatorial testing in practice). The third meaning regards the program logic itself, where testing of the execution of various logic segments must include combinations of those segments. The book also addresses this meaning (but later on) under the topic of "test coverage," and explains why that approach is black box (independent of knowledge of the source code) versus white box (requires knowledge of the source code). I was particularly pleased that this subject was covered in the book, since I assumed this was what the book was about when I signed up to review it! The most important aspect of combinatorial testing covered in the book involves research results produced by the National Institute of Science and Technology (NIST), where the authors are employed. They summarize those results in their "Interaction Rule," which states that "most failures are induced by single-factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors." (Put more simply, conditional testing can be restricted to a few key factors.) They also note that "the maximum degree of interaction in actual real-world faults observed is six." (In other words, combinatorial testing doesn't have to escalate the way you might think it must.) All of that, of course, provides a basis for the success of combinatorial testing. But now, for the bad news. Defining combinatorial test cases for a given piece of software is extremely application-dependent. There are, as previously noted, no magic rules in this book. The book makes up for this in the best way possible. It discusses almost all of the imaginable topics regarding the practical use of the approach, offers numerous case studies to illustrate its ideas, supplements each with highlighted rules of thumb for applying the approach in the context being described, provides sections on cost and practical considerations, and includes a well-written chapter summary along with some review questions for each topic (chapter). In these chapters, the authors: evaluate current test approaches to measure their effectiveness compared to combinatorial approaches; order test executions to minimize the time it takes to identify faults; describe event-driven applications, where the ordering of execution of the test cases may be crucial; compare the effectiveness of random test generation with the previously described combinatorial approach, surprisingly finding that randomly generated tests approach the effectiveness of combinatorial ones as system complexity increases (the authors also note that randomly generated tests are significantly cheaper than combinatorial ones); discuss the role of in-program, dynamic assertions in test efforts, noting that they are a dynamic and rather weak form of proof of correctness whose value may supplement combinatorial testing; and repeatedly address the test oracle problem, which is the need to have accurate test results for any test case to consider that test to be useful (this is a monumental problem for any testing approach, since it means that correct results must be known for each test case if the execution of that test case is to prove useful). Throughout the book, the authors present research findings about combinatorial testing in addition to those already noted above. Combinatorial testing, they note, provides a "pseudo-exhaustive" approach to testing where a completely exhaustive approach is usually impossible, using roughly 95 percent fewer test cases and producing "equally successful" results. They further note that "the larger the system, the greater the benefits of combinatorial testing." I was prepared at the outset to dislike this book. I expected a different subject. It was written by researchers who may or may not have been able to relate to practitioners. It was about a subject that seemed virtually impossible to make practical. But in the end, it turned out to be on a quite different subject, and the authors overcame all my concerns. I thoroughly recommend it to anyone involved in the practice of software testing. Online Computing Reviews Service