Combinatorial Testing: A Key Element in Software Validation

Combinatorial Testing must be an integral part of software validation because it not only reduces cost but increases the efficacy of your test strategies.   This is even more important since the complexity of software apps are increasing exponentially.

The key insight underlying this form of testing is that not every parameter contributes to every failure and most failures are caused by interactions between relatively few parameters.
— Information Security: Practical Combinatorial Testing, NIST Special Publication 800-142.
Testing combinations of parameters can provide highly effective fault detection. Pairwise (2-way combinations) testing is sometimes used to obtain reasonably good results at low cost, but pairwise testing may miss 10% to 40% or more of system bugs, and is thus not sufficient for mission-critical software
— Information Security: Practical Combinatorial Testing, NIST Special Publication 800-142.

Pairwise (2-way) combinatorial testing may be sufficient for low risk software applications, but a 4-way or a 6-way testing may be required for mission critical software.    The main idea behind combinatorial testing is that software bugs are discovered more effectively when it is tested using combinations of inputs or configurations.  For example, a software might work fine when one parameter input changes but may fail when two or more input values change in combinations.

To illustrate the effectiveness of combinatorial testing, let us consider the following example (Note: this example is based on an illustration in Information Security: Practical Combinatorial Testing, NIST Special Publication 800-142).  For example, let us suppose we need to validate a software application that is designed to run on 3 OSes (Windows Server, Apple OS, Red Hat Linux), supports 3 browsers (Chrome, IE, FireFox) and 3 types of DBMSes (MySQL, SQL Server, Oracle).  If you were to validate this software application on all combinations of platforms, you will have a total of 27 platforms (3*3*3).  With only 9 tests as shown in the table, it is possible to test every component interacting with every other component at least once, i.e., all possible pairs of platform components are covered.

The above example is a very simple illustration.  The complexity of test case design can increase exponentially when you are dealing with many more parameters.  For example, if a screen has 8 input fields as follows:

- Five (5) Text Fields (You would like to test at least 2 values for each of the 5 fields)
- Drop Down Field 1 - 5 Options
- Drop Down Field 2 - 40 Options
- Drop Down Field 3 - 11 Options

In order to run a full combinatorial testing on such a page, even with just 2 values for each of the 5 text fields, there were be more than 500,000 different combinations.  If you employ pairwise testing, you could reduce these combinations to just 110 test cases.

Combinatorial testing is your reliable ally in effective test design.  But you will require a good test data designer tool to generate various test cases to support n-pairwise testing.