Combinatorial Testing must be an integral part of software validation because it not only reduces cost but increases the efficacy of your test strategies. This is even more important since the complexity of software apps are increasing exponentially.
Pairwise (2-way) combinatorial testing may be sufficient for low risk software applications, but a 4-way or a 6-way testing may be required for mission critical software. The main idea behind combinatorial testing is that software bugs are discovered more effectively when it is tested using combinations of inputs or configurations. For example, a software might work fine when one parameter input changes but may fail when two or more input values change in combinations.
To illustrate the effectiveness of combinatorial testing, let us consider the following example (Note: this example is based on an illustration in Information Security: Practical Combinatorial Testing, NIST Special Publication 800-142). For example, let us suppose we need to validate a software application that is designed to run on 3 OSes (Windows Server, Apple OS, Red Hat Linux), supports 3 browsers (Chrome, IE, FireFox) and 3 types of DBMSes (MySQL, SQL Server, Oracle). If you were to validate this software application on all combinations of platforms, you will have a total of 27 platforms (3*3*3). With only 9 tests as shown in the table, it is possible to test every component interacting with every other component at least once, i.e., all possible pairs of platform components are covered.
The above example is a very simple illustration. The complexity of test case design can increase exponentially when you are dealing with many more parameters. For example, if a screen has 8 input fields as follows:
- Five (5) Text Fields (You would like to test at least 2 values for each of the 5 fields) - Drop Down Field 1 - 5 Options - Drop Down Field 2 - 40 Options - Drop Down Field 3 - 11 Options
In order to run a full combinatorial testing on such a page, even with just 2 values for each of the 5 text fields, there were be more than 500,000 different combinations. If you employ pairwise testing, you could reduce these combinations to just 110 test cases.
Combinatorial testing is your reliable ally in effective test design. But you will require a good test data designer tool to generate various test cases to support n-pairwise testing.