xLM for TestOps - Continuous Qualification for AWS/Azure

What is TestOps? How does it help you continuously qualify AWS/Azure?

TestOps is a software validation culture and practice which aims at shorter testing and validation cycles to continuously qualify any cloud IaaS/PaaS service, in close alignment with business and regulatory objectives.

Many life-science companies are asking the question “Can I move my GxP workloads into the public cloud?” Answering “yes” to this question leads to another question: “What toolset do I need to qualify the various IaaS/PaaS services that are constantly changing?”.

Our customers are leveraging our robust framework that we call TestOps. TestOps provides an end to end automation for continuously qualifying any IaaS/PaaS service running on AWS or Azure.

TestOps - How does it work?

We test “continuously” while you operate your cloud infrastructure!

In order to illustrate our TestOps framework, a simple use case to continuously qualify AWS DynamoDB is presented. Keep in mind that this framework can be applied to any IaaS/PaaS service or a combination of services.

Step 1 - Define the Intended Use

An IaaS or PaaS service can be configured to be used in many ways to meet varying end user needs. It is important to document the intended use to cover the various scenarios the service will be utilized in. This intended use forms the basis for designing a qualification strategy.

Figure 1: Intended Use for AWS DynamoDB

Step 2 - Design the Test Cases in the Coding Framework

Conventional wisdom directs us to develop test cases outside of coding using Microsoft Word as the authoring tool. This approach is surely outdated and does not provide “hard” traceability to your code. xLM’s TestOps supports “Given-When-Then” framework wherein the validation analyst can build test cases within the coding environment (IDE). These test cases are directly linked to the corresponding test automation code by the developers. In other words, both validation analysts and developers are working within the same environment and test case definitions are not separate from the code.

Figure 2: Test Case written using Given-When-Then Framework

Step 3 - Coding the Test Cases

Once the test case design is complete, the developers code the testing scenarios which typically involves driving the browser for UI Testing or leveraging the APIs. The code that is developed is organized into reusable snippets (features, pages, utilities, etc.) for better maintainability of the test automation model. As mentioned above, the traceability between test case design and code is built into the framework.

Once the code development, code reviews, testing and quality checks are complete, the code is checked into the Code Factory.

Figure 3: Code Build and Publish


Step 4: Build the TestOps Pipeline

Once the code is published, the TestOps Pipeline is configured:

  • to retrieve the published code

  • to deploy the qualified code into the test environment

  • to run the test automation code to qualify the SUT (System Under Test - in this case AWS DynamoDB)

  • to generate a test automation report aka “executed protocol”

  • to archive all the test run artifacts

Figure 4: TestOps Pipeline Stages


Step 5: Run the TestOps Pipeline

Once the Pipeline is fully configured and released by Quality, the Pipeline can be run either manually or automatically (scheduled or event based). The Pipeline run-time steps include:

  • Obtaining the Pre-approval to run the “qualified” Pipeline

  • Running the test automation code and generating the execution report

Figure 5: TestOps Pipeline Pre-deployment Approval


Figure 6: TestOps Pipeline Run History

Step 6: Review the Results

After the Pipeline run is successful, the final step is to review the test automation results to ensure all the acceptance criteria have been met. This review is documented with the Post-deployment approval.

Figure 7: Executed Test Case (Sample)


xLM’s TestOps Framework is designed to continuously qualify any IaaS/PaaS service (AWS, Azure, Google Cloud) cost efficiently utilizing test automation and pipelines. Such a framework can enable you to build a Service Catalog of “qualified” IaaS/PaaS services so that your development teams can consume these services on an on-demand basis. This model completely removes the qualification hurdle each time a project is deployed in the cloud.

This framework utilizes the following key six (6) steps:

  1. Development of the Intended Use that is traceable to all qualification models

  2. Development of the test cases using the Given-When-Then framework that is directly linked to the test automation code

  3. Development of the Test Automation Models.

  4. Configuring the Pipeline

  5. Running the Pipeline

  6. Reviewing and approving the test results