xLM TestOps Case Study

What is TestOps? How does it comply with FDA’s CSA?

TestOps is a software validation culture which aims at shorter testing and validation cycles to continuously qualify any cloud service, in close alignment with business and regulatory objectives.

Our customers are leveraging our robust framework that we call TestOps. TestOps provides an end to end automation for continuously qualifying any IaaS/PaaS/SaaS service or an Enterprise App.

This case study walks you through how TestOps is implemented to continuously qualify AWS DynamoDB Service while exceeding the FDA’s upcoming CSA guidelines.

Figure 1: FDA’s 4 Step Method for Software Assurance

Figure 1: FDA’s 4 Step Method for Software Assurance

TestOps Case study for Aws dynamodb

This case study layouts out the framework for continuously qualifying AWS DynamoDB. Keep in mind that this framework can be applied to any IaaS/PaaS/SaaS service or an Enterprise App.

STEP 1 - INTENDED USE definition

An IaaS or PaaS service can be configured to be used in many ways to meet varying end user needs. It is important to document the intended use to cover the various scenarios the service will be utilized in. This intended use forms the basis for designing a qualification strategy.

Figure 2: Intended Use for AWS DynamoDB

Step 2: RISK ASSESSMENT

In our experience very rarely a logical, useful risk assessment is performed, let alone applying it to testing strategies. The output of our risk assessment is applied to the testing strategies to determine: What features to test? What should be the extent of negative testing? What type of testing strategies to utilize?

We establish a risk-based approach to qualifying each requirement.   Our risk-based approach is achieved by assigning a Risk Priority to each requirement.  A typical Risk Assessment may be as follows:

High – A risk priority of High shall be assigned to a critical requirement which meets the following criteria:

- is not “out of the box” (OOTB) functionality AND
- is a legal/regulatory requirement.

All High priority requirements will be tested (both positive and negative testing).

Moderate – A risk priority of Medium shall be assigned to an important requirement which meets the following criteria:

- is achieved with “out of the box” features; AND
- is a legal/regulatory requirement;

All Moderate Priority requirements will be tested (positive testing) or verified (configuration verification).

Minimum – A risk priority of Low shall be assigned to a “nice to have” requirement which meets the following criteria:

- is achieved with “out of the box” software features.

Minimum Priority requirements will not be tested.

STEP 3 - DESIGN THE TEST CASES IN THE CODING FRAMEWORK

Conventional wisdom directs us to develop test cases outside of coding using Microsoft Word as the authoring tool. This approach is surely outdated and does not provide “hard” traceability to our code. xLM’s TestOps supports “Given-When-Then” (Gherkins) framework wherein the validation analyst builds test cases within the coding environment (IDE). These test cases are directly linked to the corresponding test automation code by our developers. In other words, both validation analysts and developers are working within the same environment and test case definitions are not separate from the code.

Figure 3: Test Case written using Given-When-Then Framework

STEP 4 - CODING THE TEST CASES

Once the test case design is complete, our developers code the testing scenarios which typically involves driving the browser for UI Testing or leveraging the APIs. The code that is developed is organized into reusable snippets (features, pages, utilities, etc.) for better maintainability of the test automation model. As mentioned above, the traceability between test case design and code is built into the framework.

Once the code development, code reviews, testing and quality checks are complete, the code is checked into the Code Factory.

Figure 4: Code Build and Publish

 

STEP 5: BUILD THE TESTOPS PIPELINE

Once the code is published, the TestOps Pipeline is configured:

  • to retrieve the published code

  • to deploy the qualified code into the test environment

  • to run the test automation code to qualify the SUT (System Under Test - in this case AWS DynamoDB)

  • to generate a test automation report aka “executed protocol”

  • to archive all the test run artifacts

Figure 5: TestOps Pipeline Stages

 

STEP 6: RUN THE TESTOPS PIPELINE

Once the Pipeline is fully configured and released by xLM’s Quality, the Pipeline can be run either manually or automatically (scheduled or event based). The Pipeline run-time steps include:

  • Obtaining the Pre-approval to run the “qualified” Pipeline

  • Running the test automation code and generating the execution report

Figure 6: TestOps Pipeline Pre-deployment Approval

 

Figure 7: TestOps Pipeline Run History

STEP 7: REVIEW THE RESULTS

After the Pipeline run is successful, the final step is to review the test automation results to ensure all the acceptance criteria have been met. This review is documented with the Post-deployment approval.

Figure 8: Executed Test Case (Sample)

Conclusion

xLM’s TestOps Framework is designed to continuously qualify any IaaS/PaaS/SaaS service cost efficiently utilizing test automation and pipelines. This framework utilizes seven (7) keys steps while exceeding the FDA CSA guidelines:

xLM-TestOps-versus-FDA-CSA.jpg