xLM TestOps Case Study
What is TestOps? How does it comply with FDA’s CSA?
Our customers are leveraging our robust framework that we call TestOps. TestOps provides an end to end automation for continuously qualifying any IaaS/PaaS/SaaS service or an Enterprise App.
This case study walks you through how TestOps is implemented to continuously qualify AWS DynamoDB Service while exceeding the FDA’s upcoming CSA guidelines.
TestOps Case study for Aws dynamodb
This case study layouts out the framework for continuously qualifying AWS DynamoDB. Keep in mind that this framework can be applied to any IaaS/PaaS/SaaS service or an Enterprise App.
STEP 1 - INTENDED USE definition
An IaaS or PaaS service can be configured to be used in many ways to meet varying end user needs. It is important to document the intended use to cover the various scenarios the service will be utilized in. This intended use forms the basis for designing a qualification strategy.
Step 2: RISK ASSESSMENT
In our experience very rarely a logical, useful risk assessment is performed, let alone applying it to testing strategies. The output of our risk assessment is applied to the testing strategies to determine: What features to test? What should be the extent of negative testing? What type of testing strategies to utilize?
We establish a risk-based approach to qualifying each requirement. Our risk-based approach is achieved by assigning a Risk Priority to each requirement. A typical Risk Assessment may be as follows:
High – A risk priority of High shall be assigned to a critical requirement which meets the following criteria:
- is not “out of the box” (OOTB) functionality AND
- is a legal/regulatory requirement.
All High priority requirements will be tested (both positive and negative testing).
Moderate – A risk priority of Medium shall be assigned to an important requirement which meets the following criteria:
- is achieved with “out of the box” features; AND
- is a legal/regulatory requirement;
All Moderate Priority requirements will be tested (positive testing) or verified (configuration verification).
Minimum – A risk priority of Low shall be assigned to a “nice to have” requirement which meets the following criteria:
- is achieved with “out of the box” software features.
Minimum Priority requirements will not be tested.
STEP 3 - DESIGN THE TEST CASES IN THE CODING FRAMEWORK
Conventional wisdom directs us to develop test cases outside of coding using Microsoft Word as the authoring tool. This approach is surely outdated and does not provide “hard” traceability to our code. xLM’s TestOps supports “Given-When-Then” (Gherkins) framework wherein the validation analyst builds test cases within the coding environment (IDE). These test cases are directly linked to the corresponding test automation code by our developers. In other words, both validation analysts and developers are working within the same environment and test case definitions are not separate from the code.
STEP 4 - CODING THE TEST CASES
Once the test case design is complete, our developers code the testing scenarios which typically involves driving the browser for UI Testing or leveraging the APIs. The code that is developed is organized into reusable snippets (features, pages, utilities, etc.) for better maintainability of the test automation model. As mentioned above, the traceability between test case design and code is built into the framework.
Once the code development, code reviews, testing and quality checks are complete, the code is checked into the Code Factory.
STEP 5: BUILD THE TESTOPS PIPELINE
Once the code is published, the TestOps Pipeline is configured:
to retrieve the published code
to deploy the qualified code into the test environment
to run the test automation code to qualify the SUT (System Under Test - in this case AWS DynamoDB)
to generate a test automation report aka “executed protocol”
to archive all the test run artifacts
STEP 6: RUN THE TESTOPS PIPELINE
Once the Pipeline is fully configured and released by xLM’s Quality, the Pipeline can be run either manually or automatically (scheduled or event based). The Pipeline run-time steps include:
Obtaining the Pre-approval to run the “qualified” Pipeline
Running the test automation code and generating the execution report
STEP 7: REVIEW THE RESULTS
After the Pipeline run is successful, the final step is to review the test automation results to ensure all the acceptance criteria have been met. This review is documented with the Post-deployment approval.
xLM’s TestOps Framework is designed to continuously qualify any IaaS/PaaS/SaaS service cost efficiently utilizing test automation and pipelines. This framework utilizes seven (7) keys steps while exceeding the FDA CSA guidelines: