CAM - Code Action Model - An agent based script automation and validation in CDMS

Nagesh Nama

2024-04-24 14:30:00

welcome to continuous TV where we

transform Life Sciences with Dialogue on

AI and

GXP today's episode is focused on cam

code action model in GXP your host is

nagesh Nama CEO at xlm the guest is Anan

natarajan director of solutions delivery

xlm hello everybody uh this is nagesh

Nama I am the host of continuous

TV uh today it is episode number three

uh episode one we talked about vam which

is validation action Model episode

number two we talked about Dam which is

document action model today my guest is

Anand nran and he is uh the director of

solutions delivery at xlm so we'll be

talking about

uh cam which is code action

model so let me give you a little bit of

background so Chad GPT or any llm can be

used to generate code automatically so

today we'll talk about a project at xlm

the name of the project is Maya Code

which involves not just generating uh

code automatically based on uh NLP or

natural language uh prompts but also

doing more than that so that's why we

have Anand where you know we can discuss

that and that will be the main topic for

today uh episode so Anand uh welcome

welcome to continuous TV uh can you

please uh uh tell the audience little

bit of a background you know obviously

we are into Maya Code you know we have

defined the problem we have found a

solution and our teams are working on it

and our customer also is relatively

happy at this point correct so now just

give us a little bit more of a

background around uh why did we start

why did we need Maya Cod why did how did

Maya Cod start thanks for having me

nages um yeah so I think what

essentially was the use case or at least

the industry need was as you pointed out

one of our large Cloud software

companies came to us and they recognized

the need to reduce the time that they

now spending on generating what they

call the rules and the scripts and

validating them for any kind of

clinical data management system they

were spending close to like three months

in the entire process for a typical

study that had about th000 rules and

about 33

forms um so we kind of took that on as a

challenge and we said how can we now use

our project Maya Code to deliver what

they were looking

for so essentially what uh the problem

statement was was you know

any clinical data management system it

has centerpiece is the the study that is

designed and the study is basically you

know it is a road map that handles all

the data under forceable circumstances

that are followed in a clinical data

trial and along with the study design is

what um they do is a definition of what

they call rules and rules play a crucial

role in basically collecting validating

and identifying discrepancies in trial

data so essentially you know there could

be things like you know when a patient

comes in or when a subject comes in you

need to make sure that the

electrocardiogram for that person is

done 15 minutes after his vital signs

are taken that's a kind of a stipulation

that's put in that needs to be codified

put into a programmatic syntax and that

needs to happen as the trial is

executed so as the complexities of these

rules come in uh the you know manual

processing of you know actually creating

the script having it embedded in the

IDE U validating all of this takes a lot

of time because the developers have to

parse all of these rules that are in

natural language and it's manually

intensive to then convert what they

written in uh natural language into

something that can be put put into the

system and you know executed as part of

the

trial so that was pretty much our

problem statement and we said this is

what we begin with and our aim was to

automate this entire process end to

end yeah just to summarize that uh for

the audience um I'm just going to give

it in a few sentences exactly what you

said so basic we have EDC which is

electronic data capture as part of cdms

right cdms is much larger than EDC so

EDC has uh crfs the case forms you know

which are based on a study protocol so

in order to conduct the study you need

complex forms that is designed to

conduct the study so this forms have

obviously fields and other data points

which has to be validated as this is in

datab being entered for example correct

and now for to you now to validate the

data you need to write

scripts and so that's what I think we

were talking about correct an correct so

once the study has been designed like

you said all the forms are you know laid

out and then you have all of these rules

that govern how these forms will be

presented what data points are going to

be collected uh what are the

stipulations of you know the data that

needs to be

collected what needs to happen is as

part of the validation one needs to ex

execute the entire flow of of the study

and you have to consider all the

possible permutations of the navigation

paths that can occur based on scenarios

right of the subjects and then through

that process you have to make sure that

every rule is triggered and tested right

at an correct at an event level maybe at

a form level at a field level so that is

quite an arduous task basically to make

sure you covered

everything so just to break it up you

know you have step one NLP or the

natural language specification has to be

converted into a script correct correct

assuming the stud is already set up the

study is set up in the system you know

and you need to now code meaning take

that uh NLP C into that's step one once

we have the script in the system now I

need to validate it meaning I need to

know in order to validate it it could be

in any of the screens in the workflow

depending on what tool we are testing so

we may have to populate certain screens

to get to the Target screen then and

Trigger the rule has some prerequisites

that we have to take care of trigger the

rule so it could be very cumbersome so

at xlm we always did continuous

validation which is the easy for us

because the last seven years we've been

doing continuous validation now what's

new is since last year is taking that

natural language converting that into

automatically into script like how a

human human being codes so we have a

coding agent you know that converts that

and the third piece is also what is new

or we figuring out using again um uh

lamb or large action model is using the

configuration in the that's the the

config dump from the system figuring out

how to navigate right automatically on

the fly so the three pieces if you can

put all these three pieces together

we'll have end to end workflow meaning

just take the NLP at the end we code it

we execute it we know how to navigate to

trigger that rule and then know we we uh

validate it validate the rule and give

you evidence correct am I stating it

correctly exactly so the entire the

entire channel right the Inception is

just a simple natural language prompt

correct that is taken digested converted

to what whatever is necessary and then

you have the output as the tested

evidence so Anand this is it JavaScript

or in this case is it like a platform

specific scripting language this is a

platform specific scripting language it

has it own

syntax and it relies upon the developer

being well vered with the whole

architecture right that's kind of

trained on the correct I have to be

trained on the platform right uh so to

convert that and also know how to

navigate in and Trigger it so let's back

off a little bit in order for the step

one in order for the platform or the

model to generate the script so it has

to know the syntax right and it has to

be trained

uh how does that work you know can you

just kind of walk me through the process

of training that model sure

so as you know with the Advent of llms

you know we have the ability to kind of

really take structured and unstructured

text and actually feed it to the llm and

then fine-tune the the learning aspect

of it so what in this case you know

essentially what a study comprises of is

you know we have um the knowledge base

is basic basically your study design you

have natural language descriptions you

have the manually coded programmatic

instru instructions that are already

there you have tests that have been

performed in um

retrospectively uh you have

documentation on the system architecture

the system variables the apis so we take

all of this we kind of curate it we

cleanse it we convert it into a

digestible form for the llm and then we

feed that to the llm and then we assess

and modulate and fine-tune the the

accuracy or the learning capability of

the llm and once that is ready that

becomes a trained model to now start

generating the programmatic code given a

natural language

description so they say right picture is

worth a th000 words uh is it possible

for you to share your screen and quickly

show how the model can generate uh the

script automatically sure let

me go in here and then let me pull this

up me share my screen

so what you'll be showing is uh the

first piece of it right like the mod is

already trained so when we give an NLP

prompt it automatically generates the

code correct correct let me know if you

can see the screen yep I can see the

screen now so this is basically a simple

um uh UI so we have a system promp that

maybe kind of cajo the system to kind of

um or give it some preface to kind of uh

expect to give it some expectation of

what we're going to provide it and then

what you see here is some of the

training that we have given it already

in terms of you know all of the config

specs Etc and then now I prompt it by

saying okay let's take an example of

this guy right here I'm going to take

this and it's a simple English language

description that basically

says it is a rule or a stipulation that

says give me the

script or the the actual rule in syntax

iCal format that basically ensures that

the screening of the

subject his ECG should be done within 15

minutes of the U vital signs the moment

I submit this it processes

it and then voila you have something

where it actually defines the path it

needs to take and then the actual code

making sure that the dates are not blank

and then the difference in those date

time windows are actually 15 minutes so

is the syntax correct an this is as

close as it gets now as we train this

even further uh it'll get extremely

accurate where there'll be no manual

intervention

required So based on uh your experience

so far uh what is the accuracy is it in

the 80s range or the ' 90s range you

know uh what do you think what did you

find out with our learning um algorithm

that we fed to this llm this was you

know close to a 90% when we tested it

with a small subset of data and we kind

of ran it through a couple of scenarios

some of the the rules that it generated

were spot on some of them there was some

gradiation and accuracy based on how

much data we fed it and what kind of

scenarios the llm had already seen so

there's a clear correlation between the

kind of data we

provide and what the llm outputs so

we're pretty confident that this will be

in the upwards of 90% accuracy

got it got it yeah you can stop sharing

the screen we can talk about the little

bit more detail as to how you went from

here uh uh and how do you plan to build

this out so that it can be an end to end

solution right the Maya Code solution

correct y um so one is you already

trained the thing so it may not be 100%

accurate

Let's uh we know that it's going to be

in the 90s range correct y so and we

have to push it every person to push it

up you know we have to either train more

or also give it feedback correct

there'll be like a loop so it's like the

learning you know you can constantly

give it feedback saying yep you gave me

this but it should have been this so it

can learn over time correct absolutely

so we we're going to do some of those

you know reinforced learning through

human feedback also once it reaches the

end of the pipeline we have the results

from the validation we'll take that

we'll feed that back as part of the

training to the llm and with that closed

feedback loop the llm is only going to

get uh more accurate as time goes on and

the llm will start looking at different

scenarios that we feed it and then it'll

you know kind of deepen and widen its

knowledge and that would surely get us

really accurate results and the need for

manual intervention will slowly you know

we en envisage that it'll kind of

totally be eradicated at some point and

then we can use the model in the loop

instead of a human in the loop yeah hold

on hold on we'll talk about the model in

the look you know you're jumping ahead

now uh so uh now um once so the code is

let's assume that it's in the 90s range

which we already have proved right as

part of this experiment or this project

you know uh Maya code is generating code

which is in the acceptable range right

now obviously we get try it and try to

make it better now the next step is to

validate that script so continuous

validation which you can describe that

is not we have done it at xlm that is

really not hard for us

uh to drive the browser for example you

know and and have a robot or a software

robot enter everything and then take

screenshots give you a nice report even

on our website you'll see a lot of demos

on that one so we can talk about that

that's the last step but to me uh from

what I understand the most difficult

part here is now to figuring out let's

say I want to test thousand scripts

right we can have to bundle that so that

you know one workflow may be able to

test uh 10 or 20 or 50 scripts and

trying to navigate the workflow so that

you know you can activate those screens

with all the prerequisites and do it so

that's little bit of a challenge right

correct so uh like you said I mean we're

going to be using the xm's core

automation framework for doing the

actual automation of the testing piece

but you bring up an interesting point

there because when you talk about

thousands of rules that are there as

part of the

study the the interesting part is how do

you kind of you run the entire Thousand

Rules in one shot do you run it one at a

time what kind of sequencing do you do

so this is where we've got a little

creative in terms of how we logically

group these

workflows and we also kind of providing

a platform where you can initiate the

testing and it kind of figures out based

on the rules or the collection of rules

which forms or which event groups it

needs to tackle and then kind of runs it

in small batches and make sure all of

the steps are you know tested and then

generates a nice PDF at the end of it so

the The Secret of that the sauce is uh

you know kind of really breaking down

that entire study into logical chunks of

workflows that can be consumed by this

uh testing

engine and also the platform gives you

um uh a configuration specification

right in Excel format where uh the

entire uh study configuration is

available but it has to be deciphered

right so that you know we can drive the

browser correct so information is there

uh so we need to kind figure out from

the rule kind of figure out okay where

does the rule fit into which workflow

right so then how many scripts fit into

the workflow that I want to trigger how

will I navigate so information is there

but tring to piece all this together it

is not a very simple task I'm assuming

right it is not because there's a lot of

information available but to kind of

really get the L&M to understand the

context and the sequence requires a

little bit of um you know pre-processing

and fine tuning so all of it is in the

data curation and the ingestion aspect

it yes um and also as part of this um

podcast we also have a newsletter and um

I think we will be sharing some

screenshots in the newsletter so it

becomes easier for the reader to

understand absolutely and we'll also

kind of break down at least a high level

pictorial representation of the

architecture what are the moving

components and how it is sequenced and

what the closed feedback loop cycle

would

be got it got it got it um

so um again I'm just going to restate so

as far as the project Maya Code goes

challenge one is to take NLP and convert

that over to a the scripting language of

the platform of the cdms platform then

figure out how I can the rules

automatically on the Fly you know

there's no human involved correct y now

you group those scripts into uh which

workflows each workflow may we may be

able to trigger multiple scripts uh

maybe one script maybe 10 scripts we'll

figure it out then automatically uh run

those workflows meaning a bot will drive

the workflow and that's the uh large

action model or the vamp validation

action model which was we talked about

it a couple of weeks ago y so um and

that vam or the validation action model

will eventually generate a PDF showing

that uh either the all the scripts

passed or maybe some of them failed you

have the result um so I can think of

many advantages here because you correct

me if I'm wrong one is if a human being

does everything manually correct it

takes a a long time to test each one of

those scripts or you know trigger one of

the any of these workflows and gather

the evidence and summarize it if there's

a problem to repeat it again you have to

repeat it meaning it takes time Y and

100% regression is typically out of the

out of the window because it takes way

too long y so in our case we're talking

about end to end maybe half an hour

maybe a couple of hours not more than

that correct absolutely absolutely the

yeah the the projected um savings in

terms of cost and efficiency is manyfold

so if you talk about a typical study

right U which has about 33 or 40 forms

with Thousand Rules the spending the

industry today spends about an average

of three man months to actually do the

entire end to end endend to end with

with Maya Code we project that this

would be a matter of you know maybe five

hours or less so that's an efficiency

gain of 100 times and if you think think

about the cost savings it's at a minimum

$100,000 yes uh but obviously there'll

be an Roi so Roi should be maybe a

couple of studies they can recover right

once you have the Maya Code framework

built in so one study maybe two studies

they can recover the cost any study

after that is just free meaning it's

paid for already it's paid for already

yep so Roi will be maybe two studies

maximum correct

yep that is it's unbelievable I mean I

mean that's very very efficient and also

you have other benefits also correct

because we can repeat test it very fast

now meaning 100% regression that's

possible and you're compressing the time

which is very crucial uh when you're

launching studies right so even if you

can shave off a few days they're very

happy so with this I think we can really

compress it so there it's just not one

thing cost in this one but it's there

are many other advantages also correct

absolutely and you alluded to it a

little bit earlier also nages where you

said if if there are changes to the

system or if you know there are upgrades

you know many people B at this whole

regression testing to run it end to end

with this this is a walk in the park at

this point right yep uh and the model

can only get better over time because we

have the feedback loop and it can really

get better and the technology llm

technology also might get better so we

are here combining many aspects right

one is llm large language models and we

also combining large action models right

uh and and you know then bring all this

with test automation as the backbone uh

so it's really test automation on hyper

on steroids really correct absolutely

yeah uh yeah so this is really really

fabulous and kind of ties into the

validation action model we spoke about

and also uh last week we spoke about

document action model so this whole llm

business is changing the way we build

software and where we are coming up with

Solutions uh which was almost

Unthinkable before chat GPT

agreed and uh and also this whole field

of uh L it's even more nent even when

compared to LM llm itself if you look at

it it's November of 2022 but large

action model is really this year almost

you know it's not that old because uh

rabbit R1 was launched and they kind of

coined the word L large action model

that was January of this year so that's

also pretty new and if you can combine

this with computer vision in some cases

human many computer vision so this

becomes a very very powerful platform uh

I'm also excited you know um all the

best with your project you know uh and

uh hopefully I will pull you in down the

road so you can share um more insights

as as this uh cam is going to support

I'm assuming multiple studies so uh this

is really new technology we we learn

About

In this episode the implementation of CAM (Code Action Model) in Project MayaCode @ xLM Continuous Labs.

Project MayaCode:

Goal: This project aims to develop a GenAI Framework. The purpose of this framework is to automate code generation and comprehensive validation. The focus is on code related to rules governing data acquisition in Electronic Data Capture (EDC) within Clinical Data Management (CDMS).

Context: Clinical Data Management (CDMS) is the process of collecting, cleaning, and managing clinical trial data in compliance with regulatory standards. The primary objective of CDMS processes is to ensure data accuracy, adherence to protocols, and suitability for statistical analysis. Clinical Data Management Systems (CDMS) are used to capture data for clinical trials.

A study translates protocol specific activities into generated data. A study is essentially a road map to handle the data under foreseeable circumstances and describes the CDM activities to be followed in the trial.

Concomitant to study design is the definition of rules. Rules play a crucial role in collecting, validating, and identifying discrepancies in trial data. For instance, if the screening criterion for a subject (rule) stipulates that the ECG (Electrocardiogram) readings must be taken within 15 minutes of the collection of vital signs, code will be written for checking those conditions. If for any subject, the condition is not met, a discrepancy will be generated.

Rules can also dictate contextual data capture based on subject characteristics (e.g., gender, age, ethnicity). Additionally, rules guide users to relevant forms during clinical trial execution. In effect, rules drive the mechanics of data collection during clinical trials.

Use Case: During the design of a study, rules are defined in natural language. System developers then parse these rules and convert them into programmatic instructions with specific syntax. This process is manually intensive, requiring developers to be well-versed in study design, navigation paths, and programming rules. After converting rules to programmatic instructions, the crucial step is validating their correctness.

Show less

Short Videos

Short Video

003 The Three Key Pieces of the Workflow Automation Process

Nagesh Nama

003 The Three Key Pieces of the Workflow Automation Process

Short Video

003 Projected Cost and Efficiency Savings with Maya Code

Nagesh Nama

003 Projected Cost and Efficiency Savings with Maya Code

CAM - Code Action Model - An agent based script automation and validation in CDMS

Nagesh Nama

Transcript

Anand Natarajan

Nagesh Nama

About

Short Videos

003 The Three Key Pieces of the Workflow Automation Process

003 Projected Cost and Efficiency Savings with Maya Code

Full Episode

Dig Deeper Into the xLM

CAM - Code Action Model - An agent based script automation and validation in CDMS

Document Action Model (DAM) - An intelligent GxP document generation agent

The Proof Is in Our Customers' Results

Success Stories

Learn How xLM Can Help Increase Velocity While Reducing Cost and Risk