Document Action Model (DAM) - An intelligent GxP document generation agent

Nagesh Nama

2024-04-17 14:30:00

welcome to continuous TV where we

transform Life Sciences with Dialogue on

AI and

GXP today's episode is focused on D

document action model in GXP your host

is Anand natarajan today's guests are

tan agrawal and param

nagda welcome everyone to continuous TV

this is our second episode in the series

our focus is mainly on AI and ml in the

realm of GXP

we know that gen AI Chad GPT llms lambs

have become so ubiquitous that they are

now household names and they' have

increasingly become the underpinnings of

how we as a society interact with

computers today we'll be talking about a

unique fascinating project undertaken by

exm that uses the power of gen and LMS

um and it's fairly Cutting Edge in its

nature my guests are tanay Agarwal and

param nagra who our senior AI ml lead

engineers at xlm or we like to call them

gurus in the space tanay Pam welcome to

this episode how are

you hello thank you so much for having

us over here really

excited yeah even I'm good uh great to

have meet you and have a conversation

about this topic so it it's really

exciting times now excellent so without

Much Ado let's get started um I wanted

to talk to you about the U the specific

use case you're working on where you're

using your AI and llms to help

streamline and maximize the efficiency

of certain processes around risk

assessment and documentation in the

world of QA so maybe T can you elaborate

on what the overarching goal of that

particular project

is definitely so the project we are

working on is what we call Jupiter so as

the name suggest it's an all

encompassing software solution that is

AI powered that means that not only it

goes beyond the realm of your rule-based

communication and FAQs but it has domain

specific knowledge regarding an entire

process of let's say manufacturing lines

or software developments anything use

case it has its knowledge based uh

trained and set so that it is able to

answer domain specific questions so that

we can can eliminate unnecessary

interactions between different Engineers

because when I would give a simple quote

of too many cook spil the dish and

that's what usually happens in

manufacturing because things need to be

so highly specific that a lot of

interaction tends to lead to misplacing

of some information down the line and it

also leads to wastage of time because

when you go about with documentation

processes especially the ones require

for FDA approval engineer from scratch

building those documents could take up

to four hours not just because of the

detailing required in the documents but

also a lot of information needs to be

gathered from different engineers and

different level of uh manufacturing line

workers Etc there's a different group

that you need to interact with to gather

all of those information so our use case

is to ensure that we have an AI

assistant that has all of those domain

specific Knowledge from the entire

Legacy of information that has been

gathered let's say there is a change in

a manufacturing line that happened 10

years ago and you can retrieve that

knowledge and see if that is relevant to

your current use case and at the same

time like I said it's a solution

platform so it is also able to guide you

through the entire process of creating

your very own draft essentially cutting

down the four hours of efforts that

you're going to put to Merry 10

minutes that is fascinating so this

seems quite I mean I think you're

blazing a new Trail here because I don't

think something like this has been built

and I think you're calling this the

document action model keeping the theme

with the llms and LS but how did you

start thinking about what are the

working components of a solution like

this things changed a lot in the past

two years with the revolution in

generative AI now it AI isn't just

something something that you look into

movies like Terminator it's reality now

and textual generation with logic is

possible Char GPT has taken the World by

a swop everyone uses it everyone knows

about it it's a buzz word now slowly and

gradually since last year a lot of

models are being open sourced and these

models aren the robust and huge like

chat GPT but these are most specific and

use case catered so that we can train

them to generate logical and reasonable

text based on previous data we are not

talking about just regular ml where it

looks into previous data finds something

that is relatable and gives you an

answer it can think that's the amazing

part you can train it like a child that

okay this is a process this is how you

have to answer it you train it on the

particular domain and it is able to

generate that part consistently at least

to a certain degree it's a whole

different topic of how consistent can be

maintained but a lot of these AI pieces

when you put together you have a human

kind of interaction that is able to

generate your entire process with utmost

accuracy that's how this entire solution

came into play that not only we want a

conversation that goes on we want a AI

that is able to talk to you like a let's

say a professional quality engineer that

okay based on my knowledge I have this

information regarding the potential risk

in a manufacturing line do you think

it's relevant it can talk to you and on

the go make a document for you that's

how our motivation came into play like

if we have this AI can we train it to

work like a really experienced

engineer got you now considering that

this was I mean I think you guys are

like Pioneers in this area of document

um um the what do you call the

documented the dam as we put it

right were you able to use any of the

existing AI ecosystems or leverage any

existing offthe shelf products to kind

of build upon or you had to blaze a new

Trail Al together Pam maybe I know you

had different T you had certain pieces

Pam you had certain pieces of the

solution that you were building so maybe

you can take turns describing what you

did there uh yes sure so uh like my

primary focus while building Jupiter is

to look after the AI or basically the

doent generation process so yeah as you

said that there are existing ecosystems

in AI or the goto way you can go for

building a chatbot uh but since this is

a very domain specific very regulated

industry and getting public access to

the data is not possible is nearly

impossible so we had our own challenges

in regard to having a good amount of

data to use the conventional methods and

since we are restricted over here so

rather than trying different models we

went with uh an unconventional approach

that is building a hybrid approach in

which we had the rule based as well as a

bit of the help of the llm for the

document generation process so uh right

now we are in the phase where we are

building a risk assessment document

which is the first one so instead of

using chat GPT or any other LM model to

train the documents and get an output we

are right now more focused on using that

rule based technique and through a

search engine we are able to find the

relevant systems uh for that risk

assessment of any any manufacturing line

so from that search engine we find out

the systems that are involved and after

that begins a series of question and

aners from the chatbot and the user so

it is mostly a conversational based

where we took help of AI like the gbt

and apart from that we are also using

the gpts and Lang chain function calling

uh feature as well even this helped so

there are various components we are

looking into apart from that we have

also built sort of a question answer bot

as well like it is integrated in a

chatboard where any user can ask

anything about the system and that user

will get an answer regarding the system

so all these uh various functionalities

that we have it's uh not easy to

integrate with the existing AI as they

are more developed for general purpose

cases that's why we are using not only

the help of the AI but also building

something of our our own to cater our

needs and Achieve that accuracy that we

need to successfully deploy it and and

make it as a product so yeah yeah t what

about in your case what are your

thoughts on that so not from the AI side

we had our technology now the question

comes into how can we go in into the

full stack so we looked into various

different kinds of stacks the most

popular one was M that is your

abbreviation for mongodb Express react

and nodejs an entire stack completely

based on JavaScript it's beautiful

everyone loves it but it didn't fit too

much to our use case because a lot of

operations we did was only possible in

Python function calling in a sense if I

were to give a very basic example if I

were to ask CH GPT today that okay CH

GPT can you give me an email to uh

provide me with an email uh to my

supervisor saying uh that uh the day has

been really good

the CH GPT gives you a text but with

function calling I can have that text

being placed to a Gmail draft so the

moment you put that it will go open

Gmail and it will put the draft into it

so with function calling I can access

different bits and pieces and for that

we required python so now we wanted to

make changes to this current stack of M

and that's how we went to the farm stack

that is fast API react and

nodejs uh sorry mongod DB nodejs we had

to remove it it didn't work for us I I

would really love that technology but it

didn't help in a use case because we

never didn't get that flexibility of

python in that stack python allowed us

to not only establish the CED operations

that is your create read update and

delete commands that is required for

server side asynchronously but at the

same time helped us to maintain data

integrity and let us uh let our code

jumble around different bits of code and

programs because the AI decides what to

pick and what to not it no longer it's

based on complete user the AI also has

the power to change

things got you so when you were

evaluating these two stacks the m and

the farm uh in your opinion each one has

its specific use case but in your

opinion which actually won the race I

mean of course you've used farm but in

general in terms of scalability and

other criteria you used what do you feel

between M and farm are actually better

for such

applications So based on my valuation if

you are having something that is AI

driven and you know that the AI is going

to take charge of our application I

personally prefer Farm because the

effort is significantly reduced

everything on the back end is done by

python and integration isn't difficult

of python to your frontend react while

in the M Stack the difficulty of having

function calling or different features

that Lang chain has started to offer or

llama index or these AI tools uh

Builders have started to offer is not

easily

implemented but the downside of farm

stack is that it's relatively new it's

just a few years old still in the nent

stages so the community is not too well

established but things are getting

better now when it comes to scalability

and deployment both the both of these

Stacks are very solid they are very good

so at the end of the day it comes into

use case specific if you want something

that is able to interact throughout the

application then go for the farm stack

I've tried it I've tested it and I love

it but if you want something that has

more of modular Parts where the user can

access it like menus and different

applications

and you know that that the user has to

choose all of these parts the user

decides how to interact with it then go

with the M Stack there's an entire

Community out there who support this

reasoning got you got you um I mean this

project is fairly novel in its nature

the way you're describing this entire

process so it sure would have come with

some challenges I'm sure what were some

of the typical challenges you fac in

your area Pam and in your area T maybe

you can start uh with what with some of

the challenges you faced and how did you

overcome that yeah uh so the main

challenge as I said before is the

getting enough amount of data to train

the llm model so the recent or the

successful llm models that you see in

the market that any person can use are

built on billions of parameters that is

built on vast networ with a lot amount

of data but the thing is that the data

that was used were mostly of General

purpose like having basic conversations

or mathematical or basic programming or

all all on the base level but when we

are dealing with a project or a problem

that is specific to an industry

especially the life science and the

Pharma industry which as we all know is

very regulated in terms of it

documentation which needs to be exact so

we don't have or like you don't get a

lot amount of data in that easily so

training uh with that amount of data to

a model the model tends to hallucinate

that is it tends to give answers that is

not accurate and since we don't have any

perfect evaluation Matrix right now

present to actually evaluate if how how

many outputs are accurate so instead of

training model we focused as I said

before in hybrid based approach that is

we wrote a script a rule based script

where depending on the uh information

given by the user and the answer

provided by by the chatboard the further

conversation will take place and after

the whole conversation has been done a

document will be generated uh in the

meanwhile we are with the given amount

of data we are trying to use smaller

models like models which have smaller

amount of parameters to see that if we

are able to get a good amount of

accuracy so that we can integrate but

otherwise our plan is to uh develop this

chatbot and then let different users use

it and through that we gather and data

and from a database of our own so over a

period of time we retrain it so that has

been one of the major challenges that we

have faced till now but I'm sure we can

overcome it uh in a due course nice what

about you T in your area

so there are two major issues like in

this entire project one is consistency

of answer when you go into approval of

these doc changes and processes you

can't have a model that gives you a

different answer every two to three

terms it needs to be consistent and AI

isn't the best when it comes to

consistency it's creative but it's not

consistent and that's the issue we had

because these documents are highly

specific very long pages of document and

if you were to just tell a model that

okay this is the data learn and give me

an answer it would fail utterly it would

be a complete disaster so then you have

to go about breaking each and every

solution and based on the type of

input you you evaluate that okay this

section requires this much of

information this kind of information if

I'm talking about risk so rather than

telling who did what I need to tell what

were the potential risk assessment that

were done in the previous history and

then have to go for a smaller model

train it so that it could give a more

consistent answer and that's why by

using a rule based approach we are able

to link all of these smaller models get

it modulated by a bigger model that is

GPT over here and have our entire

application control and the second part

is correlation it's like you have a

glass of water and you mix two color

like blue and yellow and it's all green

now so if I were to ask you like how

much portion was blue and how much

portion was yellow you can't answer it

and the similar issue is with

llm the process of validation is

consistent which is a very good thing

but at the same time it's night when you

go into the field of AI because a lot of

documentations tend to be similar and

because of this similarity your AI tends

to hallucinate more because it is unable

to differentiate between what

information is relevant and what is not

because all of it looks same to it so

that's why we had to curate Solutions on

our own that could tackle these use case

specifics like search algorithms for

different kinds of

documentations and we had to also go

with the evaluating rag models which is

basically providing a digital library of

all your information for that llm so

down the line based on the assessment of

all of these parts we had to curate our

own Solutions and integrate it in our

flow so

those so this is I mean it's a strange

situation where standard standardization

or you know a homogeneous set of data

actually causes problems for AI because

it's not able to pick up something

unique amongst these things yeah so that

is pretty uh yeah it's it's kind of

counterintuitive when you think about it

right when it comes to this yes now

talking about data and Life Sciences

since this project is for the life

sciences area how did you overcome or

were there issues with you know data

privacy data

Integrity um how did you overcome that

or is that even an issue in your

project so the beautiful thing about a

solution is that all of these Mo models

have been trained locally or of the

information doesn't go outside of our

ecosystem so irrespective of how the

model learns its uh weights that's what

we call its knowledge that it has

ingested in itsay will never be exposed

to an outside source and but if you're

using an outside like uh endpoint like

chart GPT then we make sure that it's

not interacting with our own data and at

the same time since we using an llm the

data nature changes from time to time a

number could be looked as as a string of

words a string could be looked at as a

number so data Integrity plays a very

crucial role and that's why the farm

stack the fast API really helped us out

because it has a it has been built on

pantic which is a python

framework specially made to ensure data

Integrity that your data doesn't changes

any form and that helps us in two

reasons first it makees sure that our

responses

are exactly how we want it to be and at

the same time it provides us prot from

injection based penetration from the

cyber security point of view so it's

like two wordss with one stone

nice nice okay and now that

this product product has been actually

built right how are you actually

evaluating its accuracy how are you

fine-tuning the model to make sure what

you're generating is closer to the

document that is expected by your

client it's actually more of a challenge

of how can we Define a matrix to

evaluate our progress because a lot of

these draft we have reviewed it and they

look really well but to measure their

consistency we are still looking into

how we can look into the we can evaluate

the difference between the final

document and the draft that our AI has

given evaluate that difference and based

on that Matrix we are able to validate

that very process and say that okay this

is the accuracy of a model and then

provide that change back to

our llm and have it learn from it so

that it keeps on continuously validating

that process so building we are still on

a nent stage of building this process

but this is our key goal in a project

that not only that AI is able to

generate it we will have another AI

block which is able to evaluate the

performance of our initial Ai and based

on all of this it is able to learn so

you can actually avoid having too much

of human intervention like a QA engineer

to drop by again and again just to check

the

output nice okay so we can expect the

accuracy to slowly and know I mean

gradually also get to the point where

there's very little human uh you know

intervention required to kind of correct

it at some point right yes definitely

like over a due course like after a few

iterations we are definitely hoping that

right now there are a few peop that are

involved two three or four people

involved in the that whole risk

assessment documentation process so in

the due course we are hoping to

eliminate all the middleman and there

should be just one one of one person

that is the quality engineer who gives

the final approval to just give in all

the information and the chatboard gives

you the document that you need in a few

minutes time maybe ask for a little more

information and depending on the

intelligence of the chatbot and you get

the document so in that way it will not

not only save a lot of time but also

save a person going back and forth to

different people communicating so it

becomes a lot easier so that is our

ultimate goal right now nice that was

kind of where I was going with this so

now that you've kind of built this

product where do you see the trajectory

of this going right in as a you know

your utopian goal or your final

solution uh so our ultimate goal is not

as I said not only to uh get give

service like that in which only one

person is required but also include

different processes like in this whole

process there's not only one risk

assessment document there are various

documents that are required so in the

due course we are uh slowly slowly

trying to integrate all those documents

so in the end we get a whole pipeline

where the user just gives the initial

input and with a little bit of

conversation with the chat bot the user

is able to get each and every document

uh with a uh without much uh what is a

without much hassle and easily can

communicate with anyone or even with the

chat bot and not only for document

generation we are even trying to focus

on getting code generation like there

are various Industries related to life

science who are making softwares for

different Healthcare Industries as well

so we are trying to uh develop some llm

models or some service in which a user

can give a description and they get a

code for that that description and also

not only that but there is is also the

packaging industry and different

industry as well where where you need an

OCR that is the text recognition or

seeing if the important cbls are mention

are present in the packaging so through

computer vision we are also trying to

develop s such systems or Services where

the user can give a PDF or a photo and

our system can detect if there's

anything missing or if there's any

unclear so right now we have a lot of

things going on and hopefully we can

unify everything to together into one

space so any person from the life

science Industry can come onto our

platform nice and I mean you were

talking about you know having a human in

the loop at this point to kind of look

at these documents and give them the

approval is the utopian goal also to

kind of slowly replace the human in the

loop with a model in the loop and then

say you know we are feeding this

document to another AI model that

actually looks at it and says yep it's

good to go yeah definitely definitely

maybe in some time we we'll be able to

build like to getting an answer from a

model we may also be able to build

another model to evaluate it and to be

so much perfect that no other human

intervention is required as soon as a

chatboard gives an output another model

evaluates it and if that model gives an

maybe an uh scoring rate or an approval

so the other human just has to pass it

on or give it for final submissions or

whatever other processes are required

but we are trying to minimalize the

human intervention as low as possible so

that is the ultimate goal nice now at

the beginning of the uh the session T

you had mentioned something about the

efficiencies gained because of this

process because there's a lot of

interaction there's a lot of misplaced

information there's a lot of human

interaction between several departments

to have this document realized what are

you seeing as the potential gain in

efficiency between how long it took in

the past versus what your model can now

do in a matter of maybe uh you know

minutes or hours where where do you see

that biggest gain what kind of factor

are we looking at this point so there

are two factors in it when we talk about

time we are looking from a time taking

just 1/8 or one/ tenth of what the

original process is going to take so if

I were to say f process takes four hours

then our system can take only 30 minutes

and that is also because it has guiding

you through the entire journey of

getting all of those components fixed so

that it can give you the draft document

and the second thing is that it has

knowledge the data a lot of our clients

have data that is massive in nature and

a human can never remember it and

usually a engineer wouldn't have been

with the organization for that long to

even recollect something that has

happened 10 years ago but having the

model trained on that those information

on that entire historical Legacy it is

able to look into risk assessment that a

person couldn't even think of it or it

might just slip their mind it can avoid

those

situations and that's what our so this

is not only is it making it efficient

but it's making it a lot more robust and

lot more reliable than the human mind

right yes because the human mind is

subject to all its memory losses it's

you know careless mistakes and this kind

of overcomes a lot of this so yeah this

is I would say this is a fairly

comprehensive solution that kind of

addresses efficiency and human foibles

in one go yeah that's true it's a

One-Stop Solution that's why we named IT

project Jupiter that it's all

encompassing it can have everything in

it and whenever someone thinks of

validation the thought that they should

have is okay let's go to

Jupiter got you nice this was a

certainly very an enlightening

experience so thank you for your time

Pam thank you for your time T until next

time with some more information that

you'll come back with we'll touch base

thank you thank you for having us

bye bye thank you for watching

continuous TV if you have any further

questions please shoot an email at info@

continuous validation.

About

In this episode Anand Natarajan discusses the implementation of DAM (Document Action Model) in Project Jupiter @ xLM Continuous Labs.

Project Jupiter: Goal: Develop a GenAI Framework that is agent based with a Chat Interface that can be used to automate generation of complex documents used in GxP workflows. ==**==

Use Case: An example use case is development of a Risk Assessment document for a medical device manufacturing. Any change in the manufacturing environment starts with a CR (Change Request) and a Risk Assessment (RA). Typically, such an RA is a complex document that includes assessing risks to product and process, evaluating the impact of change on existing risks/hazards as well as introduction of new risks/hazards, listing various related references (Product / Process Risk Analysis, etc..) and populating the required approvers based on the scope of change.

Such an RA induces migraines for the change owner and multiple trips to various document repositories and endless Teams chats with colleagues and QA folks.

Outcome: A Jupiter sized GenAI Agent that can handle the generation of a complex document like an RA through NLP (Natural Language Processing) chat interface.

Show less

Short Videos

Short Video

002 Revolution in Generative AI

Nagesh Nama

002 Revolution in Generative AI

Short Video

002: Textual Generation with Logic

Nagesh Nama

002: Textual Generation with Logic

Document Action Model (DAM) - An intelligent GxP document generation agent

Nagesh Nama

Transcript

Param Nagda

Nagesh Nama

About

Short Videos

002 Revolution in Generative AI

002: Textual Generation with Logic

Full Episode

Dig Deeper Into the xLM

CAM - Code Action Model - An agent based script automation and validation in CDMS

Document Action Model (DAM) - An intelligent GxP document generation agent

The Proof Is in Our Customers' Results

Success Stories

Learn How xLM Can Help Increase Velocity While Reducing Cost and Risk