Clinical Evidence Generation Framework to create ML Models

As we generate models and roll them out into production at our partner institutions, we’ve had to develop a template - a clinical evidence generation framework and methodology to make sure that building models is a repeatable and predictable process. It’s been an amazing learning experience and one that we’d like to share.

The first big concept is that predictive models don’t improve outcomes, analytically driven interventions do! And they always have.  

To improve outcome {X}
We need intervention {Y},
And intervention {Y} may need Model {Z}

{X}

Let’s go through each of these variables one-by-one.  First {X} or Problem Formulation:

• What is the problem that needs to be solved?
• What are the objective measures or KPIs associated with the problem?

To figure this out, we need to focus on selecting highest priority metrics and indicators to be measured. Things like: impact, success, risk or harm. And each of these needs to be able to be monitored for continuous improvement and learning over   time. Here are four steps that will help you with Problem Formulation:

1. Describe the problem in terms of one or more of the following: measurable outcomes, quality or operational metrics, patient and/or clinician experiences, or efficiency of operational processes
2. Identify any new value-generating opportunities like saving money, or generating more revenue, or offering a new business service, or simply  learning.
3. Identify regulatory and compliance standards as well as community/professional best practices. This may include reporting obligations that will be impacted and need to be addressed.
4. Identify collaboration opportunities such as data sharing, networking, research/grants and of course publication opportunities.

These four Problem Formulation steps may include any of the following components:

• Clinical Outcomes  such as mortality due to sepsis or a pre operative medical optimization
• Operational Metrics such as Length of Stay
• Patient Experience such as post operative complications or is there a risk of malpractice!
• Provider Experience: reducing burnout is always good
• Your financial and operational team will love you if you can show a Return on Investment. This may include PACU utilization, OR sequencing, OR some other optimization
• Adjacent to ROI are simple Savings and efficiencies. We built a model that predicts the risk of a patient canceling on the same day as their appointment that saved $500,000 in the first six months of use.
• You may even be able to generate new capabilities or service lines as a result of your model. 

{Y}

That is {X} - Problem Formulation. Once we’ve figured that out, we can analyze workflows and think through {Y}: Solution Formulation. This includes things like:

• Process change and interventions
• Communication and information sharing (decision support)
• Delivery, monitoring, and continued improvement 

In order to conceptualize a solution and proposed interventions or workflow changes we need to describe the change in behavior, workflows, or processes as much as possible. And we need to ensure logical and operational linkage between all proposed interventions with at least 1 of the outcomes/metrics that are being measured in {X} our Problem Formulation - and vice versa. 

Here are four categories that the solution formulation may include:

• Interdisciplinary interventions: Deploying team science and implementation science approaches to implement new workflows, processes, care paths, and/or protocols
• Decision Support tools: This must include the information, content, communications, evidence, training, and data that is needed to support the interventions. You can use the Five Rights of Clinical Decision Support to describe information needs: (1) Right information, (2) Right time, (3) Right location, (4) Right person, (5) Right format
• Consistent Delivery: How to ensure that the interventions are uniformly and consistently delivered enterprise-wide
• Prevention of Harm: This may be the most important. There are many ways that a model could introduce additional risk or harm. This includes implicit bias, discrimination, exclusion or simple clinical error.

{Z}

Once we have both our Problem {X} and Solution {Y} Formulated, then and only then can we allow ourselves to ask what kind of machine learning algorithmic model might be helpful - that is {Z}: Model Formulation. The model could fall into any of these categories: 

• Prediction problems
• Classification problems
• Timeliness problems
• Integration to workflow

1. Do we need to predict future events or outcomes? For example:
   1. Will the patient tolerate surgery?
   2. Will the patient develop organ damage during LOS? Or complications post discharge?
   3. Will the patient require invasive mechanical ventilation?
2. Do we need to classify patients or events to classes such as high/low risk?
   1. Describe different classes and explain how they are being ascertained now?
   2. How can a model improve the current state?
3. Do we need to provide a personalized or context appropriate recommendation?
   1. What are the known options and modalities for a recommendation?
   2. Use the Five Rights of Clinical Decision Support to suggest how the recommendations should be communicated at the point of care.
4. Does the prediction need to be made at a particular time? Such as:
   1. 7 days before surgical appointment,
   2. 8 hours in advance of an event
   3. at the time of discharge
   4. every day at 8am, etc.  

With this framework we have been able to figure out which models to build and with whom to build them. Of course once this is done, you still need to look at model validation, trustworthiness, the communication plan and regulatory compliance. In an effort to keep things short we will cover these topics in Part 2 of this blog. 

Meanwhile, if you’d like to learn more about Cognome’s currently available models and the TUNER BI platform visit our models page https://cognome.com/ai/ml-models.