Monitoring clinical AI after it goes live

What changes after go-live

Validation before go-live measures a model against one population, on one set of devices, at one time. Deployment changes all three and adds the people who use it. A model’s performance at a site is therefore a quantity that moves, not a property it was certified with, and it has to be measured where the model runs.

The changes come from six places. Patients and case mix move with seasons, services and referral pathways. Scanners, software versions and laboratory assays change what the inputs look like. Interfaces get remapped and fields arrive empty. Clinicians lean on the model, or learn to ignore it. Practice changes what an outcome means. And the model itself changes, when its manufacturer ships a new version or the site moves a threshold.

For imaging, the site-caused version of this problem, a new scanner, a contrast protocol or a reconstruction update, is set out in the on-premises and hybrid clinical AI framework. This piece covers clinical AI generally, of which imaging is one case.

The clearest published case of a population change moving a deployed model is the COVID-19 surge. Across 24 US hospitals in four health systems, alerts from one widely used sepsis model rose 43% a day in the three weeks after each system’s first COVID-19 case, while the number of patients in hospital fell by a third. The authors did not measure accuracy, so the study does not show that the model got worse. It shows that the model’s output moved with the population, and that the cheapest signal a site has would have seen it.

A rising alert rate does not by itself mean a worse model. If the condition becomes more common, a model with unchanged sensitivity and specificity alerts more often, and more of its alerts are right. Prevalence is the first thing to rule out before reading anything into an output rate.

Who is watching

The manufacturer and the deploying organisation watch for different things. The manufacturer surveils its installed base under rules that are specific and mostly binding. Only the deployer sees its own patients, devices and workflows, and its duty to measure whether the model still works is thinner and less specific. Where a deployer duty does exist, it is narrow.

In the United States the detailed instruments face the manufacturer. FDA’s guidance on predetermined change control plans, reissued in August 2025, expects a plan to say how users are told of an update and anticipates the manufacturer monitoring performance over time. FDA’s January 2025 draft guidance on AI-enabled device software describes a post-market performance monitoring plan in more detail than any other regulator, and it is still a draft. FDA’s 2025 request for comment on measuring real-world performance closed in December 2025 and is not guidance. How the US approval paths work is covered in Clinical AI regulation: intended use determines the path.

Three US instruments reach the organisation that runs the model, and none of them asks it to measure performance. Since May 2025 the Section 1557 rule, 45 CFR 92.210, has given covered entities an ongoing duty to make reasonable efforts to identify decision support tools that use race, colour, national origin, sex, age or disability as inputs, and to mitigate the risk of discrimination. ONC’s HTI-1 criteria require certified EHRs to support a source attribute for a predictive tool’s “validity of intervention in local data” and to let the hospital record it, without requiring anyone to fill it in; HTI-5 has proposed removing those requirements. And the Joint Commission and the Coalition for Health AI published voluntary guidance in September 2025 that names local validation and monitoring.

Canada’s instrument is guidance addressed to manufacturers. Health Canada’s pre-market guidance for machine learning-enabled medical devices, published in April 2026, says manufacturers should consider describing monitoring plans for degradation over time, input shift and changed inputs, and should consider telling users how to perform local validation and ongoing monitoring. What binds is the Minister’s power to put terms and conditions on a device licence.

Ontario’s Enhancing Digital Security and Trust Act, 2024 lets the province require prescribed public sector entities to manage the risks of AI and document its performance and monitoring, but no regulation has been made under its AI provisions. Hospitals are prescribed under its cyber security regulation, in force since July 2026, which says nothing about AI. The nearest Ontario document addressed to the deploying organisation is the Information and Privacy Commissioner’s January 2026 guidance on AI scribes, which asks custodians to monitor on a schedule and when thresholds of unexpected behaviour are crossed, and to set thresholds for switching a system off.

The EU is the one place a statute says deployers shall monitor. Article 26 of the AI Act has deployers of high-risk systems monitor operation against the instructions for use, inform the provider, suspend use where they see a risk, and keep the system’s logs for at least six months. For medical devices it applies from 2 August 2028. In the UK the manufacturer carries post-market surveillance, while imaging providers have since October 2024 had to keep an inventory of interpretation software recording the version in clinical use. The Royal College of Radiologists’ March 2026 guidance puts post-deployment monitoring on the deploying institution.

So the local question, whether a model still works for this site’s patients, is often nobody’s job. The figures on practice fit that. In 2023, 65% of US hospitals used predictive models; of those, 61% had evaluated them for accuracy with their own data and 44% for bias. Those are evaluations, which may have been done once, not ongoing monitoring.

The manufacturer’s side of the evidence, including what a change control plan asks the platform to produce, is in The data platform is the evidence package. This piece stays on the deployer’s side.

The signals, from cheapest to most expensive

Signals come in order of cost. Input and output distributions are available as cases run and say that something changed. Process signals say how clinicians are responding. Performance needs a confirmed outcome, which arrives late, partly, and skewed towards the cases the model flagged.

The order matters because the cheap signals cannot answer the question that matters most. An input shift can be harmless, and a harmful change can leave the inputs looking the same. A 2024 scoping review of methods for monitoring clinical AI found 39 sources, of which nine reported any clinical implementation, and noted that the link between label-free signals and real performance is hard to predict.

Drift tests carry a trap at volume. The Kolmogorov–Smirnov test’s threshold shrinks with the square root of the sample, so at a few thousand cases a window it flags shifts far too small to act on. The population stability index has the opposite problem: its familiar cut-offs of 0.1 and 0.25 come from credit scoring and carry no error rate. Neither number is a trigger on its own. Decide in advance what size of shift matters, then test for that.

The label problem

Performance needs ground truth, and ground truth has three problems. It is late, because some outcomes cannot exist until a window closes: a 30-day readmission cannot be known for 30 days. It is partial, because many cases never reach a confirmed outcome. And it is biased, because the cases that do reach one are mostly the cases the model flagged.

Labels are also constructed. The external validation of one widely used sepsis model, published in JAMA Internal Medicine in 2021 and carried out at a single academic medical centre, defined sepsis as meeting CDC surveillance criteria, or as an ICD-10 sepsis diagnosis with inflammatory and organ-dysfunction criteria within six hours of each other. Its results belong to that definition, and a site comparing its own monitoring with them has to use the same one. The study’s results are drawn in the regulation piece.

The bias has a name and a 1983 paper. Begg and Greenes showed that sensitivity and specificity computed only from verified cases are biased whenever the decision to verify depends on the test result. A model that flags a case sends it towards confirmation; a case it misses often goes home. A site measuring accuracy on its confirmed cases sees a sensitivity higher than the truth.

The Royal College of Radiologists’ guidance gives the practical compromise. To detect change over time, full adjudication may not be needed if the same method is applied consistently. To state sensitivity and specificity, cases need manual review against a reference.

How long it takes to see a change

Seeing a real drop takes volume and time, and the arithmetic is simple enough to run with your own numbers. Sensitivity is read only on cases that turn out positive, and only on those with a confirmed outcome, so prevalence and label coverage divide the yield before volume does anything. For a low-volume, low-prevalence model, a site’s own data may not show a meaningful fall for a long time, and the site has to decide in advance what it will rely on instead.

N = ( z(0.975)·√(p0(1−p0))
    + z(0.80)·√(p1(1−p1)) )²
    / (p0 − p1)²

cases = N / (share × confirmed)
months = cases / volume + label wait

The calculator differs from the detection horizon in Imaging AI: six handoffs between the scanner and the chart, which counts studies against a fixed concordance rate. Here prevalence, label coverage and label wait decide how many of the cases a model runs can count at all, and the comparison can be against a validated figure or against the site’s own earlier period, which needs the larger two-sample count for each period.

When the answer is too long, the cheaper signals carry the load: an alert rate, an override rate, disagreement with the reader, the version recorded on every output. They do not prove the model is worse. They say when to look harder, which is what a monitoring plan needs from them.

Watching a running total: control charts

A control chart watches a running total rather than a monthly snapshot. The cumulative sum, or CUSUM, introduced by Page in 1954, accumulates small excesses and signals when the total crosses a threshold. The threshold trades how quickly a real change is caught against how often a stable process raises a false alarm.

Risk adjustment changes the question the chart asks. Steiner and colleagues’ risk-adjusted CUSUM, published in 2000 to monitor surgical outcomes, weights each case by its own expected risk. Applied to a model’s errors, it asks whether the model is doing worse than expected for these patients, rather than whether errors are rising. The two answers differ when the case mix changes.

W = log( RA / (1 − p + RA·p) )   on an error
W = log(  1 / (1 − p + RA·p) )   otherwise
S = max(0, S + W)   signal when S > h

RA = 2: tests a doubling of the odds
p  = p̄ for every case (unadjusted)
p  = the case’s own risk (adjusted)

Two cautions come with it. A risk-adjusted chart on a model’s errors needs a second model, of each case’s expected error, which has to be built and validated too, and the method’s track record is in surgical outcomes rather than AI. And adjustment answers a narrower question than safety: if harder cases arrive and the model errs on them more often, patients receive more errors while an adjusted chart stays flat. A site needs both views.

What the platform has to record

Every signal above depends on the platform recording five things for every output and joining them: the model and its version, the input as the model saw it, the output and what the clinician was shown, what the clinician did, and the confirmed outcome. If one is missing, at least one signal cannot be computed after the fact.

Each record maps onto the platform components set out in What a healthcare data platform is made of. The version belongs in lineage and audit. The input snapshot is a data contract the site writes for itself, because no standard defines it. The outcome usually arrives through a different feed, days or weeks later, and the join that reconnects it to the output is the platform’s work.

Standards carry most of the records. In imaging, DICOM’s template 4019 makes the algorithm’s name and version mandatory content wherever it is used. In FHIR, an Observation can name the Device that produced it, and the Device can carry its version. CDS Hooks defines a feedback call that tells the service whether a clinician accepted or overrode a card. Nothing defines the input snapshot or the join.

{
  "resourceType": "Device",
  "id": "sepsis-model",
  "deviceName": [{ "name": "Example sepsis risk model",
                   "type": "user-friendly-name" }],
  "manufacturer": "Example Manufacturer",
  "version": [{ "type": { "text": "model version" },
                "value": "2.1.4" }]
}

DICOM SR, TID 4019 Algorithm Identification
  (111001, DCM, "Algorithm Name")         TEXT  "Example nodule detector"   M
  (111003, DCM, "Algorithm Version")      TEXT  "3.1.2"                      M
  (122405, DCM, "Algorithm Manufacturer") TEXT  "Example Manufacturer"       U

{
  "inference": "inf-2026-10-16-000123",
  "model": "Device/sepsis-model|2.1.4",
  "scoredAt": "2026-10-16T14:22:00Z",
  "inputs": [
    { "loinc": "2524-7", "value": 3.1,  "unit": "mmol/L",
      "source": "Observation/lab-889/_history/2" },
    { "loinc": "8480-6", "value": 92,   "unit": "mm[Hg]",
      "source": "Observation/vs-4410/_history/1" },
    { "loinc": "6690-2", "value": 14.2, "unit": "10*3/uL",
      "source": "Observation/lab-890/_history/1" },
    { "loinc": "8310-5", "value": 38.6, "unit": "Cel",
      "source": "Observation/vs-4411/_history/1" }
  ],
  "mappingVersion": "lab-feed-map 2026.09"
}

{
  "resourceType": "Observation",
  "status": "final",
  "code": { "coding": [{
    "system": "http://example.org/fhir/CodeSystem/local-models",
    "code": "sepsis-risk",
    "display": "Sepsis risk score (local code)" }] },
  "subject": { "reference": "Patient/example" },
  "effectiveDateTime": "2026-10-16T14:22:00Z",
  "valueQuantity": { "value": 0.87, "unit": "probability",
                     "system": "http://unitsofmeasure.org",
                     "code": "1" },
  "interpretation": [{ "text": "Alert shown: high risk, threshold 0.60" }],
  "device": { "reference": "Device/sepsis-model" }
}

POST {baseUrl}/cds-services/sepsis-risk/feedback

{
  "feedback": [{
    "card": "0b7f3c1e-5d4a-4c3e-9a51-2f6d8e1c4a90",
    "outcome": "overridden",
    "overrideReason": {
      "reason": { "system": "http://example.org/override-reasons",
                  "code": "already-treating",
                  "display": "Already being treated" },
      "userComment": "Antibiotics started 13:50" },
    "outcomeTimestamp": "2026-10-16T14:31:00Z"
  }]
}

{
  "resourceType": "Condition",
  "code": { "coding": [{
    "system": "http://hl7.org/fhir/sid/icd-10-cm",
    "code": "R65.21",
    "display": "Severe sepsis with septic shock" }] },
  "subject": { "reference": "Patient/example" },
  "onsetDateTime": "2026-10-16T19:05:00Z",
  "recordedDate": "2026-10-21T09:12:00Z"
}

-- The join. The window is the site's own outcome definition.
SELECT i.inference, i.model_version, i.score, f.outcome,
       c.code AS confirmed
FROM   inference i
LEFT JOIN feedback  f ON f.inference = i.inference
LEFT JOIN condition c ON c.patient = i.patient
       AND c.onset BETWEEN i.scored_at
                       AND i.scored_at + :outcome_window;

The smallest useful version of the first record is a result that names the model and version that produced it:

{
  "resourceType": "Observation",
  "status": "final",
  "code": { "text": "Sepsis risk score (local code)" },
  "subject": { "reference": "Patient/example" },
  "effectiveDateTime": "2026-10-16T14:22:00Z",
  "valueQuantity": { "value": 0.87, "system": "http://unitsofmeasure.org", "code": "1" },
  "device": { "reference": "Device/sepsis-model" }
}

{
  "resourceType": "Device",
  "id": "sepsis-model",
  "deviceName": [{ "name": "Example sepsis risk model", "type": "user-friendly-name" }],
  "version": [{ "type": { "text": "model version" }, "value": "2.1.4" }]
}

In the UK, part of this record is already law for imaging: the IR(ME)R inventory must carry each interpretation tool’s current version and the year that version went into clinical use.

When the model changes, and who decides

A deployed model changes in six ways: an update inside an authorised change control plan, an update that needed a new authorisation, a change behind a hosted service that is not a regulated device, a local threshold change, a change upstream at the site such as a new scanner or interface mapping, and suspension or retirement. They differ in who starts them and in whether anyone is told.

For each, the site needs answers settled before go-live: who reviews the signals and how often, what crossing triggers a closer look, who owns the dashboard, and who can switch the model off. The RCR’s guidance says audit frequency, sample size and escalation thresholds should be defined locally, and Ontario’s privacy commissioner asks for thresholds and a process for turning a system off. How to run a new version in shadow before it goes live is set out in the hybrid clinical AI framework.

Much of this depends on what the vendor provides, which is best settled before the contract is signed. The questions below are written to be tested: each says what a checkable answer contains. A vendor’s answer is the vendor’s claim until it has been checked.

Where the standards stop

How a result travels is standardised. DICOM structured reports, HL7 v2 and FHIR carry it, and the model’s identity can travel with it. How a reader’s verdict is sent back has trial-implementation profiles in IHE AIRA and CDS Hooks feedback. How a study of the model is reported has reporting guidelines, TRIPOD+AI among them. The manufacturer’s surveillance is governed by binding regulation.

What a deploying organisation should monitor, at what threshold, and who decides to switch a model off has guidance and nothing more. The Royal College of Radiologists, Ontario’s privacy commissioner, and the Joint Commission with CHAI have each published some; CHAI is a consortium and its guides are not standards. No standard from an accredited standards body was found that covers it.

That gap is where a deploying organisation’s own decisions go: which signals it watches, what it records, what crossing triggers a review, and who decides. None of it can be borrowed from a standard yet, and all of it depends on the platform recording the five things above.