The data platform is the evidence package

The number that should worry a platform team

A 2025 cross-sectional study in JAMA Health Forum looked at every AI and machine-learning device the FDA cleared between 1995 and July 2023. There were 691 of them. In 660 cases, 95.5%, the decision summary said nothing about the demographic composition of the training population. In 385 cases, 53.3%, it did not report the training sample size at all. Under 2% reported data from a randomised trial.

Read those as reporting failures and the conclusion is that manufacturers were careless with paperwork. Read them as platform failures and something more useful comes out: in most of those submissions, nobody could produce the answer. The training set had been assembled, transformed, joined and sampled by pipelines that kept no attribute-level record of what went into it, and by the time a regulatory writer needed the demographic breakdown, silence was the only honest thing left to file.

The same study found that 40 of those 691 devices, 5.8%, were later recalled across 113 separate recall events, mostly for software and algorithm faults. The reporting gap and the failure rate are not two findings. They are one gap, seen from before and after market.

What follows is the architecture that closes it, in the order a platform has to solve it: what the data is and where it came from, what may be learned from it, what evidence goes into the submission, how the model is served, and what happens when the field disagrees with the lab.

Lineage and provenance answer different questions

The two words get used interchangeably, and they are not the same record.

Lineage is technical cartography. It is the directed acyclic graph of how data moved: which pipeline read which table, what the transformation did to the column, where the join dropped rows, which snapshot the feature store was built from. Lineage is what turns a root-cause investigation from a week into an afternoon, because it lets an engineer walk backwards from a wrong model output to the upstream table that changed.

Provenance is a legal record. It captures where the data originated, who or what produced it, when, under which consent or data use agreement, and for what permitted purposes. Lineage tells you the transformation was a left join. Provenance tells you whether the rows on the right of it were ever yours to train on.

An audit wants both, and it wants them at attribute level. Banking arrived there first: BCBS 239 has required attribute-level lineage from source to reported figure since 2013, and examiners there stopped accepting "the data moves from system A to system B" a long time ago. Health regulators ask the same question in different words. The FDA wants proof that tuning and test populations were held separate from development data. Article 10 of the EU AI Act wants training, validation and testing sets that are relevant, representative and examined for bias. Neither is satisfiable by a diagram drawn afterwards.

Which is the practical argument for automated capture over documentation. OpenLineage instruments the orchestration layer, so Airflow, Spark and dbt emit start and complete events carrying input datasets, output datasets and run metadata as a side effect of running at all. Apache Atlas does the equivalent across Hadoop-lineage estates through execution hooks that write metadata asynchronously, off the critical path. In both cases the record is a by-product of execution rather than a document somebody remembered to update.

Tamper-evident, which is not tamper-proof

A provenance record is worth what its resistance to quiet rewriting is worth. The usual construction is a hash chain: log entries are batched, each batch carries a SHA-256 hash of the batch before it, and the batches land in storage under a write-once retention lock so no single credential can delete them.

Be precise about what that buys, because the imprecise version causes a real failure. A hash chain does not prevent alteration. It makes alteration detectable, because changing a historical entry invalidates every hash after it and the break shows up on verification. That is a sufficient property for an audit. Calling the log immutable is what invites a team to skip the scheduled verification job, and the verification job is the only thing that converts the property into evidence.

The privacy budget is platform state, not a project setting

De-identification is not a defence against a determined linkage attack, and it is no defence at all against model inversion, where the trained weights are interrogated for the records that produced them. Differential privacy is the response, and it is a definition rather than a product.

The definition bounds how much any one person can change what comes out. A randomised algorithm A satisfies epsilon-differential privacy if, for any two datasets differing in a single record and any set S of outputs, P[A(D1) in S] <= exp(epsilon) * P[A(D2) in S]. Epsilon is the privacy loss. Lower means less can be inferred about any individual, and more noise has to be injected to guarantee it.

The architectural point is what happens next. Every query answered from a protected dataset spends part of that budget, and the composition theorem says the spend accumulates. A data scientist running forty exploratory aggregates against a feature store has not run forty independent private queries. They have spent forty times something, and if nobody is counting, the effective epsilon at the end of the quarter bears no relation to the one written in the policy. So the budget belongs in the platform: a central accountant tracking consumption per dataset and per principal, refusing the query that would overspend. A budget each project sets for itself is not a budget.

Mechanism follows data type. Laplace noise for numeric queries, scaled by the query sensitivity over epsilon. Gaussian noise for the (epsilon, delta) relaxation, where delta is a small probability that the guarantee does not hold at all. The exponential mechanism for categorical and text outputs, selecting probabilistically from a scoring function.

One thing to resist. There is no general curve from epsilon to model accuracy. The relationship depends on the dataset, the query sensitivity, the mechanism, the clipping norm and the number of training steps, and a chart showing utility falling away at some universal value of epsilon is a picture of an assumption. Measure the trade-off on the actual data and the actual task, and report the measured pair. Anything else hands a reviewer a number that cannot be defended.

For training on sensitive records directly, DP-SGD clips the L2 norm of every per-example gradient to a fixed threshold, bounding what one record can do to the step, then adds calibrated Gaussian noise to the summed batch gradient before the update. PATE takes a different route: partition the sensitive data into disjoint subsets, train one teacher per subset, have the teachers vote on unlabelled public data through a noisy aggregation, and train the released student model only on those labels. The student never computes a gradient over a patient record.

The submission is an export, not a writing exercise

ISO/IEC 42001, published in December 2023, is the certifiable management system. It brings the Plan-Do-Check-Act shape and the Annex A control set, and its distinctive requirement is the AI system impact assessment at Clause 8.4, which asks who could be harmed and requires that the assessment be repeated at planned intervals and whenever something significant changes. The data controls, covering provenance, quality and preparation, sit in Annex A rather than in Clause 8.

The NIST AI Risk Management Framework is the other half, and it is voluntary. Govern, Map, Measure and Manage give an organisation a vocabulary and a method for finding socio-technical hazards, and no operating model whatsoever. Treating the two as competitors wastes both. NIST says what to look for. ISO 42001 is the auditable machinery for proving somebody looked, on a schedule, with records.

The platform is the bridge between them, and the bridge is mechanical. If lineage telemetry already carries cohort composition, transformation history and the separation between development and test populations, then the Data Card, the Model Card and the bias testing record are queries over metadata the pipeline emitted anyway. If it does not, they are a writing exercise performed under deadline by somebody reconstructing decisions from memory. That is the whole distance between 95.5% and a number worth reporting.

Worth saying plainly: that study covers clearances through July 2023, before the FDA finalised its change control guidance and before most of the current transparency expectations landed. Devices cleared since are not necessarily documented that way. The finding is still the right warning, because no amount of guidance changes what a platform is able to retrieve after the fact.

A change control plan is a control loop

Static approval and continuous learning do not fit together. A locked algorithm cannot degrade quietly, and it also cannot improve. The Predetermined Change Control Plan is the regulator’s answer: an authorised description, filed before market, of exactly how the model may change afterwards without a new submission. The FDA finalised its guidance on 4 December 2024, broadening it from machine-learning devices to AI-enabled device software functions generally and moving its data vocabulary from training and testing to training, tuning and testing. Health Canada runs a parallel arrangement for machine learning-enabled devices.

A plan that passes has four parts, and every one of them is a demand on infrastructure.

• Triggers. Named, observable events that start a retraining cycle: an accumulated volume of adjudicated feedback, a scheduled interval, a monitored metric crossing a stated bound. "We will retrain when needed" is not a trigger.
• Data expansion protocol. Explicit inclusion and exclusion criteria for records entering the next training set, including which demographic groups are being added and on what basis.
• Performance envelope. The bounds the updated model has to stay inside, stated as numbers beforehand. Sensitivity at or above one figure, specificity at or above another, evaluated on a held-out set the platform can prove was never used in development.
• Rollback. Automated reversion to the previous version when the envelope is breached, fast enough to matter clinically.

Assembled, that is a closed loop with the data platform in every position. It observes the trigger, assembles the expanded set under the privacy controls, evaluates the candidate against the authorised baseline, promotes or reverts, and writes each step into the audit record mapped to the quality management system. None of those steps is optional, and none of them is a document.

Serving is where the cost lives

A cleared model still has to run, and in the field the binding constraint is usually memory rather than arithmetic.

Autoregressive generation keeps a key-value cache holding the context of every token produced so far. Traditional serving stacks reserve that cache contiguously, sized for the longest sequence the model might emit. Real sequence lengths vary enormously, so most of the reservation goes unused, and existing systems waste 60 to 80% of KV cache memory. PagedAttention, the idea vLLM is built on, borrows paging from virtual memory and allocates the cache in fixed-size non-contiguous blocks as tokens arrive, taking that waste under 4%.

The freed memory is what makes continuous batching work. Rather than waiting for every sequence in a batch to finish, the scheduler admits a new request the moment a slot opens. The vLLM authors measured up to 24 times the throughput of HuggingFace Transformers, and up to 3.5 times that of Text Generation Inference. Quote the baseline whenever quoting the multiple: 24x against a library that was never a serving stack is a different claim from 3.5x against one that was.

NVIDIA Triton solves a different problem. A clinical inference path is rarely one model. A study is read by a vision model, the report text by an embedding model, the combination scored by a gradient-boosted classifier, and the summary written by a language model. Triton runs that ensemble on one GPU as a single served pipeline, which removes the network hop between four microservices and the four sets of idle accelerators behind them. Its TensorRT-LLM backend adds ahead-of-time compilation and aggressive quantisation on top.

Published head-to-head throughput numbers between the two flip between releases, and are almost always measured on a model and an accelerator that are not the ones in question. Treat any fixed percentage lead as a claim with a date attached, and benchmark the pair actually going into production. Most estates end up hybrid anyway, routing ensemble and classification work to Triton and high-concurrency generation to autoscaling vLLM behind one gateway.

The loop the field closes for you

Deployment is where the risk starts rather than where it ends.

NIST published the Generative AI Profile, AI 600-1, in July 2024, cataloguing twelve risks that foundation models make worse. Confabulation is the one clinicians meet: fluent, confident and wrong. The profile is written largely around single-turn content generation, and the gap worth naming is agentic operation. A prompt injection against a single-turn model produces bad text. The same injection against an agent holding tool access produces actions against real interfaces, and each step compounds the last. If a model in the estate can call an API, the controls belong in front of it at runtime: input filtering, intent-scoped permissions, and rate limits per tool rather than per user.

The clinical failure mode is quieter, and it runs straight into the retraining loop. Automation bias is over-reliance, where the clinician stops checking and misses the error buried in the detail. Dismissal bias is the opposite, caused by false positives: alerts that are wrong often enough get ignored on the occasion they are right. Both distort what a clinician does with an output.

Then the loop closes. A clinician accepts a wrong recommendation. The acceptance is logged as confirmation. The trigger fires, the platform retrains on the accumulated feedback, and the error is now in the weights with the model more confident than before. That is feedback loop bias, and it is the one failure mode where automated retraining actively makes the system worse.

The defences are unglamorous, and all of them are platform features.

• Monitor the input distribution against the training distribution and alert on divergence, rather than waiting for an accuracy drop that arrives as a complaint.
• Log disagreement between clinician and model as a first-class signal instead of discarding it.
• Adjudicate a sample of accepted recommendations against outcomes, rather than treating acceptance as ground truth.
• Hold labels produced by a model-assisted decision in a separate class from labels that were not, so a retrain can exclude them or weight them down.

The recall figure is what the absence of those looks like at population scale. Forty devices, 113 recalls, mostly software and algorithm faults, drawn from a population where most submissions never described the training data in the first place.

What this changes about build order

The uncomfortable part is the sequencing. Lineage capture, privacy accounting and evidence generation cannot be added to a platform after a model works. They are properties of how the data was moved, and a pipeline that did not record the movement cannot be made to remember it later.

So the order is fixed. Capture provenance and attribute-level lineage from the first ingestion. Put the privacy budget under central accounting before the first exploratory query runs. Make the Data Card an export from that metadata rather than a document with an owner. Serving topology and monitoring can be changed later at ordinary cost. The record cannot be backfilled at any cost.

Compliance is not the report written at the end. It is whether the system can answer the question when somebody asks. Of 691 devices, 660 could not.