Most AI accuracy debates are academic. A model scores well on a benchmark, a competitor scores slightly better, and the practical difference is invisible to anyone using either system. Then there are the deployments where accuracy carries legal consequences, where a single wrong output can be pulled from a database years later and examined by someone with subpoena power.
American healthcare has just become the largest live example of the second category, and the engineering lessons emerging from it deserve attention from anyone deploying AI in a regulated environment.
The deployment
Health insurers covering more than thirty million older Americans are paid according to how ill their members’ documented diagnoses show them to be. Those diagnoses live in clinical notes: unstructured text, written under time pressure, in specialty-specific shorthand, running to years of history per patient. Reading it all is beyond human capacity at scale, so natural language processing entered the workflow more than a decade ago.
For most of that period, the systems were evaluated on volume and speed. How many charts processed per hour, how many additional billable diagnoses surfaced, at what cost per chart. Accuracy was reported in aggregate, and aggregate accuracy is the kind of metric that sounds rigorous while hiding almost everything that matters.
The audit that broke the metric
Then the regulators arrived with a different question. Not how accurate is your system overall, but why did it produce this specific output, for this specific patient, on this specific date.
Federal auditors, scaled to roughly two thousand certified coders working a rolling quarterly cycle, began re-checking the industry’s output. Reviews published this spring found that at three insurance plans, 81 to 91 percent of certain sampled high-risk diagnosis codes lacked adequate supporting documentation. A major Medicare Advantage insurer settled federal claims for 117.7 million dollars over technology-assisted review programmes that added diagnoses aggressively while almost never flagging ones that should be removed.
Aggregate accuracy provided no defence, because the auditors were not sampling the aggregate. They were sampling individual records and demanding individual justifications.
What survived
The systems that came through the transition share an architecture worth studying, and evaluations of modern ai tools for medicare risk adjustment coding accuracy now lead with these properties rather than throughput.
Evidence attachment at the inference level. Every suggested code ships with the exact sentence in the clinical note supporting it, the documentation rule that the sentence satisfies, a confidence value, and the identity of the human who confirmed it. The output is not a conclusion, it is an argument with its sources included.
Bidirectional correction as architecture, not policy. The same pass that surfaces missed diagnoses also flags recorded diagnoses lacking evidence. This is harder than it sounds to retrofit: systems designed to search for additions have data models that assume additions, and bolting on a removal pathway after the fact tends to produce a second-class workflow that quietly degrades under deadline pressure.
Deterministic reconstruction. Model versions, rule versions, and source data snapshots are pinned so that a decision made in 2024 can be reproduced under 2024 logic, years later, without relying on anyone’s memory. Teams treating model updates like schema migrations, versioned and reversible, pass audits. Teams hot-swapping weights do not.
Meaningful human checkpoints. Regulators have been explicit that AI in this context is assistive rather than autonomous. The review interface must present evidence rather than conclusions, and the reviewer’s decision becomes part of the permanent record.
The generalisable point
Accuracy, it turns out, is not one property. There is statistical accuracy, which benchmarks measure, and there is defensible accuracy, which means any individual output can be justified to a hostile examiner long after the fact. A system can score brilliantly on the first and fail completely on the second, and healthcare has now demonstrated at nine-figure cost that only the second one holds up when someone checks.
The requirement is spreading. Regulatory frameworks across multiple jurisdictions now demand traceability and human oversight for high-risk AI systems. Financial regulators expect model decisions explainable to examiners. The pattern is consistent: wherever an AI output carries consequence for an identifiable person, the ability to explain that specific output becomes a functional requirement rather than a nice-to-have.
For engineering teams, that suggests a design question worth asking early. Not what is our accuracy rate, but this: if a regulator selected any single output from the last three years and demanded a full justification, could we produce one from stored data, without reconstruction? Healthcare answered that question late, and expensively. The answer is cheaper to build in than to retrofit.
