Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo

The note that looks completely fine

00:12This is a clinical note an AI wrote from a real consultation. Take a few seconds and read it. It reads like a routine headache. A new headache likely tension type. Take some paracetamol. Come back if it doesn't settle. Looks completely fine, doesn't it?

00:25Here's what's missing in the room. She also mentioned her jaw aches when she chews. A new headache over 50 with jaw pain on chewing. That's giant cell arthritis. And untreated, it can take her sight within days. It's the same day start steroids now emergency.

00:42And that one line, it never made it into the note. On the page, it's a paracetimal headache. And nothing in the note is technically wrong. It's the dangerous part is what isn't there. And so that's what I'm going to talk about today. The dangerous failures are often the ones that actually look completely fine.

01:07Firstly, who am I? I'm Seb medical doctor by background and now run Composo where we build AI evaluation systems for high stakes domains. So, that one was a subtle kind of error, but sometimes it's not subtle at all. A man in his 20ies sees his GP for a sore throat, tonsillitis.

The obvious errors, and how common they are

01:29The AI writes that up. It gives him chest pain, suspect angina, diabetes medications he's never taken, and an address for a hospital that doesn't exist. And I I really like the LLM for this one. I think it's it's a good attempt at hospital name.

01:37Um, and weeks later he's invited to diabetic eye screening for diabetes he doesn't have. That's genuinely a real case that happened recently. Obviously, these kind of crazy ones someone notices, but it's those quiet ones that sit in the record uncaught that are the most challenging and can actually do a lot more damage.

02:06And they're not rare at all. In the largest real world study of these notes, about one in 20 carried an error that was serious enough that it could cause significant harm to the patient. One in 20. That's not theoretical in testing. That's in production on real patients.

02:23And that's only the serious ones. If you widen that lens to all errors, nearly one in five had an important emission and more than one in 10 had a hallucination. And AI is being deployed at scale across healthcare fast. Ambient scribes are one of the leading cases already in about a third of US practices and climbing.

02:43Physician AI used doubled last year. And none of this is tracked. So for most of these systems, there's no reverse event reporting at all. The errors never show up as incidents. They just sit in the record. So errors this common that are going unseen, it's quite hard for me to believe that it's not already affecting patients.

02:57It's not that we checked and it's fine. It's that we are flying blind. And this isn't just a healthcare problem. It's every high stakes use of AI. Healthcare shows it more viscerally because here being confidently wrong can be life and death.

03:16But everything I show you can map straight back onto other domains as well. So, here's what I want to do. I'm going to show you what exactly is going wrong, why it's going wrong, why the systems we build to catch it don't work, and a suggestion at how maybe we can start to fix that.

03:39So, first, what's going wrong and why? So, LM are getting good. Obviously, they don't make stupid mistakes anymore most of the time. So, it's not about dumb errors. Everything here came out of three of the best production ambient scribes on the market.

03:50ones that we all know. We generated a load of notes across them last week and this is exactly what's going on right now. It says every failure we found. Each dot is an error colored by type left to right how much it matters bottom to top

Mapping every failure across three production scribes

04:08whether a strong automated check catches it. And that split is the point. A handful up top get caught. But almost everything sits below the line. The ones I care about most are these on the bottom right. the high stakes and list ones. Let me show you what a couple of those looks like.

04:26So, a woman comes in with a headache. Doctor asks, "Did it come on suddenly or build up gradually." She says she doesn't know it just happened. The note records that as abrupt, sudden onset and sudden onset is a red flag. You can see why it just happened could maybe be interpreted and inferred as abrupt onset.

04:48But that's a feature that points to a bleed on the brain. She never said it. The model decided it. And now that one word drives the whole work up. Here's another. Doctor suggests running some tests. The patient says, "Can we just try antibiotics instead?"

05:00They agree. Hold off from the tests. Treat and see how it goes. Note records the opposite. arrange test today. It kept the plan that they talked out of, not the one they chose. Every line in the note reads fine because it's not really a hallucination at all.

05:15It's not wrong. It was there in the original, but it's just not what they ended up deciding. So, why are these happening?

Mishearings, additions, changes, omissions

05:33There's, you know, in Ambient Scribes, there's first transcription and then generation. A lot of it does happen on the transcription layer. It can be words misheard for their sound alikes. So humalogue heard as humalin two insulins on completely different timelines.

05:49So swapping them could crash a blood sugar. Hyperthyroidism becomes hypothyroidism the opposite condition or a dropped to no on uh no evidence of cancer that becomes evidence of cancer. So these these are really hard problems and they are common, not the ones I'm going to focus on because most of what goes wrong is actually even with a perfect transcript.

06:14It's the model reading the words correctly and still doing one of three things. Either it adds something that was never said, it changes something that was, or it emits something that should be there. Now the blatant version of each of these is is really easy to catch.

06:28The hard part in all three is the same. It's telling whether that thing that was added or changed or dropped actually matters. It's detecting that slight over inference versus the dangerous fabrication. The harmless rephrase versus the meaning flip.

06:44A dropped line of small talk versus a dropped allergy. So the ones that matter slip through along with all of the ones that don't that call which different matters is taste effectively not aesthetic taste but essentially

The hard part is knowing what matters

07:08judgment it's it's whether in this context a missed allergy might kill someone or is not important and I think there's there's three properties that really matter about this it's tacit so your domain experts have it but they can't fully write it down.

07:23It's contextual. So the same detail is critical in one note noise in the next. And it's moving. The model changes, guidelines change. Two good doctors disagree. Different hospitals have different definitions. And so there's no fix to target to write down.

07:39And so the model knows the facts. Ultimately, they're extraordinarily capable. But what they lack is a sense of what matters here for this specific example. And that's why even brilliant models make these mistakes. So one natural move, you're never going to make that generator perfect.

07:53And generator is cheap. Generation is cheap.

Put a checker after the generator

08:00So stop fixing it at the source. Let it write. Put a checker after it. Pass only what close the bar. And that checker should be the easier job. The generator has to get everything right and it pay attention to lots of varying instructions. Whereas the checker only has to find the one thing that's wrong and just focus on that task.

08:15You can also give it more time, more tokens, the exact failure modes to hunt for. Evaluation should be easier than generation. It's the asymmetry of verification verifies law. And that's why AI is raced ahead anywhere you can cheaply check the answer, maths and code.

08:35And doing this is exactly what the best teams do. They put a lot of energy into evaluation. It starts with the gold standard which is expert humans reviewing notes which obviously works offline but you can't put a human on every note in production.

08:42So they automate it. They build a serious system and some of the best versions of this that I've seen are you take the transcript and the note and context put it in front of the judge a detailed rubric for faithfulness with worked pass and fail examples.

09:06the rubric maybe auto optimize with Jeeper or something like that. Maybe you have some deterministic NLP to sort of count up medical concepts that are differing between the two. That's a powerful system and yet I pulled all of those errors earlier out of ambient scribes in an afternoon.

09:20So if the evaluation is this good, how are these errors still getting through? So I built this system and ran those same notes through it.

The best judge waved a fifth of them through

09:40and it scored most of them fine. It flagged a handful of them and signed off the rest. But one in five of those clean passes still had some sort of serious error buried in it. And often that was an emission. The things that should have been there and actually quietly weren't.

09:55And that's the best version of a judge that I've seen in a lot of teams and it waved them through. Why did it do that? It's not stupid. It's a frontier model. Serious engineering behind it. More than clever enough to read the whole encounter and catch every obvious error.

10:09And it's not blind either. And and that's part of the trap. If you take a note that says start a moxicillin when the real decision was actually to wait and see. It's faithful to the words did come up, but it's a lie about the intent. A good judge might catch that, might, but whether it flags that versus the other dozen other things that it could comment on depends on it knowing what decision matters most.

10:41And so it's not blind. It just can't tell what counts essentially. So the note passes confidently and you put a judge like that in front of your system. You've not added a safety net. You've added a second silent failure that just nods along with the first.

10:59And here's the root of it. So in mats or code, the verifier comes for free, a unit test, a compiler. But for is this note safe and complete, there's no unit test. You have to build the verifier yourself. And verification is only easier than generation for the easy bit, i.e.

11:16spot the difference between transcript and note. But that's not the hard bit. The hard bit is knowing of all those differences you've seen which matter and that's harder than writing that plausibly good gener note in the first place because that standard of good was never written down anywhere that the judge can read it.

11:31A rubric that you prespecify is only the taste you could write down. The taste that matters is the part that you couldn't. And so here's here's a bit more detail on what what matters looks like. Two patients both with blood in their urine. Both notes dropped the same kind of line where they'd been on holiday.

11:51One had been to France, the other's late Malawi. Same emission, same shape, same mistake. Well, not really because blood and urine obviously worrying you away and you're going to investigate it. But the France

France versus Lake Malawi

12:09trip is irrelevant. The Lake Malawi trip is the diagnosis. Fresh water in subsaran Africa means just someis until proven otherwise. And it completely changes what the management plan is. So that same drop line in one note is pure noise. In the other, it's the answer.

12:24And which one it is, you simply just can't write all of that down in advance. So if you can't write it down, you can't write taste down. How do you get that into your evaluator and your whole application system? Well, we've answered a version of this before.

12:36RHF exists because you can't write the reward function for good. You learn it from examples by showing it. The only question is where you keep what you've learned. And there's three places. You can either specify it up front. You can stuff the prompt, write the perfect rubric.

12:52We just watch that fail. Essentially, you can bake it into the weights, fine-tuning or continual learning. But for a standard that's still moving and a score that has to be explainable, the weights, I think, are the wrong place to keep that.

13:08They go stale. They can't tell you why. and you can't change them without a retrain. So there's the third option which I'll show you which is you essentially just keep the taste as the examples themselves. Past judgments, expert corrections, references, and for each output you retrieve the ones that bear on it into the judges context.

13:30Add one and it's live on the next call and you can point at exactly what moved the score. For this problem, it's both better and also cheaper to do. So that's the way to do that. That's one repeating loop. Three steps. Discover the failure modes from real outputs.

Discover, capture, calibrate

13:51Capture how your experts judge them. Calibrate every output against that. And when the standard moves, the loop moves with it. So in more detail, discover. You don't write that rubric in a vacuum. You have to put the system in production and look at the real outputs.

14:03You cluster what goes wrong and the failure modes surface on their own. You name them. This is your failure mode ontology. discovered from your data, not guessed on a whiteboard. And you can't shortcut it. The ways that a real system goes wrong are effectively unbounded.

14:20And synthetic test cases only cover the failures you already imagined. The ones that hurt you are often the ones that you didn't. And you only find those in real outputs. So this ontology is your map what to capture judgment on, what to retrieve against, including the failures that you'd never thought to check for.

14:39After that, it's capture and then calibrate. So those discovered modes, they're not a checklist that the judge runs, but they organize everything. What your what you ask your experts about, how you index the cases that you'll retrieve. And capturing is a simple part.

14:54You put real outputs in front of your experts. Clinician spends a focused few hours leaving comments. A session doesn't have to be a monthsl long labeling project to start with. And you collect their judgment, not just a score, but the reasoning and corrections.

15:02And over time, you build up that record of how your experts actually judge. You then calibrate. That's the the the generic part of this. You can write down once easily. For example, be faithful or don't drop anything important. But what you can't write down is what counts as a serious miss for this specific note.

15:25That's contextual and it shifts from note to note. So what we recommend is you assemble that on the fly. For each output, your judging agent pulls in everything that bears on this one case. It's memory of the most similar outputs that it's judged before and how they scored, the expert corrections that apply, the reference drops and guidelines.

15:47It's just context engineering per output and crucially not just one prespecified rubric in a vacuum and not a model that you have to retrain every week, but a full case specific standard assembled for this output. And it's a loop as well. Every output you judge, every correction sharpens the next.

16:03And when a brand new failure mode appears, discovery surface it and it flows straight back in. And so to make that a little bit more concrete, that headache that I opened with, the one that was really a possible blindness emergency, here's the kinds of things that you would want to pull in for that note.

16:20the nearest cases that your experts have judged. Not this exact patient, but the same shape. Maybe a red flag filed as routine. Uh the corrections that apply, like a new headache over 50, um suggest something that you need to check red flags on and some criteria and guidelines.

16:35And you pull all of that in. It hasn't memorized this case. It's a capable model and handled the right context to reason from. Held against that. The drop red flag stands out. It was never actually hard to catch. It just didn't know what mattered.

16:55And so if you take that same data set of generated notes from the start and pass it through these three judging systems, the first a strong offtheshelf judge with a rubric frontier model. Um it's better than a coin flip, but it misses most of what matters.

Three judges on the same notes

17:17The second that sort of serious system that we talked about before rubric jeeper maybe some deterministic checks better again but still missing quite a lot of what counts. The third judge running this loop discovered failure modes calibrated for output against what experts judged is performing a lot better on this specific data set.

17:31same notes. The only thing that changes is what the judge was shown. And the difference here, it's not more compute or a better prompt. It's that the first two fight, taste, and lose. They guess the criteria, they freeze one standard, and they go stale.

17:49This repeating, evolving loop does the opposite. It discovers the modes, fits the standard to each note, and keeps learning. So you might not write clinical notes, but if you ship anything where being confidently wrong has a cost, the

Beyond healthcare

18:08contract review that misses the clauses that change the deal, the support agent that promises a refund you don't offer. The same thing is true for all of those. It's watched if at all by a judge with no taste for what matters in your domain.

18:18So three things. Discover your failure modes from real outputs. Don't guess them. Capture your experts judgment on them, the standard that they can't write down. Calibrate every output against the cases that they've already judged, not a static rubric, not a retrained model.

18:40Then keep that loop running. And if you take one thing away, easiest place to start is your experts leaving free form comments on real outputs. That's the real that's the raw material for everything else. Your judge can verify anything that you write down in advance, but the standard of good never could be.

18:55And so stop trying to write it all down in advance and just start capturing it case by case and evolving it. That's why evaluation can't be a thing you build once and freeze. The standard it checks against doesn't exist on paper. It has to be discovered from real outputs, captured from the people who hold it, and kept alive as it moves.

19:17Evaluation isn't something you have. It's something that you do continuously over time. Thank you.