Confidence Is Not a Signal
On the night of July 29, 1984, a man broke into Jennifer Thompson’s apartment in Burlington, North Carolina, and raped her at knifepoint. She made a decision during the assault that the criminal justice system has always rewarded: she studied his face. She made herself memorize it. Afterward she picked Ronald Cotton out of a photo array, then picked him again out of a live lineup, and at trial she told the jury she was certain. She was certain at the second trial too. Ronald Cotton served more than ten years before DNA matched a different man — Bobbie Poole, a convict who had reportedly bragged about the crime, and whom Thompson, when she finally saw him, did not recognize at all.
Her confidence was total. It was also wrong. And the thing the jury did with that confidence is the thing this essay is about: they treated it as evidence about Cotton, when it was only ever evidence about Thompson. It told them how her memory felt to her. It told them nothing reliable about who had been in the room.
Thompson is not an outlier. Of the first 375 people exonerated by DNA in the United States, 69% had been convicted at least in part on a mistaken eyewitness identification — more than any other single cause. For most of the twentieth century the law ran on a folk theorem that a confident witness is an accurate witness, and that theorem put hundreds of innocent people in prison. Confidence and accuracy are different quantities. One is a feeling the speaker has about a claim; the other is a property of the world. They can move together, and under the right conditions they do — I’ll come back to that, because it is the most important part. But they are not the same measurement, and a system that reads one as a proxy for the other will be wrong in exactly the cases where it most needs to be right.
I am writing this because I have spent the last few months building the machine version of that mistake, and I did not notice until the tool told on itself.
Most days I run a fleet of agents — coders, reviewers, investigators — and a tool I keep records every consequential decision they make, along with the agent’s own stated confidence in it. As I write this, in mid-June 2026, that trail holds 2,804 decisions from 51 distinct agents. The first thing you notice in the aggregate is that the confidence numbers barely move. The median is 0.80. More than three-quarters of every decision my fleet has ever made lands between 0.70 and 0.90 — regardless of the agent, regardless of the task, and regardless of whether the decision later turned out to be right.
That chart is the distribution of a feeling. It is not evidence the feeling is wrong — because, and here is the part I am least proud of, for almost none of those decisions do I actually know what happened. The tool has a field for the outcome. The field is mostly empty: barely five percent of those 2,804 decisions — 141 of them — ever get a result recorded against them. And yet the tool, totaling up that thin five percent, had been cheerfully reporting a single summary flag: calibrated: true. I had glanced at that flag for weeks and felt reassured by it.
It was Jennifer Thompson’s jury in a config file. A confident-sounding summary, resting on a sample far too small to support it, and I was treating the confidence of the summary as evidence about the fleet. It was only evidence about the summary.
The meter was built to read full
The reason this is worth more than a personal embarrassment is that the number my agents emit — the 0.8 they nearly always emit — is structurally untrustworthy, and the research on this has gotten sharper and more alarming in the last year.
Start with the size of the gap. A 2025 study of five models found they “overestimate the probability that their answer is correct between 20% and 60%.” That alone would be survivable if the error were uniform — you could subtract it off. But the same study found the bias is not uniform: it “increases sharply” exactly as the model becomes less sure it is right. The overconfidence is largest in precisely the region where you were hoping the number would help — the hard cases, the ones near the boundary, the ones you wanted the model to flag.
And this is not a bug someone forgot to fix. It is installed by the training. OpenAI’s own GPT-4 technical report noted that “the pre-trained model is highly calibrated (its predicted confidence in an answer generally matches the probability of being correct)” — and that “after the post-training process, the calibration is reduced.” The figure caption is blunter: “The post-training hurts calibration significantly.” The very step that makes a model pleasant to talk to — alignment to human preference — teaches it to sound sure, because sure is what humans reward. We trained the meter to read full.
Nor does it improve when the model “thinks.” A June 2025 paper on reasoning models — the ones that deliberate at length before answering — found them “typically overconfident, with self-verbalized confidence estimates often greater than 85% particularly for incorrect responses.” Read that clause again: the highest confidence was pinned to the wrong answers. The deliberation did not surface doubt; it manufactured conviction. And the picture is not unanimous — a NeurIPS 2025 paper found nearly the opposite, that reasoning models express confidence better than their non-reasoning counterparts in 33 of 36 settings, because slow deliberation gives them room to revise a number downward. Both can be true, and that is the point. “The model said 0.9” is not a fact you can build on, because whether that 0.9 means anything depends on the model, the task, and the way you asked — in ways the number itself does not disclose. A measurement you cannot interpret without reading a research literature is not a signal. It is a number that feels like one.
What it costs to read it as a signal
The cost is now showing up on invoices, and I have totted some of them up before: Deloitte refunding the Australian government part of a A$440,000 contract over a report that cited academic papers which did not exist and quoted a federal judge who never said it; courts through 2025 sanctioning lawyers for the identical failure, two of them fined $3,000 each in a Colorado case that July over a brief riddled with citations to cases that were never decided. The point here is narrower than that ledger, and its sharpest illustration is also the newest. In April 2025, the AI support bot for the code editor Cursor told users that the product “is designed to work with one device per subscription as a core security feature.” There was no such policy. The bot had invented it and stated it with the flat assurance of a published rule, and customers — reading the assurance as a signal — cancelled subscriptions over it before a human caught the mistake.
In every one of these, the fabrications were not hedged. Nothing in their presentation marked them as lower-confidence than the true sentences around them. That is the failure, exactly: the model’s confidence carried no information about its correctness, so the reader had nothing to go on but the prose — and the prose was immaculate. Confident and wrong does not announce itself — and that is what makes it expensive.
The exception that tells you the rule
So is confidence simply noise? No — and the exception is the most useful thing in this essay, because it tells you what would have to be true for a confidence number to mean something.
Go look at a weather forecast. When an American meteorologist says there is a 30% chance of rain, it rains about 30% of the time. This is not a figure of speech; it is measured. In a 1977 study of every precipitation forecast issued by National Weather Service forecasters in Chicago over four years — 17,514 of them — the days assigned a 30% probability saw rain on 28.5% of occasions. Across the whole range the forecasts tracked the diagonal so closely that their reliability error came to 2.8 percentage points. Weather forecasters are among the best-calibrated human beings ever measured.
They are not well-calibrated because they are humbler, or smarter, or because rain is easier to call than a code review. They are well-calibrated because of one structural fact: every forecast is scored against what actually happened, the next day, every day, for decades. The discipline even has a name for the scoring rule — the Brier score, in continuous use since 1950. It can be decomposed to isolate calibration specifically, so a forecaster who says 70% and is right half the time gets that error handed back as a number they answer for. Calibration is not a virtue the forecaster brought to the job. It is a residue left by a feedback loop. Take the loop away and the calibration leaves with it.
That is the whole thing. Confidence becomes a signal only inside a system that scores it against outcomes and feeds the score back. Weather built that loop. The courts, for most of a century, did not — a witness’s confidence was never systematically checked against whether they were right, so it drifted free of accuracy and nobody noticed until DNA supplied the missing answer key. An LLM ships with no loop at all: its confidence is a single forward pass with nothing on the other side. And my fleet — the embarrassing punchline — had the field for the loop and left it empty, then read its own unscored summary back to me as reassurance.
The number is moving into load-bearing positions
This matters now because the industry is busy wiring stated confidence into places that hold weight. Confidence thresholds decide whether an agent acts on its own or stops to ask a human. They gate which outputs get auto-approved and which get a review. They route work between a cheap model and an expensive one. Every one of those designs treats the number as a signal about correctness — as the meteorologist’s 30% — when what it has under the hood is the eyewitness’s certainty: a single, unscored, training-inflated feeling, largest exactly when it should be smallest. A gate keyed on that number does not fail randomly. It fails toward letting the confident-and-wrong through, because confident-and-wrong is the model’s specialty. And a gate that clears exactly the cases it was built to stop teaches its operator, soon enough, to stop reading its verdicts at all — the alert fatigue that saturates every noisy monitor, now pushed down a level and wired into the confidence score itself.
There is a constructive half to this, and it is not “trust the number less.” A scalar you distrust is still a scalar you are using. The more durable move is to stop asking the model how it feels and start measuring something outside the feeling. The most reliable uncertainty signal anyone has found does not come from the self-report at all; it comes from disagreement across samples — ask the same question several times, or several ways, and watch whether the answers converge or scatter. When Oxford researchers published a method in Nature in 2024 for catching confabulations, the usable signal was the semantic spread of multiple answers, not the confidence of any one of them. This is something you can act on now: sample the same prompt three or five times, cluster the answers by meaning, and gate on whether they agree — not on the lone scalar the model volunteers. It is a small feedback loop you can run at inference time, and unlike the self-report it fails honestly, flying apart exactly when the model is lost. It is a poor man’s Brier score, and it is worth more than the number the model hands you for free.
What I am still figuring out
Whether confidence is a signal at the first, uncontaminated pass. I owe the eyewitness research one more turn, because the modern synthesis is not “confidence is worthless.” Wixted and Wells argued in 2017 that under pristine conditions — an initial identification, a fair lineup, no suggestive feedback — a high-confidence eyewitness ID is in fact remarkably accurate. Confidence carries real information at the moment of first, uncontaminated measurement, and then bleeds it: a single confirming remark from an officer (“good, that’s the one”) can inflate a witness’s remembered confidence by close to a full standard deviation while doing nothing for their accuracy, severing the link. I do not know whether agent confidence has a pristine moment. Maybe the number worth keeping is the one from a single clean forward pass — before the model has seen its own retries, my leading follow-ups, three rounds of self-critique — and everything after it is contaminated confidence, inflating away from the truth. If so, our habit of letting models revise and re-rate themselves is the officer leaning over the lineup.
Whether I can even build the loop. The honest reason my fleet is uncalibrated is not that calibration is impossible; it is that scoring a decision against its outcome is expensive and often ambiguous. A weather forecast resolves itself by tomorrow. Whether an architectural call was “right” may not be knowable for months, if ever, and may have no clean ground truth at all. I can build the Brier loop for the decisions that resolve cleanly. I do not yet know what to do about the ones that don’t — which are also the ones where the confidence number is doing the most unearned work.
Ronald Cotton and Jennifer Thompson later became friends, and have spent years explaining to anyone who will listen that she was not lying and not careless. She was certain, in good faith, and certainty was the wrong instrument. The lesson never required doubting her sincerity. It required noticing that sincerity is not accuracy, and building a process that checks one against the other instead of accepting the first as a stand-in for the second.
The model that returns 0.9 is sincere in exactly that way. It is reporting, as faithfully as its training allows, how the claim feels from the inside. The mistake is not the model’s. It is ours — every time we read that number as if it came from a meter that had been checked against the world, when most of the meters in front of us, mine included until recently, have never been checked at all. Confidence is not a signal. It is a feeling with a decimal point, and it becomes a signal only when something keeps score.