The Detector That Said Yes to Everything
In 1955, Paul Meehl and Albert Rosen published a paper about screening tests that look good and are not. Their example was an Army inventory meant to find the inductees who would wash out of basic training. It caught 55 percent of the men who did wash out and flagged 19 percent of the men who did not.
Set that against a population where 5 percent wash out. Per 10,000 inductees it flags 2,080 men and is right about 275 of them. Calling every recruit fine, with no test at all, is right 95 percent of the time. The test is right 80 percent. Their rule: a device has to beat the base rate to be worth running. Their second warning took me longer to earn — a test scored on the groups it was built from reports validity that is “spuriously high.”
For months, I improved Akashi’s conflict detector one false positive at a time. A review found a pattern it should not flag. I added a suppression rule or another sentence to the prompt. The test passed. The next review found a different pattern.
This looked like iteration. It was a detector accumulating excuses.
So I stopped tuning and blind-labelled all 2,772 scored pairs. Dozens of language-model agents saw the two decision texts and structural metadata, never the detector’s verdict.
A model grading a model is a fair objection. My answer is blinding, plus an independent re-rate of 200 pairs that agreed at kappa 0.766 on contradiction versus everything else. That does not make the labels ground truth. It makes them the first ones I did not write myself.
The detector was a constant function
Akashi records agents’ decisions. Conflict detection is meant to find two current decisions that cannot both be true. That is what makes the record more than a log.
The blind labels found 93 contradictions in the 2,772 pairs. More than a fifth were supersessions, a later decision replacing an earlier one, which is normal progress rather than conflict.
The shipped detector said “contradiction” for 97.8 percent of the pairs it scored. Yes, 2,711 times. Ninety-two of those were right. It found supersession 61 times where the blind labels found 627.
This was not a weak or miscalibrated classifier. It was nearly a constant function.
The old evaluation suite had 122 handwritten pairs and reported 1.000 precision and 1.000 recall. I had written the prompt and chosen the examples from the same picture of a contradiction. The suite could check whether the prompt followed the picture, not whether the picture was wrong. A reference defined by the thing it measures drifts without telling you.
The blind corpus could. That is the difference between a test set and a measurement.
It graded me too. Bulk dismissals were 97.5 percent correct; of conflicts I read and resolved with a declared winner, 11 percent were real. An alarm over budget degrades the judgments that remain. A detector that cries wolf poisons the record needed to fix it.
The base rate changed the question
The failure was not only the prompt. It was what I had optimized for. At prevalence p, sensitivity s, and majority-class false-positive rate f:
precision = p·s / ( p·s + (1−p)·f )
False positives decide whether the queue is usable: raising recall from 30 to 80 percent gains about 23 precision points; halving false-positive rate from 2 to 1 percent gains 17. This is Meehl’s induction station with my name on the cutting score.
The scorer has features of its own. significance weights a decision’s consequence; topic_similarity is the embedding overlap that nominates pairs. Against the blind labels they score 0.60 and 0.62 AUC, on intervals about twelve points wide. outcome_divergence scores 0.43, which means it points the wrong way. The strongest predictor is temporal_decay, a staleness weight, at 0.73. These can be a recall funnel, not the decision.
F1 obscured the same fact more politely. gpt-5-mini had the best sample F1, 0.704, and 17.3 percent corpus-projected precision. gpt-5 scored worse on F1 and projected to 41.5. I report precision, recall, and queue size now.
The work was an experiment, not a prompt rewrite
I ran the changes against a blind 200-pair gold set, then reweighted to corpus proportions. A stratified sample can flatter a rare-event detector; the queue is what ships.
The first rewrite failed. I made chronology decisive: a later decision is supersession, never contradiction. Supersession then became the default sink for 54 to 65 percent of every class. Contradiction recall fell to 1.1 percent.
The version that survived is ordered: name the one question both decisions answer, require explicit replacement language for supersession, then ask whether current answers are incompatible. A contradiction verdict must name its disputed question or the parser downgrades it. A contract, not another request to be careful.
Judge capability mattered more than either rewrite. On the same procedure and the same corpus, the judge alone moved corpus-projected precision from 8.1 to 41.5 percent.
For a few hours I believed that number was 65.2. The first estimate came from a 47-pair sample the judge got entirely right. Zero of 47 cannot distinguish 65 percent precision from 19. A 300-pair remeasure found six false positives, and the estimate fell to 41.5. The same arithmetic that convicted the old detector almost flattered the new one.
What survived is still a projection: 41.5 percent precision at 50.5 percent recall, a queue of about 113 where the old detector flagged 2,711. It still has to earn those numbers live.
Some conflicts should not be judged
Twenty-seven percent of the 93 contradictions were two decisions setting the same named parameter to different values. conflict_llm_timeout = 15s and conflict_llm_timeout = 120s do not need a language model. They need a join.
Bindings are first-class data now. Two current decisions that bind the same parameter to different values make a conflict by lookup. No prompt, no threshold, no false-positive rate.
That covers only contradictions whose structure already exists. Four attempts to recover it from prose failed; 59 percent of contradictions never name the artifact they affect. Preserve structure where it exists.
What is live now
Production now runs gpt-5 over a 30-day window and keeps a deterministic 5 percent sample of structurally suppressed pairs outside the conflict queue.
The first cutover inherited the 15-second timeout, so 159 of 200 gpt-5 calls timed out. A timeout skips the candidate, silently lowering detections and imitating precision. conflict_llm_timeout = 15s against 120s was the bug.
The sample creates no operator work and cannot block a conflict. It creates rows for the next blind label. The rater will see the decisions, not the rule that suppressed them. That is how a suppression becomes a measurable claim instead of an accumulated hunch.
The data
The published data reproduces the corpus counts, detector outcomes and feature AUCs. conflict-labels.csv has all 2,772 labels and no text. conflict-pairs-recoded.jsonl has 192 joinable, redacted pairs for running another judge; its 7.3 percent base rate is not the corpus rate. teaching-set.jsonl has 60 authored examples across ten failure modes. The label-noise correction in the next section needs nothing beyond that first file, which supplies the base rate; the two re-rate constants are quoted there.
The release does not include triage history, decision dates, or text for the whole corpus. Everything past the corpus counts rests on my measurement alone: the triage split, the binding-collision share, the no-artifact figure, the dispute count, and every number from the judge experiment.
The labels cannot carry the decimal point
The re-rate that produced kappa 0.766 produced two numbers I did not look at hard enough. The second rater found 81.7 percent of the contradictions the first one found, and called 5.7 percent of the non-contradictions contradictions. The corpus says contradictions are 3.35 percent of pairs. The false alarm rate of the instrument is larger than the thing the instrument is measuring.
There is a standard correction for a noisy screen, Rogan-Gladen. For an observed positive rate p, a sensitivity Se and a specificity Sp, the corrected prevalence is (p + Sp − 1) / (Se + Sp − 1). Specificity is one minus the 5.7 percent false flag rate, so Sp is 0.943 and Se is 0.817. The numerator is 0.0335 + 0.943 − 1 = −0.0235. The corrected base rate is −3.1 percent.
A negative prevalence is not a finding that there are no contradictions. It is the arithmetic refusing to answer: the observed rate is lower than this labeller’s own false alarms would produce on a corpus that contained none, so the protocol cannot resolve a rate this low. For any positive answer the false flag rate has to sit below the base rate itself, below 3.35 percent. Mine is 5.7.
Move that one input. At a 3.35 percent false flag rate the corrected base rate is zero. At 2 percent it is 1.7. At 1 percent it is 2.9. At zero it is 4.1. The true rate is somewhere in that spread and these labels cannot say where.
This does not retract the detector going from 3.4 percent precision to a projected 41.5. Nothing here touches the size or the direction of that move. It also rests on constants I did not publish per pair, so treat 0.817 and 0.943 as sensitivity inputs rather than measurements you can check.
What it retracts is the decimal point. Every number downstream of the base rate inherits its uncertainty, including whether the queue is worth running. At the shipped operating point, 50.5 percent recall against a 2 percent false-positive rate, a missed contradiction has to cost 1.14 times a false alarm if the true prevalence is 3.35 percent, close to the 1.4 I quote below. At 2 percent it is 1.94 times. At 1 percent it is 3.92. That answer is a band, not a number.
The fix is human labels on a calibration subset, weighted hard toward the boring class, because specificity on the majority is the arm that sets the interval. Not sampled in proportion, though. Thirteen positives would put a 38 point interval on sensitivity and settle nothing, so all 93 contradictions get labelled and the sample is spent on the negatives. That is 480 labels, and I have not done it yet. The labels contradicted the detector. Now they contradict themselves.
What I am still figuring out
How much the funnel misses. Top-twenty retrieval leaves 33,151 pairs above the similarity floor unscored. Two of 200 blind labels were contradictions, projecting roughly 332 unseen conflicts against 93 found: funnel recall near 22 percent, interval 7 to 70. Direction, not a settled number.
The structural rules suppress about 56 percent of candidate pairs before any judge. The first 116-pair sample found zero contradictions: an upper bound near 2.6 percent, not proof of zero.
Whether 41.5 percent is worth running. Against not running the detector, a missed contradiction must cost at least 1.4 times a false alarm for the single-judge point to win. That is not a comparison with the cascade. The measured cascade reaches 74.2 percent precision at 38.7 percent recall, but I have not built it or measured the attention cost that decides the trade-off.
Whether the pair is the right unit. Everything above scores two decisions against each other. Work on consistency checking with noisy LLM oracles shows pairwise checks cannot certify a whole set: three decisions can be compatible in every pair and impossible together. The paper offers an adaptive search for a minimal inconsistent subset at polynomial cost. Nothing I run today would see the problem.
Whether the feature deserves its prominence. Across 25.3 weeks the corpus holds 62 distinct disputes, or 2.45 a week. Real value, but not an automatic case for a headline feature. The 0.766 kappa also works against my own figures: some of the apparent room for improvement is label uncertainty rather than detector failure.
The old process had a test after every change, so it could reliably tell me each change matched the last thing I had noticed. The current one claims less: corpus measured, operating point projected, filters observable, live results still to come.
I wrote in June that a confidence number becomes a signal only when something keeps score. This is what keeps score. It is not a victory graph. It is a measurement that has already contradicted me twice: about the old detector, and about my first estimate of the new one. An instrument that can contradict me is worth more than either number it produced.