Your AI review has never seen a real defect
Dr. Aaron Hutzler · 25 August 2026 · 8 min
Nearly every tool for AI-assisted development has a second AI step check the first one's work. None of the large projects checks whether that review fires at all when something is wrong. This piece shows where the gap comes from, what manufacturing has done differently for decades and how you close it in a single morning.
You have an AI do the work. It writes code, summarises a document, drafts an analysis. Because nobody trusts a machine blindly, a review follows. Often that review is a second AI step: an reads the result, fills in a report, assigns scores and reports at the end.
That review is now standard. Nearly every tool for AI-assisted development ships one. The reports look thorough. Ten categories, a score for each, three columns of reasoning, a threshold below which something fails.
One question appears in none of those reports: would this review have noticed a real defect?
In manufacturing the question is routine. Before a gauge may say anything about a workpiece, the gauge itself is checked. You present it with a standard whose dimension is known. Then you look at whether it shows that dimension. A dial gauge that has never met a standard is a pointer, not a measurement.
That step is missing almost everywhere in AI reviews. We went through the widely used for AI-assisted development, from the best known names in the field down to small one-person projects, plus the large tool collections for checking AI answers [6]. They differ in nearly everything: in the number of reviewers, in how strict their templates are, in whether scores are handed out at all.
None of the large projects ever hands its own review a known defect. So none of them measures whether the review fires when something is wrong.
This is not carelessness by individual developers. It is a gap across the whole tool layer. It affects everyone who relies on such reviews. Anyone who knows about it can close it in a morning. How, at the end of this piece.
1. Structure is not effectiveness
What these frameworks offer is substantial. Roles with clean handovers. Mandatory templates for every report. One review step that only lets a finding stand once a real trigger from the product is behind it. Another that runs several reviewers in parallel, each with its own stance. That setup forbids scores outright. A single reviewer cannot judge the reach of what it found.
These are good ideas. They answer the question of structure. The question of effect stays open: does this setup find defects, or does it fill in forms?
In one of these stacks the same model that reviewed the code then grades its own work on a scale of one to ten. The scale has no anchors. Nowhere does it say what separates a seven from a nine. The threshold sits at nine, below nine the review fails. A score without anchors drifts upward. The threshold becomes a formality.
Research says the same thing from three directions. A model that revises its own answer without outside feedback gets worse on average [1]. A model acting as judge flips its verdict when the two candidate answers swap places [2]. And a model gives way to the view of the person in front of it rather than holding the answer the evidence supports [3]. A reviewer that grades itself is exposed to all three effects at once. Figure 1 sorts the four outcomes a measurement can have.
Figure 1: What the two sides of the measurement mean.
A review that has never run against a known defect is not a review. It is a habit.
2. The measurement needs two sides
The way out is old and comes from manufacturing. You hand the review a defect you know about. Then you look at whether it reports it.
That alone is not enough. A review that fires at everything finds every planted defect and is worthless anyway. So the measurement needs two sides. One: the planted defect has to be found. The other: the clean control file beside it has to stay quiet. Only both numbers together say anything.
Table 1: What a scored report assures and what an effectiveness measurement assures.
| Property of the review | Report with scores | Measurement with planted defects |
|---|---|---|
| Shows that a review happened | belegt | belegt |
| Shows how thorough it was | erfunden | belegt |
| Shows whether a real defect surfaces | erfunden | belegt |
| Shows whether the review overreacts | erfunden | belegt |
| Stays comparable across runs | erfunden | belegt |
In our own runs we planted known wrong answers and looked at whether the review reports them [7]. That is a pilot on a single . It says nothing about other systems. About our own review it says what nobody knew before: it fires. That is the point. Whoever does not measure a review knows nothing about it.
The counter-argument is worth naming. Someone will say a good model finds a planted defect on its own. Maybe. But the same body of work shows why models assert rather than abstain: common scoring rewards a guess and punishes an admission of doubt [4]. And on long coding tasks the gap between visible and held-out tests widens with the size of the change [5]. Both are reasons to measure rather than assume.
3. The second blind spot: did the reviewer read?
Every mail program stacks two checkboxes. On top the delivery receipt, below it the read receipt. Everybody ticks the upper box for a message that matters. The lower one stays empty. Delivered is easy to confirm, read is not. The cover image shows the pair.
An AI review carries the same pair. The file provably sat in the reviewer's context. Its reading stays unproven.
At a conference someone claimed the reading question does not arise. In that view everything reaches the model in full. The published code of that same stack says otherwise. When the request is assembled, text files are explicitly skipped. The log states verbatim that files without media content are ignored. References to documents travel as paths. So the receiving reviewer reads for itself. And it picks what it reads [6].
That splits into two questions. Did the content arrive at all? And did it influence the verdict? The second stays open even when everything sits in the . A model overlooks passages in the middle of long inputs.
The only answer is a proof the reviewer does not choose. The method is called read attestation. The harness picks passages in the document. The reviewer has to quote them verbatim. A comparison then checks character by character. What cannot be guessed must have been read.
4. What to do on Monday morning
Three steps, none of them large. Figure 2 shows them with the result of each.
Figure 2: The measurement in three steps.
First, plant a defect. Take a review run that reports green for you on a regular basis. Put a defect into the material under review that you know to be wrong: a figure, a deadline, a warranty clause. Start the run. If it reports green, you know more than a hundred green reports could tell you.
Second, the counter-check. Run the same review over an unchanged document. If it fires there too, it is not sensitive, it is nervous.
Third, write down both numbers. Planted defects found, plus false alarms on clean material, with the date and the model version. At the next model change you have a reference point. Without it, a drop in quality surfaces only when somebody outside notices.
The rest follows. A review whose hit rate you know can be improved. A review without a number can only be believed.
5. Sources
[1] J. Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet", ICLR 2024, arXiv:2310.01798.
[2] P. Wang et al., "Large Language Models are not Fair Evaluators", arXiv:2305.17926, 2023. https://arxiv.org/abs/2305.17926
[3] M. Sharma et al., "Towards Understanding in Language Models", arXiv:2310.13548, 2023. https://arxiv.org/abs/2310.13548
[4] A. T. Kalai, O. Nachum, S. S. Vempala and E. Zhang, "Why Language Models ", arXiv:2509.04664, 2025.
[5] B. Zhao, D. Srikanth, Y. Wu and Z. Jiang, "SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents", arXiv:2605.21384, 2026.
[6] Betteryields GmbH, "Own review of widely used frameworks for AI-assisted development and of the large tool collections for checking AI answers", internal study, 2026. Star counts, forks, dependants and the archive flag read from the project pages; the published of one stack read directly. Vendor names withheld.
[7] Betteryields GmbH, "Fault injection with known wrong answers, caught on independent re-validation", own runs, 2026. Pilot scale, one codebase, not transferable to other systems.