Why the results of reasoning models still need checking
Dr. Aaron Hutzler · 19 August 2026 · 10 min
Since 2024 the big vendors advertise models with a visible thinking process. These models break a task into steps. They weigh options and correct themselves along the way. The results on exam are impressive. Out of this grows a tempting conclusion: a model that thinks checks itself. So the human check can go.
The published research says otherwise. Three findings carry it.
1. The best reported hit rate is 74.4 percent of the problems
On the AIME 2024 math olympiad qualifier the first large reasoning model solved 74.4 percent of the problems [4]. The predecessor without a thinking process solved about 12 percent of them. On the competitive programming platform Codeforces the reasoning model reached the 89th percentile.
Both are real progress. Both are probability. 74.4 percent of the problems solved means one problem in four goes wrong. The 89th percentile means eleven human contestants out of a hundred score higher. Neither number means every answer is correct. The vendor claims that nowhere. The guarantee exists only in the head of the reader who turns "thinks first" into "checks itself".
A second finding from the same vendor is more serious. The system card of April 2025 [5] tested the newer reasoning models on two factual benchmarks. The third generation model on 33 percent of the person questions. Its predecessor hallucinated on 16 percent of the same questions. On the short-fact test the new model produced wrong answers in 51 percent of the cases. More thinking steps produce more claims. The wrong ones grow along with the right ones. The thinking process is an amplifier. It is no filter. Figure 1 puts the three values side by side.
Figure 1: Fabricated answers by model and test.
2. Self-correction without outside help fails
Researchers at a large AI lab tested what happens when a model revises its own answer with no outside feedback [1]. Performance drops on average. The model discards correct answers as readily as wrong ones. It judges both with the same apparatus that produced the error. A second group showed the same effect on graph coloring tasks [2]: the model solves them poorly. It judges solutions just as poorly. In the loop of generate and self-critique the content of the critique barely influenced the result. The follow-up work [3] measured the internal verifier approving solutions that a formal checker rejects.
One more result concerns the visible chain of thought itself. A 2025 study [7] compared what reasoning models write in their thinking trace with what actually influenced their answer. In most setups the trace revealed the true driver in fewer than 20 percent of the cases tested. Reading the chain of thought is reading a report written after the fact.
Asking the model "are you sure?" is therefore no check. It queries the same mechanism that made the error. The answer is a confident yes with high probability.
3. What works: verification outside the model
The same papers show what does work. Improvement appears once feedback comes from an outside source: a test run, a calculator, a formal checker, a knowledge base [1]. The LLM-Modulo [3] turns this into an architecture. The model drafts candidates. A verifier outside the model accepts or rejects them.
The size of the effect is on record [6]. In a single shot an older model solved 48.1 percent of the HumanEval coding tasks. The best model of that time solved 67.0 percent of them. The same older model inside a loop of write, test and fix solved 95.1 percent of them. The architecture lifted the solved share from 48.1 to 95.1 percent of the tasks. The newer model on its own lifted it from 48.1 to 67.0 percent of them. Figure 2 puts the three arrangements side by side.
Figure 2: Tasks solved per arrangement.
Table 1: What each side can assure about an answer.
| Property of the answer | Reasoning model | Quality gate |
|---|---|---|
| Finds a plausible route to the answer | belegt | erfunden |
| Shows its intermediate steps | belegt | erfunden |
| States whether the figure is in the document | erfunden | belegt |
| Says no when the value is missing | erfunden | belegt |
| Holds the same verdict across runs | erfunden | belegt |
Figure 3: Three tiers and their verdict.
Figure 3 sorts the three tiers by what each one assures. The check can sit in three places. At the exit: a rule system inspects the finished draft. In the loop: the model writes a query or a small program while an engine outside the model executes it. In the data base: the facts do not live in the model at all.
4. Knowledge graphs and neuro-symbolic systems move the check, they do not remove it
The third place deserves its own section. A whole class of tools is built on it. A knowledge graph stores facts as a net of nodes and edges. Ulm is the birthplace of Einstein. Paracetamol relieves headache. A query on that net runs deterministically. The same holds for a rule engine with its if-then rules. It holds for a solver that proves whether a set of constraints can be satisfied at all. The umbrella term is neuro-symbolic AI. The language model handles the language. The symbolic part handles the facts and the rules.
The appeal is obvious. The model no longer answers from memory. It reads from a curated store. It resolves relations across several hops where plain text search fails. The store is corrected in place without retraining. In such a system a rule engine or a solver takes one of two roles. It stands after the draft as a barrier. The model proposes a discount above the internal ceiling and the rule engine rejects it. Or it stands inside the loop as the calculator. The model extracts the parameters. The solver computes the answer. The model puts the result into a sentence. Both roles already have a name in this text. They are .
That is the point the vendor pitch usually skips. The architecture moves the check. It does not remove it. Three seams stay stochastic. First the graph has to be filled. Filling it from documents is the job of a language model. A wrong value enters the store and comes back out wearing the label "verified". Second the user question has to be translated into the query language. On the BIRD benchmark the best model produced a correctly executing query for roughly 55 percent of the questions asked [8]. A wrong query then executes without an error message. It returns the wrong answer with the air of certainty. Third, a solver proves only inside the model handed to it.
Figure 4: Three seams, three gates. The graph itself is not the problem.
Table 2: Where a language model still writes and what the gate there has to check.
| Seam | What the model does | What the gate checks |
|---|---|---|
| Filling the store | extracts facts from documents | the value against the source document |
| Translating the question | writes the database query | does the query answer the question that was asked |
| Wording the answer | turns the result into a sentence | does the sentence still carry the value that was returned |
Figure 4 shows the three seams in the flow. So the honest formula is not language model plus graph database. It is language model plus graph database plus gates. One gate per seam.
For code the third tier has existed for decades: compilers, test runs, static analysis. For documents, research and data analysis it mostly has not. The consequence is the same in both worlds. Reasoning raises the probability. The gate delivers the verdict.
Reasoning raises the probability. The gate delivers the verdict.
5. What to do on Monday morning
Three questions decide the setup. None of them is about the model.
When is a reasoning model worth it? Use it where the task has several steps that build on each other: a design, a derivation, a plan, a draft in code. There the visible intermediate steps pay off. A reader sees the place where it went wrong. Do not reach for it on plain factual lookups. On the vendor's own person-question test the newer reasoning model hallucinated on 33 percent of the questions and its predecessor on 16 percent of them [5]. More thinking produces more claims. The wrong ones grow along with the right ones.
When is a knowledge graph worth it? Ask what your questions look like. If they run across several linked facts, then a graph pays for itself: which part fits which device, under which standard, with which certificate still valid. If your work is checking single documents, a graph is the wrong effort. Then the cheaper control is a comparison against the original document, value by value.
Note one distinction here. A note tool with links between pages is not a knowledge graph, however useful it is for finding things again. A knowledge graph has typed edges, a query language and a answer. Links between notes give you a person who reads faster. They do not give you a check that says no.
What is the smallest gate that helps today? Pick the values that would hurt if they were wrong: the three figures, the deadline, the cancellation clause. For those, look them up in the source yourself, or have a tool compare them. Everything else the model may summarise. The distinction is not how much you trust the model. It is which sentence you are putting your name under.
That is the whole method. Let the model draft. Let a check outside the model decide. And keep the check on the values that carry the decision.
6. Sources
[1] J. Huang et al., "Large Language Models Cannot Self-Correct Reasoning Yet", ICLR 2024, arXiv:2310.01798.
[2] K. Stechly, M. Marquez and S. Kambhampati, "GPT-4 Doesn't Know It's Wrong", NeurIPS 2023 Workshop, arXiv:2310.12397.
[3] S. Kambhampati et al., "Position: Can't Plan, But Can Help Planning in LLM-Modulo Frameworks", ICML 2024, arXiv:2402.01817.
[4] "Learning to Reason with LLMs", vendor announcement, 2024. https://openai.com/index/learning-to-reason-with-llms/
[5] "o3 and o4-mini System Card", vendor system card, April 2025. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf
[6] A. Ng, "Agentic Design Patterns", The Batch, March 2024. https://www.deeplearning.ai/the-batch/how-agents-can-improve-llm-performance
[7] Y. Chen et al., "Reasoning Models Don't Always Say What They Think", arXiv:2505.05410, 2025.
[8] J. Li et al., "Can LLM Already Serve as A Database Interface?", NeurIPS 2023, arXiv:2305.03111.