In October 2025, Deloitte handed money back to the Australian government. The firm had delivered a AU$440,000 review to the Department of Employment and Workplace Relations; the published report contained a fabricated quote from a federal court judgment and citations to academic papers that do not exist (The Guardian, 2025). A revised version disclosed that GPT-4o had been used in drafting. Later reporting put the refund at roughly AU$97,000 — the contract's final instalment.
One detail matters more than the money. The fabrications were not caught by the firm's review, or by the client's. They were caught by an outside academic who checked the references. The deliverable had passed every layer of scrutiny it was designed to pass, because nothing in it looked wrong.
That is the category: not the model that fails, but the model that returns. Ask a language model to total revenue across two hundred rows and you get an answer with the right magnitude, the right currency symbol, and a confident sentence around it. What you do not reliably get is the sum. A missing answer gets caught. A wrong answer with good posture gets forwarded to the board.
And the posture never breaks, because the confidence is also generated text. The model does not know it is wrong; there is no internal alarm to suppress. The sentence asserting the number and the number itself come from the same place — the next likely token.
Plausible is not computed
The instinct is to file wrong numbers under model maturity: surely the next version adds correctly. The research says the failure is structural, and it is unusually consistent about why.
GPT-4, asked to multiply two three-digit numbers, got 59% right zero-shot; ChatGPT, 55% (arXiv, 2023). At four digits, GPT-4 fell to 3%. At five, 0%. The Faith and Fate authors traced the mechanism: transformers handle compositional problems by matching linearized fragments of what they have seen — pattern lookup, not procedure. The cliff sits exactly where the patterns run out.
Addition is no safer. A 2025 analysis found that LLMs add with a one-digit lookahead heuristic rather than an algorithm: accuracy collapses precisely where carries cascade beyond one digit, regardless of prompting or tokenization, and the failures are predictable from carry structure alone (arXiv, 2025). Systematic, not noisy. The model is not almost-computing. It is doing something else that often agrees with computing.
Apple's GSM-Symbolic closed the remaining loophole — that maybe this only afflicts hard arithmetic (Apple, 2024). Take grade-school word problems and change nothing but the numbers: every state-of-the-art model tested got worse. Add one plausible-sounding but irrelevant clause: accuracy dropped by as much as 65%. And accuracy varied noticeably across fresh instantiations of the same question template — the same problem, reworded, returns a different distribution of answers. That last finding explains the demo problem. A demo is one draw from the distribution. An audit is the distribution.
The authors' verdict — "current LLMs are not capable of genuine logical reasoning" — is unusually blunt for a research paper. Nobody has retired it.
The obvious fixes, measured
Every team that meets this failure reaches for the same four fixes. Each one has been measured.
Retrieval. FinanceBench asked GPT-4-Turbo 150 questions about public financial filings, with a retrieval system supplying the documents. It incorrectly answered or refused 81% of them (Patronus AI, 2023). Sixteen configurations — GPT-4-Turbo, Llama 2, Claude 2, vector stores, long context — showed weaknesses across 2,400 manually reviewed answers, and models hallucinated figures whenever the evidence pages were not perfectly supplied. Retrieval fetches the document. The model still botches the number.
Grounding. The BBC gave ChatGPT, Copilot, Gemini and Perplexity direct access to its own articles and asked them about the news. 51% of answers had significant issues; 91% had at least some. 19% of answers that cited BBC content introduced factual errors — wrong statements, wrong numbers, wrong dates — and 13% of quotes attributed to BBC articles were altered or never existed (BBC, 2025). The source was in hand. The numbers still bent.
A smarter model. OpenAI's own system card reports o3 hallucinating on 33% of PersonQA prompts and o4-mini on 48% — against 16% for the older o1 (OpenAI, 2025). By the vendor's own measurement, the newer reasoning models hallucinate two to three times more than their predecessor. o3 simply makes more claims — more correct ones and more fabricated ones, delivered at the same temperature of certainty.
Better prompting. Step-by-step scratchpad prompting lifts GPT-4's three-digit multiplication from 59% to 92% (arXiv, 2023). Better — and still one wrong product in twelve. Even fine-tuning GPT-3 exhaustively on four-digit multiplication produced about 40% on unseen four-digit problems, and 0% at five digits. The ceiling is the mechanism, not the prompt.
Every fix moves the odds. None of them changes who does the arithmetic. The answer is still sampled from a distribution of plausible numbers, and exactly one member of that distribution is the sum.
Change who does the arithmetic
The structural answer is a division of labor drawn exactly along the competence line. Language models are superb at language and unreliable at ledgers, so the model should never be the thing that computes.
SQAI is built on that split. The model authors a typed intent — sum amount where region = "east" — and a deterministic engine executes it as compiled code: validated against a hash-pinned contract, checked against policy, computed in float64 on a single thread, returned with provenance.
{
"status": "ok",
"value": 2130.5,
"rows_matched": 5
}
That 2130.5 is not a likely token. It is the sum, produced by a deterministic kernel from a surface of 4,574 read-only capabilities across 445 modules — 4,564 of them fully deterministic — answering in 0.83–0.93 ms warm.
Deterministic means checkable. finance.npv(0.1, [-1000, 300, 420, 560, 680]) returns 505.020148896933 under computation hash b74f67d0… — the same value and the same hash whether the call is made from TypeScript or Python, because results are hashed over one canonical JSON form. The claim is scoped honestly: deterministic within the declared execution scope, with an envelope recording runtime version, platform, precision mode and thread count rather than overclaiming. A model's answer to the same prompt cannot promise to match itself on the next run. This one replays byte-identically, months later, in either language.
There is a second leak, and most stacks leave it open. Attach a compute engine, then pour the raw query rows into the context "for summarization" — and the model quietly re-derives, and re-invents, numbers from the rows. SQAI treats the context as a governed boundary: at most 25 rows, 250 cells and 32,000 bytes ever reach the model, behind a default limit of 100 rows and a hard execution ceiling of 1,000. Truncation is always declared, so the model cannot honestly claim to have totaled what it saw. Partial rows are never shown, so no half-record tempts it into completion from imagination. Aggregates are computed before the boundary. The model receives the answer, not the homework.
And the policy is not a prompt. Allowed sources, fields and functions are fixed in code at createSQAI() time and checked in-process before execution; the model's tool input carries no policy fields at all, so there is nothing for it to widen. Asking for a capability outside the policy does not produce a creative workaround — it produces policy_denied_function. The argument is the same one behind not letting agents write SQL: don't govern generated text with best-effort instructions; govern execution with allow-lists the text can't touch.
Who did the arithmetic?
Next time an AI system hands you a revenue figure, one question sorts every architecture on the market: who did the arithmetic?
If the answer is "the model," you hold a number with excellent posture and unknown provenance, and the only verification path is a human redoing the work — the thing you were trying to automate. Deloitte's fabrications were caught because one academic outside the process decided to check. That is not a control. That is luck.
If the answer is "a deterministic engine, and here is the hash," verification is a replay: same intent, same engine, same bytes. The failure gets subtler — and quieter — when the wrong number comes from a fanned-out join rather than wrong addition, which is its own story. But the discipline fits in one sentence: the model writes the question, never the answer.
The wrong revenue number doesn't announce itself. It arrives formatted, cited and confident — and at five digits of multiplication, never right. Build the pipeline so it can't get in.