The $40 Fee That Cost $4,000, and Why Finding AI Hallucinations Is Harder Than Anyone Admits

The most dangerous thing an AI agent can do is
be confidently, fluently, invisibly wrong.

The short version. An AI hallucination is when an agent states something confidently that no real source supports. In a consumer chatbot that’s an annoyance. In a bank, hospital, or insurer’s agent, one confident false statement can be a lawsuit or a safety event. The hard part isn’t catching obvious nonsense. It’s that the dangerous hallucinations look exactly like correct answers, which is why “just ask an LLM to find them” breaks in production, and why detection has to be cheap enough to run on every conversation. Here’s what actually makes it hard.

The $40 Answer

Maya was about to send an international wire transfer, most of her savings, headed overseas to close on an apartment. Before she hit send, she asked her bank’s AI assistant one simple question: “What’s the total fee on a wire transfer this size?” The bot answered instantly and with total confidence. A flat $40, it said. No currency-conversion margin, no other charges on transfers to that country.

Reassured, she sent the wire. The statement came a week later. The $40 outgoing fee was there. So was a ~3% FX-conversion margin and a correspondent-bank charge the bot had flatly denied. A little over $4,000 in all. The bot had been fluent, specific, and completely wrong.

Nothing in the chat looked like an error. No “I’m not sure,” no red flag, no crash. Maya discovered the hallucination only when the real money was already gone.

That’s a hallucination: an AI agent stating something confidently that no real source ever supported. A plausible, well-formatted claim with nothing true underneath. An agent tells a customer a loan was approved when no system said so. It quotes a refund policy that doesn’t exist. It reassures a patient that a medication is safe with their prescription, a sentence no record supports. The dangerous part isn’t that the model is unsure. It’s that it sounds exactly as certain when it’s wrong as when it’s right.

The most dangerous AI hallucination isn’t the one that sounds crazy.
It’s the one that sounds right.

This isn’t hypothetical. In 2024, a tribunal ordered Air Canada to compensate a passenger after its chatbot invented a bereavement-refund policy that didn’t exist. The airline’s defense, that the chatbot was “a separate legal entity responsible for its own actions,” was rejected outright. You own what your bot says. That’s the direction regulators and courts are moving too: a bot’s output is treated as the company’s own statement, not the vendor’s. It isn’t only live chatbots. In 2025, Deloitte Australia refunded part of the fee on a roughly AU$440,000 government report after reviewers found fabricated citations and a court quote that never existed. Confident fabrication reaches real deliverables, and the people relying on them rarely catch it in time.

In a consumer chatbot, a hallucination is an annoyance. In a secured, high-stakes agent (a bank, a hospital, an insurer) it’s a different category of event. A fabricated financial commitment. A wrong dosage instruction. A compliance breach. A wrong number on a real customer’s real account. One confident false statement is a lawsuit, a regulatory incident, or a patient-safety event.

The Hallucination You Never Find Is the One That Hurts You

Here’s the uncomfortable part of Maya’s story. Nothing failed loudly. No exception, no timeout, no low-confidence warning.

A hallucination that is obviously wrong is almost harmless. The user shrugs it off. The dangerous ones are the opposite: plausible, specific, confident, and attached to a number that matters. Those get acted on. Because they look identical to a correct answer, no one flags them. Not the user, and not a basic monitor.

So the real target isn’t “catch obvious nonsense.” It’s catch the silent ones. The false statements that sail through with a perfect surface, where the only signal that something was wrong arrives later, as real-world damage, in a statement or a chart or a court filing. A simple detector misses exactly these, because a simple detector is also judging plausibility, and the hallucination looks perfectly plausible.

The whole reason detection exists is to find the errors you would otherwise only discover after they’ve already cost someone.

Has the Field Actually Solved AI Hallucinations?

Not yet. If it were solved, the numbers would say so. They don’t.

0.8 to 15%. On Vectara’s grounded-summarization benchmark, the best models hallucinate around 1 to 2% even with the document right in front of them, and weaker ones run to ~15%. None are at zero. This is the easy case: one document, handed to the model, nothing to retrieve. (Vectara Hallucination Leaderboard)

Up to 45.9%. How much a detector’s measured accuracy falls when you grade it against human judgment instead of the lenient automatic metrics the field had been using. Much of the reported progress in hallucination detection is a scoring artifact, not real improvement. (“The Illusion of Progress,” 2025)

The takeaway isn’t that models are hopeless. It’s that most organizations don’t know how to find their hallucinations, and the ones who think a single check has it covered are the most exposed.

What Can an AI Agent Hallucinate Against?

Detection isn’t one problem. It depends on what the claim was supposed to be grounded in:

Against its system prompt. The agent asserts a capability, policy, or behavior that contradicts its own instructions. Claims it can do something it was told it can’t, or invents a rule it was never given.

Against tool outputs. The API returned X. The agent says Y. Or worse, it computes on the data and gets it wrong (see Maya’s $40).

Against knowledge-base items. The agent states something the KB simply doesn’t support.

The harder cousin: the KB refutes the claim. Not “no source mentions this,” but “a source says the opposite.” The KB says check-in is 15:00, the agent says noon. That’s not unsupported, it’s refuted, and must be caught and labeled differently.

Most teams build for one of these and assume the rest come for free. They don’t.

Can One Prompt Catch AI Hallucinations?

Here’s the pitch a lot of companies are running on. Take the conversation, hand it to an LLM, say “find the hallucinations,” then ship. It demos beautifully. It catches the obvious cases. Everyone declares victory. Then it meets production, and every hidden assumption in that one prompt breaks at once.

Here’s the shape of the problem. A one-prompt detector confuses missing evidence with a fabrication. It flags true statements when the retriever surfaces the wrong chunk. It rubber-stamps derived numbers because the checker does the same bad math the agent did. It marks a hallucinated claim from turn two as a “fact” and lets it launder itself in turn six. When the agent invents an account balance without calling any tool at all, a “does a source contradict this?” check finds nothing to contradict, so it stays quiet.

The detector itself is high-variance, too. In our own testing, a one-line, cosmetic wording change to the classifier prompt swung precision from 0.97 to 0.52, with false positives going from a handful to over a hundred on the same conversations. You can’t tune this by intuition. Every change has to be gated against a real benchmark, or you ship a regression wearing an improvement’s clothes.

It All Has to Run on the Cheapest Model You Own

Everything above is the accuracy problem. Now stack the economics on top, because this is where most projects quietly die.

Detection isn’t run once, on one conversation. It runs on every conversation, continuously, thousands at a time. Monitoring a production agent means re-reading, extracting, retrieving, and judging across an entire firehose of live traffic, in near real time. The unit economics aren’t “cost per check.” They’re “cost per check × every conversation your agent ever has.”

A lot of teams prototype this with their biggest model. Sonnet, Opus, frontier GPT-class models. They get a beautiful number on ten hand-picked conversations and declare victory. Then they point it at thousands of live conversations at once and the wheels come off. Per-conversation cost multiplies into a bill nobody will sign, and latency turns real-time monitoring into next-day batch.

A detector that only works on a frontier model isn’t a product. It’s a demo with an invoice attached.

So the real constraint is brutal. You have to solve the hardest version of the accuracy problem, buried facts, cross-entity grounding, derived-value errors, fuzzy contradiction labels, while running on the smallest, cheapest models available. Nano-tier models. Distilled small models. Low-cost open-source models. At production scale, the frontier model is the one tool you can’t reach for.

What Moves the Needle

“Find the hallucinations” is one line. Making it true, in a bank or a hospital, across every conversation, on a model that costs almost nothing to run, is the entire job. Maya’s $40 was the cheap version of finding out.

Most tools in this space ask whether the model is performing: LLM-as-judge scoring, prompt benchmarks, trace inspection. Useful, but they grade one answer at a time, and a single LLM checking a single answer inherits the same blind spots as the agent it’s checking. Catching the silent ones is a system problem. It needs checks that run on every conversation, that tell a real fabrication apart from a data-capture gap, and that the people who own the business outcome can calibrate themselves without filing an engineering ticket.

This is the problem Avon is built for. Detection this hard shouldn’t be something your engineers build from scratch and then babysit forever, so Avon is built around three moves the people who own the business outcome can run themselves:

• Watch every conversation. Continuous monitoring on cheap models, not spot checks by a frontier model on ten hand-picked chats.
• Cut through the false positives. Alerts worth acting on, not the swamp of noise that makes monitoring unusable at scale.
• Put the controls with the business. Rules, thresholds, and guardrails that a compliance, ops, or product owner can change on their own. No engineering ticket required.

Signal, not noise, and no standing engineering tax on the team shipping the agent.

The point is to catch it before Maya does.

Key Takeaways

• The dangerous hallucinations aren’t the obvious ones. They’re the confident, specific, plausible statements that look exactly like correct answers, and they get acted on before anyone checks.
• An agent can hallucinate against four different things: its system prompt, tool outputs, what the knowledge base supports, and what the knowledge base directly contradicts. Building for one doesn’t cover the rest.
• One LLM prompt isn’t enough. It catches the obvious cases, then breaks in production on buried evidence, bad math, and fabrications with no source to contradict.
• Detection runs on every conversation, so it has to run cheap. A detector that only works on a frontier model is a demo with an invoice attached.
• The most exposed teams are the ones who think a single check has it covered.

Frequently Asked Questions

See It in Action

Avon watches live agents for the silent ones, on every conversation, and puts the controls with the people who own the outcome, not only the engineers who shipped the agent.

Coming next: the ways naive hallucination detection breaks in production, and what a detector actually needs to do to catch the silent ones.

Sources

• Air Canada chatbot liability: Moffatt v. Air Canada, BC Civil Resolution Tribunal (Feb 2024); coverage in Forbes
• Deloitte Australia hallucinated-report refund: Fortune (2025)
• Retail bank FX spreads (~2–4%): World Bank Remittance Prices Worldwide; Wise pricing analyses
• Model hallucination rates: Vectara Hallucination Leaderboard (grounded-summarization benchmark)
• Detection over-reported: “The Illusion of Progress: Re-evaluating Hallucination Detection in LLMs,” 2025 (arXiv:2508.08285)