How to Stop Your AI Chatbot from Hallucinating Wrong Answers

In February 2024 the British Columbia Civil Resolution Tribunal decided a case that every business running a support chatbot should read. Jake Moffatt asked Air Canada's chatbot about bereavement fares after his grandmother died. The bot told him he could book at full price and apply for the discount retroactively within 90 days. Air Canada's actual policy, published on the same website, said the opposite.

Air Canada argued that it could not be held liable for what the chatbot said, and, in a line the tribunal quoted with visible incredulity, suggested the chatbot was effectively responsible for its own statements. The tribunal disagreed. Moffatt v. Air Canada found the airline liable for negligent misrepresentation on the grounds that it had not taken reasonable care to ensure the chatbot was accurate. The damages were small, $812.02 including fees. The precedent was not.

The useful lesson is not "chatbots are risky". It is that a fabricated answer is a business statement, and that preventing fabrication is an engineering and editorial responsibility with known controls. This guide covers what those controls are, in rough order of how much they actually move the number.

Two Different Failures Share One Word

"Hallucination" gets applied to two mechanically distinct faults, and confusing them wastes a lot of effort.

Closed-book fabrication is the model answering from its own parameters because nothing was retrieved. Asked about your refund window with no relevant document in hand, it produces a plausible refund window, because plausible text is what it was trained to produce. This is the failure most people picture.

Unfaithful generation is subtler and, in production support bots, more common. The right passage was retrieved and the model still said something the passage does not support: it merged two policies, carried a number over from the wrong row, softened a hard condition, or inferred a step that is not written anywhere. The evidence was on the desk and the answer still drifted from it.

The two need different fixes. Closed-book fabrication is solved by retrieval coverage and refusal behaviour. Unfaithful generation is solved by citation enforcement, verification, and prompt constraints. A team that only addresses the first will keep shipping wrong answers with correct sources attached.

Why Models Guess Instead of Abstaining

The most useful explanation of why models fabricate at all came from OpenAI researchers Adam Tauman Kalai and Ofir Nachum in Why Language Models Hallucinate, published September 2025 and later carried in Nature. Their argument is not about architecture. It is about scoring.

Most benchmarks grade on a binary: the answer is right or it is wrong. Under that scheme, saying "I do not know" earns exactly the same score as a wrong answer, which is zero. A model that always guesses when uncertain therefore outperforms a model that abstains when uncertain, and training pipelines optimised against those benchmarks select for the guesser. The paper describes this as an epidemic of penalising uncertainty, and proposes a socio-technical fix: change the scoring so confident errors cost more than admitted uncertainty.

The practical consequence for anyone deploying a bot is blunt. The default behaviour of a language model is to produce an answer, not to decline. Silence is a behaviour you have to explicitly design, prompt for, and reward. It will not arrive on its own.

The Long Tail Is Where Your Business Lives

The second structural fact is that hallucination rate is not a single number for a model. It varies enormously with how often the fact in question appeared in training data.

Read that chart with your own knowledge base in mind. Widely repeated facts, the capital of France, how HTTP works, sit in the 1 to 3 percent band. Your shipping cutoff for Tuesday orders, your enterprise tier's seat minimum, the exact wording of your cancellation clause: those are long-tail facts. They appear in training data approximately never. Left ungrounded, that is a 15 to 40 percent fabrication band, and anything that changed after the model's training cutoff is worse still.

This is why the argument "we will just wait for a better model" fails. Frontier models have genuinely improved on grounded summarisation, from a 3 to 8 percent hallucination range in 2023 to roughly 1 to 2.5 percent on the Vectara HHEM leaderboard in 2026. That improvement is real and it is irrelevant to your refund policy, because no model was ever going to know your refund policy.

It is also worth knowing that those headline numbers describe an easier task than yours. When Vectara rebuilt the leaderboard on 7,700 enterprise-length articles instead of short documents, measured rates rose across every model tested. Longer and messier source material is harder to stay faithful to, and real support documentation is long and messy.

Three Benchmarks, Three Different Questions

Vendors quote hallucination numbers without saying which measurement they mean. The three in common use answer different questions, and only one of them describes a support bot.

Figures compiled from Presenc AI's 2026 hallucination benchmark survey. The number to internalise is the middle row. Even with good retrieval and a frontier model, roughly one answer in twenty contains an unsupported claim before you add any controls of your own. Everything below is about pushing that down and catching what gets through.

Control 1: Ground Every Answer, and Verify That Grounding Happened

Retrieval-augmented generation is the baseline, not an optimisation. If you are unfamiliar with the mechanism, the primer on how retrieval-augmented generation works covers it properly. The short version is that the bot searches your documents first and answers only from what comes back.

What teams get wrong is treating RAG as a binary that is either on or off. It is a pipeline with a failure point at every stage, and a retrieval miss produces exactly the conditions for closed-book fabrication: a question, no evidence, and a model whose training rewards answering anyway.

Two checks are worth wiring in before anything else:

• Log the retrieved passages alongside every answer. Without this you cannot tell a retrieval failure from a generation failure, and the two have opposite fixes.
• Measure retrieval separately. For a fixed set of test questions, was the correct passage in the top five results? Precision at five below roughly 0.8 means your hallucination problem is a training data problem and no amount of prompt work will fix it.

Control 2: Make "I Do Not Know" a Legitimate Output

Given the scoring argument above, abstention has to be built rather than assumed. Three things make it real.

Say it in the system prompt, specifically. Vague instructions such as "be accurate" do nothing. Something closer to this does:

Answer only using the provided context. If the context does not contain the answer, say that you do not have that information and offer to connect the customer with the team. Never infer policy details, prices, dates, or eligibility rules that are not written in the context.

Enumerate the categories that always abstain. Pricing not in the corpus, legal or medical advice, promises about future releases, anything involving an individual account, competitor claims. A list is enforceable; a principle is not.

Give abstention somewhere to go. A bot that says "I do not know" and stops is a dead end that annoys customers and hides gaps from you. A bot that says "I do not know, let me get someone" is a working escalation path. This is the practical value of human handover in AI chat: it converts the safest possible answer into an acceptable customer experience, which is what makes teams willing to enable it.

Control 3: Enforce Citations

Requiring the model to attach the source passage for each claim does two things at once. It gives the customer something to verify against, and it constrains generation, because a model instructed to cite has to locate support before asserting.

Recent work in high-stakes domains supports the second effect. A 2026 study on citation enforced prompting in medical RAG systems found that mandatory evidence grounding reduced unsupported claims relative to standard RAG prompting, and inference-stage interventions of this kind are among the more promising mitigations in the current literature precisely because they need no retraining.

Two implementation notes. Citations must be checkable, meaning a link or document reference the customer can actually open, not a bare filename. And an uncited sentence in an answer that was supposed to be cited is a signal you can act on automatically, which leads directly to the next control.

Control 4: Set a Confidence Floor and Abstain Below It

If the best retrieved passage is only weakly similar to the question, the honest read is that you do not have the answer. Answering anyway is where most production hallucinations originate.

A confidence-aware pipeline layers three cheap checks, an architecture Microsoft's Azure engineering team described in a 2026 write-up on confidence-aware RAG:

1. Retrieval confidence scoring. If the top result's similarity falls below a threshold, abstain before generating.
2. Citation validation. After generation, check that each cited passage actually contains the claim attributed to it.
3. Model-based abstention. Ask the model, in a separate call, whether the context genuinely supports the answer it just produced.

The honest caveat is that these thresholds are not portable. Published work on hallucination detection notes that confidence scores are poorly calibrated across datasets, so a cutoff of 0.5 might give 90 percent recall on one corpus and 60 percent on another. Tune the threshold against your own questions, and expect to retune it after a significant content change.

Control 5: Narrow the Scope Before You Widen the Prompt

The instinct when a bot answers badly is to add instructions. Usually the better move is to subtract territory.

A bot scoped to answer questions about your product, billing, and shipping will refuse a question about tax treatment. A bot scoped to "help customers" will attempt it. Scope is the cheapest hallucination control available, because a question that never reaches generation cannot be fabricated. The related discipline of writing responses that fail gracefully is largely about deciding in advance which questions are out of bounds and what happens to them.

Two scope decisions worth making explicitly:

• Never quote a number that is not in the retrieved context. Prices, limits, dates, and durations are where wrong answers become expensive, and they are exactly the details a model will confidently interpolate.
• Never state a policy in the customer's favour that is not written down. This is the Air Canada failure mode precisely, and it is the one with legal consequences.

Control 6: Verify After Generation

Everything above operates before or during generation. A second pass afterwards catches what slipped through.

The mechanism is straightforward: take the generated answer and the retrieved context, and score how much of the answer is actually supported. Academic work formalises this as an unsupported sentence ratio, computed by decomposing the answer into claims and checking each against the source. Deterministic implementations use character-level n-gram overlap and cosine similarity; model-based implementations ask a second model to judge.

For a business deployment the useful version is cheaper than either. Sample fifty answers a week, mark each sentence as supported, unsupported, or partially supported, and track the unsupported share over time. It is a manual metric and it will tell you more than any dashboard, because you will read the failures as you count them.

What the Controls Cost

Not every control is worth it for every deployment. Roughly, in order of return on effort:

The last row is deliberately last. Model choice matters at the margins, and on your own long-tail content it is close to noise compared with grounding and abstention.

The Trap: Optimising Refusals to Zero

There is a failure mode on the other side, and it is easy to walk into after a bad week. A team gets burned by a wrong answer, tightens the confidence threshold, broadens the refusal list, and ends up with a bot that declines half the questions it could have answered correctly. Customers learn within two interactions that it is useless, and traffic collapses to the contact form.

The metric to watch is not refusal rate on its own. It is the pair: unsupported claim rate and unnecessary refusal rate, where an unnecessary refusal is one where the answer was in the corpus and retrieval or the threshold missed it. Almost every unnecessary refusal is a retrieval problem masquerading as a safety setting, and the fix is upstream in the content, not in the threshold.

Keep a running list of refused questions. It is simultaneously your content backlog and your calibration data, and it is the single most valuable byproduct of running a bot that is willing to say no.

Where the Platform Fits

Most of what is above is design and editorial work that no vendor removes. What a platform can remove is the plumbing: retrieval that is grounded in your own sources by default, scheduled recrawls so answers do not drift from published content, an instruction layer where scope and abstention behaviour are configured rather than coded, handover so a refusal becomes a conversation with a person, and transcripts you can actually sample for the weekly verification pass. Paperchat is built around that division of labour, which leaves the two things that genuinely cannot be outsourced: deciding what your bot is allowed to talk about, and keeping the source documents true.

The Bottom Line

Hallucination in a business chatbot is not an unpredictable glitch. It is the expected output of a system that was trained to always answer, pointed at facts it was never taught, with no instruction about when to stop. Each of those three is addressable.

Ground every answer in retrieved content and log what was retrieved. Instruct abstention explicitly and give it an escalation path so refusing is not a dead end. Enforce citations, set a confidence floor, and narrow the scope until the bot is only speaking where you have evidence. Then sample real answers weekly and count the unsupported sentences, because the alternative is finding out the way Air Canada did.

The goal is not a bot that never says anything wrong. It is a bot whose wrong answers are rare, detectable, and cheap, and whose uncertainty is visible instead of dressed up as fact.