The most famous objection to AI, and rightly so. But it's a known problem with known solutions.
An AI hallucination is an answer that sounds smooth and confident but is factually incorrect: a made-up source, a non-existent function, a prize that comes from nowhere. It arises because a language model generates the most likely continuation instead of looking something up.
IN COMMON LANGUAGE
Why it happens
A language model does not consult a database. It predicts word by word what follows logically based on patterns. When the knowledge is lacking, it still produces a fluent response: because fluency is exactly what it is trained to do.
There is no built-in difference between knowing for certain and finding it plausible. Both come off equally convincing, and that's what makes it dangerous in a business context.
So it is not a malfunction that will disappear with an update. It's inherent to how these models work, and you tackle it with the design around it.
WHY IT MATTERS
How we counter it
The most important measure is to have the agent answer from your documents instead of general knowledge. Then there is something to refer to, and an answer is verifiable.
In addition, we explicitly teach the agent that "I don't know, I'll connect you" is a good answer. Sounds obvious, but without that instruction a model always chooses an answer.
- Answers from our own sources — the most important measure.
- Show source — an answer without origin is not verifiable.
- Allowing not knowing — instruct explicitly, otherwise the model guesses.
- Human approval — in everything with financial or legal consequences.
- Save conversations — so that you can find errors and make adjustments.
- Watch at the start — you just read along for the first few weeks.
FREQUENTLY ASKED QUESTIONS
More about an AI hallucination
Can you rule out hallucinations completely?
No, and anyone who promises that is exaggerating. You can greatly reduce the risk by working with your own sources, and limit the damage through human approval for actions that have consequences. Residual risk remains, and you should inform a customer honestly about this.
Is this a reason not to use AI?
For some applications yes. The consequences of legal or medical advice are too serious. When answering frequently asked customer questions from your own documentation, the risk is small and the return is high. It depends on what goes wrong when it goes wrong.
How do I notice it when it happens?
By saving conversations and reading the first weeks. Then randomly, and with a report button for users. Without a logbook, you only notice it when a customer calls about it.
CONTINUE READING
Of course, continue reading about this subject.
AI with boundaries?
We build agents who refer when they are unsure. Tell us what process you have in mind.