Misinformation and overreliance in AI features
Models can be confidently wrong. The danger grows when your product makes wrong answers look trustworthy, or when people and systems act on them without checking.
In 60 seconds
Misinformation and overreliance in AI features
Models can be confidently wrong. The danger grows when your product makes wrong answers look trustworthy, or when people and systems act on them without checking.
Where it causes harm
| Feature | What goes wrong | Harm |
|---|---|---|
| Customer support bot | Invents a refund or discount policy | Angry customers, complaints, cost and possible legal trouble |
| HR policy assistant | States the wrong leave or notice rules | Employees make decisions on false information |
| Coding assistant | Suggests a library that does not exist, or insecure code | Supply-chain attack, vulnerabilities in production |
| Security or IT helper | Summarises an alert as harmless when it is not | A real incident is missed |
| Health or finance information | Confident but wrong guidance | Real harm to people; needs professional review |
An example: package hallucination
- Check that a suggested package exists, is the well-known one, and has a real history and maintainers before installing.
- Watch for names that are close to popular packages but slightly different.
- Use lock files, a private mirror or an allow-list of approved packages for important projects.
- Run dependency scanning in your pipeline (Unit 4).
Design for doubt
- Every answer looks equally certain
- No sources, or sources that cannot be opened
- A tiny disclaimer at the bottom of the page
- No way to reach a human
- The bot says "I could not find this in our documents" when retrieval finds nothing
- Clickable citations next to the claims they support
- A short, clear note next to high-stakes answers
- An easy path to a human, and a feedback button
Match review to the stakes
- 1
Classify
Sort the feature's outputs by what happens if they are wrong. - 2
Low stakes
Drafts, suggestions, search help: automatic, with sources. - 3
Medium stakes
Policy answers, summaries used for work: sources required, clear warning, easy escalation. - 4
High stakes
Money, health, legal, access changes, security actions: a qualified human reviews before anything happens.
Watch and read more
Lab
A redesigned answer screen and a stakes table for one AI feature, plus a checked package list.
The problem
Part A (pen and paper): a shop's support bot answered "Yes, you can return any item within 30 days for a full refund" with no source. The real policy is 7 days.
- Sketch a better answer screen: where the source appears, what the bot says when it is not sure, and how the user reaches a human.
- Make a table of five question types this bot gets, with a stakes level (low / medium / high) and the review rule for each.
Part B (browser only, no installing): ask a chatbot for three Python libraries for an unusual task. For each name, check the official package website and write: exists? well-known? maintainers? first release date?
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your bot cites a real policy document, but the answer still misstates it. How could you catch this automatically, and what are the limits?Reveal
You can run a second check that compares each claim in the answer to the cited text, using rules for numbers and dates, or another model as a grader. This catches some errors. But the checker can also be wrong, and it may miss subtle changes of meaning. Use it to flag answers for review and to measure quality over time, not as proof of correctness.
Q2Why might showing a confidence percentage next to each answer do more harm than good?Reveal
Models are not reliably calibrated, so the number may not match real accuracy. A precise-looking number like "92%" makes answers feel more scientific and increases trust. Honest signals, such as whether a source was found and whether the claim is quoted from it, are usually more useful.
Questions people ask
Can we remove hallucinations completely?
Not today. Better models, retrieval and grounding reduce them, but a language model can always produce fluent text that is wrong. Design the feature so that errors are caught before they cause harm.
Is RAG enough to stop misinformation?
It helps a lot, but the model can still misread a document, mix two sources, or fill gaps with guesses. The documents themselves may be old or wrong. Citations let users check, and evaluation tells you how often it goes wrong.
Why is overreliance a security problem, not just a quality one?
Because attackers can use it. They can plant false information for the model to repeat, register hallucinated package names, or rely on staff trusting an AI summary. Any output that people act on without checking is an attack path.
Who decides what is "high stakes"?
The product owner, with security, legal and domain experts. Write the decision down per feature. When in doubt, and especially for health, legal or financial matters, require review by a qualified professional.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.