Output failure
Hallucination in LLMs
False, unsupported, fabricated, or ungrounded information produced confidently by an AI system.
What failed
Hallucination is a failure mode where an AI system produces information that is false, unsupported, fabricated, or not grounded in the available evidence. In enterprise systems, hallucination can appear as invented facts, incorrect summaries, fabricated citations, unsupported recommendations, or confident answers that contradict source material.
Architecture context
Customer support assistants, enterprise search, knowledge-base copilots, sales enablement tools, legal or policy summarizers, analyst workflows, and RAG systems.
Impact
Hallucination can damage customer trust, mislead employees, create compliance risk, and cause downstream workflow errors. The risk is highest when users assume the system is grounded in internal knowledge or when outputs are used to make business, legal, financial, medical, or operational decisions.
Symptoms
- The answer contains facts not present in retrieved sources.
- The model invents a policy, date, metric, quote, customer, or citation.
- The response sounds confident but cannot be verified.
- The answer contradicts the source material.
- The system fills gaps instead of asking for clarification.
Detection signals
- Low source overlap between answer and retrieved context.
- Claims without citations or evidence.
- Contradictions between output and source passages.
- User corrections or negative feedback.
- Repeated unsupported claims in the same workflow.
Mitigations
- Require grounded citations for factual answers.
- Add refusal or clarification behavior when evidence is missing.
- Improve retrieval quality and source ranking.
- Use claim-level verification for high-risk outputs.
- Add human review for severe or uncertain cases.
- Regression-test hallucination-prone workflows after model or prompt changes.