Output failure
Hallucination in LLMs
False, unsupported, fabricated, or ungrounded information produced confidently by an AI system.
Definition
Hallucination is a failure mode where an AI system produces information that is false, unsupported, fabricated, or not grounded in the available evidence. In enterprise systems, hallucination can appear as invented facts, incorrect summaries, fabricated citations, unsupported recommendations, or confident answers that contradict source material.
Why it matters
Hallucination can damage customer trust, mislead employees, create compliance risk, and cause downstream workflow errors. The risk is highest when users assume the system is grounded in internal knowledge or when outputs are used to make business, legal, financial, medical, or operational decisions.
Where it appears
Customer support assistants, enterprise search, knowledge-base copilots, sales enablement tools, legal or policy summarizers, analyst workflows, and RAG systems.
Symptoms
- The answer contains facts not present in retrieved sources.
- The model invents a policy, date, metric, quote, customer, or citation.
- The response sounds confident but cannot be verified.
- The answer contradicts the source material.
- The system fills gaps instead of asking for clarification.
Detection signals
- Low source overlap between answer and retrieved context.
- Claims without citations or evidence.
- Contradictions between output and source passages.
- User corrections or negative feedback.
- Repeated unsupported claims in the same workflow.
Example scenario
A support assistant is asked whether a customer is eligible for a refund. The retrieved policy does not cover the customer case, but the assistant confidently states that the customer qualifies and invents a 30-day exception that does not exist.
Severity scoring
Low
Harmless unsupported detail with no user impact.
Medium
Incorrect answer that creates user confusion or internal rework.
High
Unsupported recommendation that affects customer, legal, financial, compliance, or operational outcomes.
Critical
Hallucination triggers an automated action or causes material harm.
Eval strategy
Create test cases where the answer must be grounded in specific source documents. Include negative cases where the correct behavior is to say that the information is unavailable. Evaluate whether the model distinguishes known facts from unsupported assumptions.
Runtime monitoring strategy
Monitor answer-source alignment, citation quality, source coverage, contradiction signals, and user correction patterns. Track hallucination rates by workflow, model version, prompt version, retrieval source, and customer segment.
Mitigation strategies
- Require grounded citations for factual answers.
- Add refusal or clarification behavior when evidence is missing.
- Improve retrieval quality and source ranking.
- Use claim-level verification for high-risk outputs.
- Add human review for severe or uncertain cases.
- Regression-test hallucination-prone workflows after model or prompt changes.
Where FailureModes.ai fits
FailureModes.ai helps teams detect hallucination patterns, classify severity, connect failures to retrieval or prompt causes, and turn recurring hallucinations into evals and production monitors.
Related
Continue exploring.
- →
Retrieval Failure
When an AI system retrieves stale, irrelevant, incomplete, conflicting, or poorly ranked context — often the root cause of bad RAG answers.
- →
Context Drift
Gradual loss or distortion of important task context as a conversation or workflow progresses.
- →
Evaluation Blind Spot
When an AI system passes the tests a team has built but still fails in production because the eval suite missed the relevant scenario.
- →
Model Regression
When an AI system performs worse after a model, prompt, retrieval, tool, policy, or orchestration change.
- →
Schema Violation
Outputs that don't match a required format, contract, or structure — malformed JSON, bad fields, invalid tool arguments.