Lifecycle failure
Model Regression
When an AI system performs worse after a model, prompt, retrieval, tool, policy, or orchestration change.
What failed
Model regression occurs when an AI system performs worse after a model, prompt, retrieval, tool, policy, or orchestration change. The system may improve on average while becoming worse on specific workflows, user segments, or failure modes.
Architecture context
Model upgrades, prompt revisions, tool updates, RAG pipeline changes, policy changes, routing changes, and agent orchestration updates.
Impact
LLM and agent systems change frequently. A new model can improve reasoning but increase refusals. A prompt change can reduce hallucination but break structured output. A retrieval update can improve freshness but introduce irrelevant documents. Without regression monitoring, teams may ship changes that silently damage production reliability.
Symptoms
- Quality drops after a deployment.
- Refusal behavior changes unexpectedly.
- Schema violations increase.
- Tool-use accuracy declines.
- Latency or cost changes after routing updates.
- A previously solved failure mode reappears.
Detection signals
- Metric shifts by version.
- Eval failures after deployment.
- Increased user corrections.
- Changed refusal, hallucination, or schema-violation rates.
- Incident clusters tied to release dates.
Mitigations
- Use canary deployments.
- Maintain versioned eval suites.
- Segment metrics by workflow and failure mode.
- Roll back risky changes quickly.
- Use shadow testing before production rollout.
- Require approval for high-risk model changes.