Case Studies: When AI Goes Wrong
Three critical production failures that taught us everything about building reliable AI systems. Each case study shows the problem, diagnosis, solution, and lessons learned.
Why These Failures Matter
These aren't theoretical examples -- they're real production failures that caused business impact, user confusion, and system downtime. Each case study includes actual Langfuse traces, before/after metrics, and production-ready code solutions.
Retrieval and context issues that lead to dangerous answers
Outdated information and version control problems
Tool validation and agent behavior failures
The Coffee Machine Reimbursement Trap
When Partial Context Creates Dangerous Answers
A user asked, 'Can I expense a new coffee machine for my home office?' The retriever fetched a permissive policy document but completely missed the exclusions list that explicitly forbids kitchen appliances.
- Faithfulness alone is insufficient - high faithfulness with wrong context is dangerous
- Contextual Recall is critical for policy and compliance use cases
- +2 more lessons
The Outdated PTO Policy Nightmare
When Time Becomes Your Enemy
Users consistently received answers based on outdated PTO policies. The current policy existed in the knowledge base but was buried in a different section with poor metadata.
- Temporal metadata is essential for any time-sensitive information
- Hard filters prevent outdated content from reaching users
- +2 more lessons
Agent Tool Hallucination Crisis
When AI Agents Invent Their Own Reality
Production agents were calling non-existent tools or using malformed parameters, causing system crashes and complete task failures.
- Tool validation is non-negotiable for production agent systems
- Circuit breakers prevent cascading failures from tool errors
- +2 more lessons
Go deeper with the course
Master AI evals with hands-on projects, real case studies, and production-ready templates. From failure taxonomy to CI/CD quality gates.