Скачать или смотреть Evals for Product Managers - Part 2 | AppliedAI Club

Join AppliedAI Club for future events: https://www.appliedai.club

---

Part 2 of our deep dive on LLM agent evaluation for product managers.

Key Takeaways:
• Data is the Foundation: Secure high-quality, statistically significant datasets from production logs, domain experts, or synthetic generation
• Human-in-the-Loop (HITL) is the Blueprint: Use HITL to discover an agent's unique failure modes through open coding and axial coding
• Automate with Judge LLMs: Convert identified error patterns into Judge LLMs for scalable evaluation, calibrated against human experts (80-90% correlation)
• Monitor for Drift: Continuously monitor production for performance drift and analyze both output and input changes

Topics covered:
The Data Challenge (quality, diversity, statistical significance)
Human-in-the-Loop Evaluation workflow
Error Analysis Framework (Open Coding & Axial Coding)
Automating with Judge LLMs and calibration
Drift Detection & Fixing techniques
Live demos in Langfuse

CHAPTERS:
00:00 Intro
03:03 Why Evals are Complex
05:23 Data Preparation
12:37 Langfuse Demo
18:28 Human-in-the-Loop
24:50 Judge LLMs
30:39 Custom Evaluators
38:34 Drift Detection
47:54 Fixing Issues
52:25 Q&A

---

Event Details:
Date: 2026-01-24
Presenter: Gautham Muthukumar
Applied AI Products, GenAI, ML & Evals Platforms - Microsoft, Intuit



---

AppliedAI Club brings together AI practitioners to share real-world implementation experiences. Join our community at https://www.appliedai.club

#AppliedAI #AIEngineering #MachineLearning #AI