Скачать или смотреть A Deep Dive Into Automated RAG Evaluation with open-rag-eval

Confidence in RAG output is non-negotiable, but manually providing golden chunks and golden answers is slow and subjective. This talk explores `open-rag-eval`, an open-source framework designed to automate RAG evaluation and build trust at scale. Building on pioneering research from the University of Waterloo, this framework integrates innovative tools like UMBRELA for reference-free relevance scoring, AutoNuggetizer for automated fact-checking, and HHEM for hallucination detection. Designed with a flexible connectors architecture, it seamlessly plugs into any RAG pipeline while delivering fast, transparent, and interpretable metrics on retrieval, generation, and hallucination in RAG. Join us for a live demo to discover how `open-rag-eval` can help you enhance RAG response quality, and accelerate your RAG development efforts.

⏱️ Chapters
00:00 – Intro: Why RAG Evaluation Is So Hard
00:50 – The Golden Answers Problem
01:45 – What Is open-rag-eval?
02:50 – Live Demo: Side-by-Side RAG Comparison Dashboard
04:20 – UMBRELA: Reference-Free Relevance Scoring
05:30 – AutoNuggetizer: Atomic Fact Detection Without Labels
06:40 – Metrics: Citation Faithfulness & Hallucination Detection
08:10 – How It Works Without Golden Chunks or Answers
09:10 – Flexible Connectors: LangChain, LlamaIndex & More
10:15 – In-Progress: Response Consistency Metric (Preview)
11:30 – How To Get Started With open-rag-eval
12:30 – Q&A: Chunking Strategies & Multimodal RAG


🔗 More on open-rag-eval: https://github.com/vectara/open-rag-eval
📌 Learn more about Arize community events: https://arize.com/community/