Скачать или смотреть AI Evals with Claude + LangGraph + Tavily Web Search + Arize Phoenix Pt. II 🦅🔥 — AI PM BY DESIGN

Running Evals on ReAct Agents Using Arize Phoenix

Picking up right where we left off last week…

🔗    • AI Evals with Claude LangGraph Tavily Ariz...  

Today, we’re running evals on a ReAct Agent built with:

• Tavily Web Search (real-time info)

• Wikipedia Search (historical facts)

• Current Datetime (dynamic context)

It’s powered by LangGraph and Claude 3.7 Sonnet for reasoning. 🧠

Why This Matters?

Because evaluating AI agents ≠ evaluating LLMs.

We’re not just testing outputs—we’re testing decision-making paths.

Thanks for reading AI PM BY DESIGN! Subscribe for new posts on how to Design Your Career with AI 💯 FREE. Hope you’re ready ~

Evals Ran

✅ Agent Function Calling — Did the Agent select the right tools?

✅ Agent Path Convergence — Did it reason efficiently?

✅ Q&A Retrieval Accuracy — Did it actually answer the question correctly?

💡 Key Concepts

1. Traces = Full journey of a query through your AI system (user → agent → tools → response)

2. Spans = Individual building blocks (LLM calls, tool usage, retrieval steps)

✅ Eval 1: Agent Function Calling

This is where the magic happens.

We pulled span data from the Phoenix server and converted it into a pandas DataFrame for manipulation.

We’re using LLM-as-a-Judge (LLMaaJ) to classify if the agent’s tool usage was appropriate.

Prompt Template Inputs:

User question

Tool definitions

Tool the agent actually used

🎯 Output = Binary (Correct ✅ or Incorrect ❌)

Why binary? Because LLMs struggle with nuance in numeric ranges.

📌 Bonus: Phoenix lets you log annotations and write them back to the platform for future analysis.

🔄 Eval 2: Agent Path Convergence

Here we’re asking:

How efficient is the agent’s reasoning path to the correct answer?

Used a synthetic dataset of semantically similar questions:

Q → “Where was Christopher Nolan born?”

Q → “What is the birthplace of Christopher Nolan?”

Q → “In which city was Christopher Nolan born?” 🦇

Measured path length = sum of all message types (human + system + tool + AI reasoning).

🧪 First run: avg. ~8 steps

🧪 Second run (post-prompt tuning): avg. ~7 steps

Prompt tweak:

Good/bad examples

Clearer step-by-step tool instructions (Tavily + Wiki)

💡 Insight: Prompt quality impacts reasoning efficiency. Even small edits matter.

🎯 Eval 3: Q&A Retrieval (Wikipedia Tool)

Created a trivia-style synthetic dataset:

Capitals

Famous TV characters

Historical facts

Initial results displayed correct answers; having said that, there was label repetition observed in the explanation column.

👀 Worth digging deeper—even when it looks right, annotation explainability matters.

Wrapping Up

Evals like these don’t just show if your agent “works.”

They reveal how and why it’s working (or not).

If you’re serious about agentic apps, “delve” into evals.

You’ll thank me later 🦾

Repo 🔗 https://github.com/scarnyc/agent-eval...

Hey, I’m Will 👋

I'm an AI Product Coach who's helped dozens of clients land AI PM jobs at FAANGs + Fortune 500s by designing their careers with AI. Even in today's Job Market. Every PM needs to Become an AI PM. ↳ PM openings up ~54% from 2023 lows.↳ AI PMs earn a 35% salary premium vs. Traditional PMs (Amazon survey).

AI Product Management is absolutely exploding right now.

DM me — it’s open enrollment.

Hope you're ready!

Thanks for reading AI PM BY DESIGN! Subscribe for new posts on how to Design Your Career with AI 💯 FREE. Hope you’re ready ~