Stop Vibe-Testing: Run Real Agent Evals

AGNTCon + MCPCon North America 2026

Oct 22, 2026 · 10:20–10:45 · America/Los_Angeles

Most teams ship AI agents the same way: run a few prompts, eyeball the output, decide it "looks good," ship it. That isn't testing, that's vibes. Vibes don't catch regressions, don't survive a prompt change, and don't tell you whether a new model is actually better. Agents are too complicated and unpredictable for vibes-based testing to work. In this talk I walk you through what real agent evaluation looks like: what evals are, why they're necessary, what separates a useful eval from one that just produces noise, and how to use eval results to actually ship better agents. You'll leave with a concrete framework for closing the loop: instrument, trace, evaluate, validate the evaluator, iterate.

Session status: scheduled

Room: 210 CG

Track: Evals & Testing

Official session / agenda

Saving a session is not registration. Check the official agenda for availability, eligibility and any additional fees.

Speakers and their event affiliations

• Laurie Voss

  Affiliation at this event: Arize AI

Parent event

In person · San Jose McEnery Convention Center, San Jose, California, United States

This describes the parent event. Confirm the session’s attendance requirements with the organizer.

Companies participating in the parent event

These are event-level relationships, not confirmed sponsorship of this session.

• 10ki · Participating company Source
• 33 Creative Partners · Participating company Source
• 451 Research (S&P Global) · Participating company Source
• A Place for Mom · Participating company Source
• A Place for Mom · Speaker company Source
• A.I. Hero, Inc. · Participating company Source
• Abbvie · Participating company Source
• AccessTM · Participating company Source
• ACS · Participating company Source
• Action State Group · Participating company Source
• Action State Group, Inc. · Speaker company Source
• ActiveState · Participating company Source
• Adaptive Security · Speaker company Source
• Adobe · Participating company Source
• Adobe · Speaker company Source
• Aduna Global · Participating company Source
• Adyen · Participating company Source
• Adyen · Speaker company Source
• Aembit · Participating company Source
• Aembit · Sponsor Source
• Aerospike · Participating company Source
• Aerospike · Speaker company Source
• Aerospike · Sponsor Source
• AgentCat · Sponsor Source

Showing 24 of 512 company relationships. View all on the event page.