Don't Merge That: An Agentic Approach to Catching Outages at 10,000 PRs a Week

AGNTCon + MCPCon North America 2026

Oct 22, 2026 · 10:20–10:45 · America/Los_Angeles

At Datadog, we ship more than 10,000 pull requests a week. At that scale, human reviewers can't catch every ticking time bomb — the missing timeout that causes a cascading failure, the silent behavioral change that breaks every caller at once, the cold cache that thundering-herds a dependency into the ground. In this talk, we share how we built an agentic review system that catches these patterns automatically, before merge, on every high-risk PR. You'll learn how we combined risk scoring with fine-tuned models, tool-augmented LLM agents, and reflection loops to go from "this looks fine" to "here's the exact line that will page you at 3 AM — and here's the fix." We'll walk through the hard lessons: what makes an LLM a good reliability reviewer, how to keep it focused on outages and not style nits, how reflection loops measurably improve flaws catching quality, and how to wire all of this into a production system your engineers will actually trust. Whether you're building agentic tooling or just trying to scale code review, you'll leave with concrete patterns for putting LLMs on the critical path of your engineering workflow — without losing your oncall's sleep.

Session status: scheduled

Room: LL21 ABC

Track: Building Reliable Agent Systems

Level: Intermediate

Official session / agenda

Saving a session is not registration. Check the official agenda for availability, eligibility and any additional fees.

Speakers and their event affiliations

Parent event

In person · San Jose McEnery Convention Center, San Jose, California, United States

This describes the parent event. Confirm the session’s attendance requirements with the organizer.

Companies participating in the parent event

These are event-level relationships, not confirmed sponsorship of this session.

Showing 24 of 512 company relationships. View all on the event page.