From Intent to Enforcement: Hill-Climbing AI Agents
Date to be announced · Time to be announced
You cannot manage what you cannot measure. For AI agents, evaluation is the same discipline as testing for traditional softwares. It is about how teams manage risk and business outcomes—not just pass a release check—and optimize for safety and quality in a systematic way. In this two-hour, demo-driven session, we will turn business intent into a repeatable eval-fix loop. Using open-source ASSERT, Agent Control Specification (ACS), we will generate targeted tests, trace agent trajectories in OpenTelemetry, ground LLM judge rubrics scoring in your scenario requirements, and choose improvements across deterministic and model-based controls, beyond prompt tuning. We will explore knowledge bases and how retrieval evidence and tool-call traces reveal failures that input/output scoring misses. Then we will hill-climb: compare changes on frozen cases, measure both unsafe behavior and legitimate work preserved, and include cost and latency in the decision. Use Rego for hard rules, classifiers for nuance, and LLM judges for complex reasoning—within a broader governance model of identity, authorization, containment, and audit evidence. Leave with an intent → evaluate → diagnose → fix → remeasure workflow, plus a path from production observations to reviewed regression tests and CI/CD gates as well as promoting that pre-deployment judges to monitoring to detect drifts. The goal: continuously improve the agent while maintaining control over what it is allowed to do. Organizer timezone/event-window conflict: normalized date/time deliberately null. Published local schedule: 10:30–12:30. Timezone requires verification.
Session status: scheduled
Room: MPR B-C
Saving a session is not registration. Check the official agenda for availability, eligibility and any additional fees.
Speakers and their event affiliations
Parent event
In person · Microsoft Toronto, Toronto, Ontario, Canada
This describes the parent event. Confirm the session’s attendance requirements with the organizer.
Companies participating in the parent event
These are event-level relationships, not confirmed sponsorship of this session.