The Anatomy of an LLM inference Engine

ODSC AI West 2026

Oct 27, 2026 · Time to be announced

Most of us treat inference engines like a black box: you send a request to your favorite LLM and you get tokens back. This talk walks through every single component of LLM inference. We'll follow one request end to end, from the HTTP POST to the streamed output tokens, and see what the engine is actually doing at each step. Tokenization and the input pipeline. How the scheduler decides what runs next, and how continuous batching lets a single forward pass serve hundreds of users at once. How chunked prefill keeps one giant prompt from ruining latency for everyone else. How the KV cache is managed like virtual memory with PagedAttention, and how prefix caching quietly saves you work you didn't know you were repeating. Then sampling and detokenization on the way back out. An inference engine really is just a big machine for keeping the GPU busy by orchestrating every single calculation involved in running a LLM and in this talk we'll cover everything. Aimed at AI engineers who use these systems and want a real mental model of what's happening underneath. No prior knowledge of engine internals needed.

Session status: scheduled

Official session / agenda

Saving a session is not registration. Check the official agenda for availability, eligibility and any additional fees.

Speakers and their event affiliations

Parent event

Hybrid · Hyatt Regency San Francisco Airport, Burlingame, California, United States

This describes the parent event. Confirm the session’s attendance requirements with the organizer.

Companies participating in the parent event

These are event-level relationships, not confirmed sponsorship of this session.

Showing 24 of 90 company relationships. View all on the event page.