Serving LLMs Efficiently: A Hands-On Journey from Single-GPU Basics to Distributed Inference

ODSC AI West 2026

Oct 27, 2026 · Time to be announced

Serving a large language model is not like serving a normal ML model. The model writes one word at a time, and every word means re-reading gigabytes of weights and a growing memory cache from the GPU. That is why LLMs feel slow and expensive — and why simply adding more GPUs often makes things worse instead of better. This hands-on workshop takes you from the ground up. You do not need prior serving experience. We start with the simple ideas that explain everything else: why inference is limited by GPU memory bandwidth, and how to calculate the memory and speed of any model on the back of a napkin. From there we get practical. On a single GPU (a free Colab or cloud instance is enough), you will make a model faster and cheaper yourself: shrink its memory cache, turn on smart caching and quantization with vLLM and SGLang, and measure the speed-up. Then we scale up. Through guided, pre-built demos you will see how models are split across many GPUs, and how the largest systems separate the "prefill" and "decode" stages across machines using NVIDIA Dynamo and llm-d on Kubernetes. Whether you are new to LLM serving or already run it in production, you will leave able to size a deployment, cut its cost, and understand exactly how inference scales from one GPU to a cluster. All notebooks and code are open-source on GitHub.

Session status: scheduled

Official session / agenda

Saving a session is not registration. Check the official agenda for availability, eligibility and any additional fees.

Speakers and their event affiliations

Parent event

Hybrid · Hyatt Regency San Francisco Airport, Burlingame, California, United States

This describes the parent event. Confirm the session’s attendance requirements with the organizer.

Companies participating in the parent event

These are event-level relationships, not confirmed sponsorship of this session.

Showing 24 of 90 company relationships. View all on the event page.