Understand how LLM inference works token by token, why it gets expensive at scale, and how the KV cache eliminates redundant computation by storing and reusing intermediate values.
Short CourseIntermediate
Efficient Inference with SGLang: Text and Image Generation
Instructor: Richard Chen
Earn an accomplishment with PRO
- All Courses/
- Short Course/
- Efficient Inference with SGLang: Text and Image Generation
Efficient Inference with SGLang: Text and Image Generation
- Intermediate
- 7 Video Lessons
- 3 Code Examples
- 1 Graded AssignmentPRO
- Earn an accomplishment withPRO
- Instructor: Richard Chen
- RadixArk
- Learn more about Membership PRO Plan
What you'll learn
Implement SGLangâs RadixAttention to extend caching across users and requests, and measure the real speedups it delivers.
Apply SGLangâs caching and parallelism strategies to diffusion models, accelerating image generation using the same principles as text.
About this course
Introducing Efficient Inference with SGLang: Text and Image Generation, built in partnership with LMSys and RadixArk, and taught by Richard Chen a Member of Technical Staff at RadixArk.
Running LLMs in production is expensive. Much of that cost comes from redundant computation: every new request forces the model to reprocess the same system prompt and shared context from scratch. SGLang is an open-source inference framework that eliminates that waste by caching computation thatâs already been done and reusing it across future requests.
In this course, youâll build a clear mental model of how inference works (from input tokens to generated output) and learn why the memory bottleneck exists. From there, youâll implement the KV cache from scratch to store and reuse intermediate attention values within a single request. Then youâll go further with RadixAttention, SGLangâs approach to sharing KV cache across requests by identifying common prefixes using a radix tree. Finally, youâll apply these same optimization principles to image generation using diffusion models.
In detail, youâll:
- Build a mental model of LLM inference: how a model processes input tokens, generates output token by token, and where the computational cost accumulates.
- Implement the attention mechanism from scratch and build a KV cache to store and reuse intermediate key-value tensors, cutting redundant computation within a single request.
- Extend caching across requests using SGLangâs RadixAttention, which uses a radix tree to identify shared prefixes across users and skip repeated processing.
- Apply SGLangâs caching strategies to diffusion models for faster image generation, and explore multi-GPU parallelism for further acceleration.
- Survey where the inference field is heading, including emerging techniques and how the optimization principles from this course apply to future developments.
By the end, youâll have hands-on experience with the caching strategies powering todayâs most efficient AI systems and the tools to implement these optimizations in your own models at scale.
Who should join?
Developers and ML practitioners who want to better understand and optimize LLM inference in production. Familiarity with Python and basic language model concepts is recommended.
Course Outline
7 Lessonsă»3 Code Examples- IntroductionVideoă»3 mins
- Overview of InferenceVideoă»10 mins
- LLM Inference FundamentalsVideo with Code Exampleă»11 mins
- Advanced LLM Inference OptimizationVideo with Code Exampleă»18 mins
- SGLang DiffusionVideo with Code Exampleă»19 mins
- The future of inferenceâ where do we go from here?Videoă»6 mins
- ConclusionVideoă»1 min
- Quiz
Gradedă»Quiz
ă»10 mins
Instructor
Additional learning features, such as quizzes and projects, are included with DeepLearning.AI Pro. Explore it today
Want to learn more about Generative AI?
Keep learning with updates on curated AI news, courses, and events, as well as Andrewâs thoughts from DeepLearning.AI!