This is our last lesson. By this point, you have implemented attention from scratch, built KV cache and extended across requests with prefix caching. to build an optimized diffusion pipeline. You've probably noticed a common thread running through the course. Find a redundant computation and cut it. This lesson is about where the field is taking these ideas next and how the foundation you built in this course gives you the tools to go there too. Lesson one through four give you a toolkit of optimizations that you can apply right away. This final lesson is a little different. You're going to look at five open problems in the field and how you can use open source framework you've learned about in this course to tackle them. You probably noticed during the course that each different type of input and output, like text and images has its own inference stack. In production, that means an application handling text images, and audio would need three separate systems running in parallel, which also means three GPU allocations, three scheduling systems, and three potential sets of bugs. The field is moving toward a single unified engine for all input types. A coding agent that reads a text description looks at a screenshot of a UI and generates code. Right now, that takes three separate inference calls routed to three separate systems. But with a Unified Engine, it's one call. same runtime, same scheduler, same memory pool. Everything you built in lesson two through four using PyTorch under the hood, because it's the most widely used framework for training and serving models. And it has the richest ecosystem with the most straightforward path from research to deployment. Google's framework JAX. It compiles code automatically, runs natively on TPUs, and is built around a functional programming style that a lot of researchers prefer. Much of the cutting-edge work coming out of Google DeepMind happens in JAX. People in this field have been actively working on the serving infrastructure for JAX. and SGLang is part of it. SGLang JAX is a port of the SGLang optimization stack that applies the same techniques from this course. RadixAttention, prefix caching, scheduling, and memory management. So a model training in JAX can now be deployed without having to rewrite it first. This means JAX is now a viable option if you want faster compilation or TPU access. without giving up the serving infrastructure you already been working with. Frontier 3: Reinforcement learning, or RL from human feedback is how a raw language model like ChatGPT gets shaped into something useful. But in RL, inference shows up somewhere unexpected. inside training. The RL Training Loop has three steps. Generate responses, collect reward scores, and update the weights and repeat for millions of times. So notice that for step one, generating responses is actually inference. That step alone takes 50 to 80% of total RL training time. The weight update The part traditionally thought of as training is actually pretty fast. The bottleneck is text generation. Current tools are painfully slow at the inference step. SGLang team has released Miles, a framework for RL-optimized serving. which also applies KV cache management and prefix caching to the rollout step, the same techniques from lesson two and three. But in a training context, instead of a serving one. Miles collects reward in parallel, so the training loop doesn't stall while waiting on evaluations. This is an active area of developing across the field. Many open source communities and frameworks listed here are all working on the same problem. We believe that training runs that used to take weeks will be now done in just days. Agentic workflows are one of the most active areas in AI development right now. The pattern is simple. A user gives a high-level instruction, analyze this data set or fix this bug. and an AI agent breaks it down into steps, writes code, executes it, reads the output, and iterates. But this creates a fundamentally different inference pattern than what you've been optimizing for. In a standard LLM call, you decode once and you're done. In an agentic workflow, the model decodes, generating code or tool call as output, then pauses while the tool executes. Then it resumes based on the result. This can happen dozens of times in a single interaction. That creates new optimization problems. The KV cache has to be preserved across pauses. So the model's context from before the tool call needs to be available when generation resumes. The GPU sits idle during tool execution, but other requests could be using it. And the context keeps growing because every tool call adds to the conversation history. Traditional inference engines weren't designed for this stop and go pattern. They assume a request comes in, generate a response and leave. Agentic inference doesn't work that way, and building serving infrastructure that does is an open problem. New AI chips are launching constantly. Each new iteration has a different architecture, different memory hierarchy, and different programming model. This creates a software problem. Every new chips potentially needs its own kernel implementations and a memory management strategy. If your inference engine was built assuming GPUs, porting to your new chip can mean rewriting significant parts of the stack. SGLang approaches this problem by building optimizations at the right level of abstraction. How you batch requests, manage the KV cache, do prefix caching with RadixTree are algorithmic, not hardware specific. This is the same philosophy behind PyTorch and JAX. Separate the computation from the execution backend. SGLang applies that to the serving layer. If any of the topics covered in this course spark something in you, The codebase is there for you to explore. We'd love you to join our community and be part of forging the future of inference together. Feel free to join the Slack and check out the website of www.sglang.io
