Every time you send a message to your chatbot or ask a coding agent to fix a bug, there is a moment where you wait. That wait between sending your request and seeing tokens stream back, that's inference. And for most companies, it's the single biggest cost in running AI at scale. This course teaches you how to make it fast, efficient, and scalable using SGLang. Let's start by building a mental model that everything else in the course depends on. Great. Here's our roadmap for the course. Five lessons, each building on the last. The first lesson will give you the mental model of what inference is, why it's expensive, and where SGLang fits in the stack. Lesson two is where you implement attention from scratch. Then optimize it by building KV cache, a foundational optimization that eliminates redundant computation in the generation loop. Lesson three builds on that foundation. You cache across requests. So when two users send the same system prompt, you compute it once instead of twice. Lesson four moves from text to images. You build a complete diffusion pipeline where the same caching principle applies in a different form. Lesson five looks at open problems in the field and where inference optimization is headed. That's the arc. Let's get started. Training is what most people think of when they think of AI compute. To train a model, you'll take a massive data set, feed it through the model billions of times, then adjust the weights through back propagation. This is computationally brutal. it might take weeks on thousands of GPUs. Yet it only needs to be done once. You train a model, you save the weights, and you're done. Inference is the other side of the equation. Inference happens when you take that trained model and you use it. A user asks a question, a model generates an answer. Someone uploads a photo, the model classifies it. Every autocomplete suggestions, every chatbot response, every AI generated image, that's inference. Here's why this matters economically. Training is a fixed cost. You pay it once. But inference scales linearly with usage. 10 times more users means 10 times more inference compute and 10 times the bill. For most AI companies in production today, inference has already overtaken training as the bigger expense. To make this more concrete, think about an AI coding agent, something like Cloud Code or Cursor. During the training, the model learn from billions of line of open-source code, Stack Overflow, debugging sessions, tool used patterns, it learn how experienced developers think. Now that the model has been trained, every interaction becomes an inference call. A user types, 'Fix the failing test in authorization python doc'. The agent doesn't answer in one shot. It reads the file, runs the tests, generates a patch, then runs the test again to verify. That's four or five inference calls for a single task. Multiply that across thousands of developers at the same time, and every millisecond you cut compounds across the entire system. The training cost was large, but fixed the inference cost grows every time you add new users. So what's actually driving that cost? Inference is expensive for three big reasons. Scale. You're serving thousands or even millions of users simultaneously. And each request requires running a full neural network forward pass. This isn't like serving a static web page. the response can't be cached. Speed. Latency matters. A chatbot that takes 5 seconds to respond creates a bad experience for end users. A coding agent that lags behind your typing breaks your flow. Users perceive quality through speed. And complexity. Models keep getting bigger. More parameters means more compute per request. and more memory required to store the weights. Then, there's the hardware reality. Serving large AI models requires massive infrastructure investment. A large model might need expensive GPUs just to fit in memory. And that memory is under constant pressures. Model weights, activations, and the KV cache you will build next lesson are all competing for the same space. On top of that, real world traffic happens in bursts. Most deployments ran at 30 to 40% utilization. So you end up paying for capacity you only use part of the time. So how do you serve AI models faster, cheaper, and at scale? Answering this question starts with a fundamental tradeoff. Two things matter in any inference system. latency. How fast does one user get their response? and throughput. How many requests can you process per second? You want to maximize both, but they're in constant tension. If you optimize purely for latency, you process one request at a time with the full GPU. If you optimize purely for throughput, you are forced to batch as many requests together as possible. Every inference system sits somewhere on this tradeoff curve. The goal isn't to pick a random point to aim for, is to push the whole curve. better latency and better throughput at the same time. That's what KV cache does. That's what continuous batching does. The techniques in this course don't have to trade one for the other. Instead, they eliminate waste so you get more of both. Now you understand the tradeoff. Let's look at the inference layer where performance is won or lost. At the bottom sits your hardware, NVIDIA GPUs, Google TPUs, AMD GPUs. Custom ASICs. At the top are your applications. Chatbots, Code Assistants. This is the layer your users interact with. Their needs are universal. fast and accurate responses. In between sits the inference layer, and this is where performance is won or lost. It's doing much more than calling model forward. It's batching requests for efficiency, managing memory to fit as much as possible on a GPU, and scheduling which request gets compute next. Several frameworks operate here. TensorRT-LLM, Ollama, and vLLM. This course uses the SGLang framework because it's open source and handles scheduling, batching, and memory optimization. You will use SGLang's caching approach, not just within a single request, but across requests in the next two lessons. But first, it helps to understand why LLM inference makes all of this really hard. In this example, the input is the cat sat on. The model predicts the, t h e. So the next input becomes the cat sat on the. Next, it predicts mat. The input becomes the cat sat on the mat. Then the model produces a period. Each token is generated by looking at everything that came before it. That's the fundamental constraint, and that's what we mean by autoregressive. Autoregressive creates three conditions that makes inference hard. First, generation is sequential. You can't produce step three before step two, because step three depends on what is produced in step two. If you need 100 tokens, that's 100 sequential forward passes. No shortcut. Second, Memory grows with every token. At each token, the model computes key and value tensors for the new token and stores them. By step 1000, you're holding keys and values for all thousand previous tokens. That adds up fast, especially across many simultaneous users. Third, this is more complex to scale than training. Training can be split across hundreds of GPUs working independently. But since autoregressive generation is inherently sequential, you can't throw more GPUs at a single request to make it faster. The growing memory cost of storing keys and values at every step is what KV cache addresses. Instead of recomputing keys and values from scratch every time, you compute them once and store. You will build this concept yourself in the next lesson. Three things make SGLang well suited for this problem. First, a fast runtime built for high throughput and low latency. The techniques that power it, RadixAttention, Prefix Caching, Continuous Batching, all follow the same principle. Find a computation that's repeated and stop repeating it. Second, Broad model support. SGLang runs large language models like Llama, Qwen, DeepSeek, and Gemma, but also embedding models, reward models, and diffusion models that is compatible with many APIs out of the box. Third, extensive hardware support. Using SGLang framework can be compatible with many vendors, which matters a lot because the hardware landscape is evolving fast. SGLang has already received broad adoption across the industry. It's running at AI native companies like xAI and Cursor, and even helping prepare research in universities like MIT, Stanford. Berkeley and CMU. In the next lesson, you will implement the attention mechanism line by line, see exactly where the redundant computation happens, then build KV cache to eliminate it. See you there.
