Welcome to this course on Fast & Efficient LLM Inference built in partnership with Red Hat. Open source LLMs can be so large that deploying them efficiently for a large number of users can be challenging, especially if you need low latency and reasonable cost. In this course, you learn to take an open source LLM and serve it efficiently using vLLM, which is a widely adopted open source serving system. You'll learn key ideas behind it like PagedAttention, which lets your model serve many requests at once without wasting GPU memory. You'll also learn how to compress the model size and evaluate how well it can handle real-world traffic. I'm delighted to introduce your instructor, Cedric Clyburn, who is senior developer advocate at Red Hat. Thanks Andrew, and I'm really excited to work with you on this course. When an LLM answers a prompt, it generates text one token at a time, taking into account all previous tokens to decide what comes next. These computations rely on two things in GPU Memory: the model's Weights and the KV cache, that is the keys and values representing the context from these previous tokens. These two behave differently. Weights are loaded once and stay fixed regardless of how many requests you serve. The KV cache is dynamic, and every request has its own, and it grows with every token generated. In this course, we'll be looking at a 70 billion parameter model. The weights of which take about 140 GB, thus requiring at least two 80 gigabyte GPUs to load. In practice, you use even more GPUs to leave room for the KV cache. So the memory requirements add up. Maybe about 2.5GB for 8k-token request and 10GB for a 32k-token long context request. Multiply that by many concurrent users and managing this memory becomes critical. So, to serve LLMs efficiently, you can apply Quantization to shrink the weights by storing them at lower precision so they take up less space and also move faster through memory. The KV cache is more complex to manage because it grows dynamically and you don't know its final size ahead of time. Previous methods reserve one big block per request, sized for the maximum context, and that left 60 to 80 percent of the memory unused. But with PagedAttention, you can fit a lot more requests onto your GPUs by splitting the KV cache into small, fixed-size blocks that can sit anywhere in memory. And here's how you'll put all this into practice. You'll start with the fundamentals like why efficient deployment matters, what happens during inference and the core ideas behind LLM optimization. And then you'll go hands-on. You'll use the LLM Compressor to quantize an open source Qwen model and measure the accuracy tradeoff. You'll then serve it with vLLM and see PagedAttention in action, along with other vLLM techniques. such as continuous batching and prefix caching. Finally, you'll benchmark your deployment with GuideLLM, measuring latency and throughput metrics and evaluate model quality with LM-Eval. By the end, you'll have run the full optimized deploy benchmark workflow on a real model and understand the tradeoffs between accuracy, speed, and cost well enough to apply this workflow in production. Many people have worked to create this course. I'd like to thank, from Red Hat, Saša Zelenović, Michael Goin, and Sawyer Bowerman. From DeepLearning.AI, Hawraa Salami also contributed to this course. Please join Cedric in the next video where he'll walk you through what makes efficient LLM inference critical and the challenges of serving open source models in production.