Log In to continue learning
Fast LLM Inference with Cerebras
Intermediate
1h41m
Topics
LLM Serving
GenAI Applications
Agents
AI Coding
Inference
In the previous lesson, you saw how inference speed enables applications built on LLM reasoning and complex agentic workflows. Now, you'll see what fast inference looks like in practice. In the first part of this lesson, you will examine how inference speeds have evolved over time and what ultra-fast inference means concretely. In the second part, you will get hands-on experience with Seb and run fast inference yourself. All right, let's have some fun. So how are AI models actually getting faster? Speed comes from improvements at three different levels working together. First, Inference Optimizations. These are software techniques applied at runtime like batching, caching, and quantization that squeeze more performance out of the hardware you already have. Second, Model Architecture. This is about how the models themselves are designed. Choices like mixture of experts or attention optimizations that let a model do less computation to produce each token. And third, Hardware, the physical chips the models run on, which ultimately determine how much memory and computation are available. In this course, we're focusing on the hardware layer. Of those three levels, hardware is the foundation everything else builds on. So let's start there by looking at how inference speed has actually evolved over the past few years. In 2023, the frontier models operated at just a few dozen tokens per second. Today, LLM inference speeds typically start at over 100 tokens per second and can reach thousands of tokens per second, depending on the hardware provider, inference infrastructure, and model architecture. Each point here is a model, color-coded by the hardware it runs on, and you can see the Cerebras stars sit in the range above a thousand tokens per second. While many providers rely on general purpose GPUs, Cerebras developed the Wafer-Scale Engine, a custom AI chip purpose-built for large language model workloads. This is an independent benchmark from artificial analysis, a snapshot as of 11:58 a.m. on Wednesday, June 3rd. This benchmark measures output speed for the same model, GLM-4.7, across eight different inference providers, all running within a 10000 token input. Cerebras powered by the Wafer-Scale Engine delivers 1213 tokens per second. The next fastest provider, BaseTen on GPUs, reaches 245. That's roughly 5 times slower. Google's TPU offering comes in at 189, about 6.4 times slower. And from there, it drops off steeply. Most of the remaining GPU providers land between 30 and 110 tokens per second. So what makes this possible? The key difference comes down to where the model's weights live during inference. On a GPU or TPU, the model's weights are stored in a separate off-chip memory. And for every single token the model generates, those weights have to be shipped from memory over to the compute units. Cerebras takes a different approach. The Wafer-Scale Engine is a single large chip with enough on-chip memory that the model's weights stay on-chip right next to the compute. They don't travel from memory to compute every token. And that eliminates the data movement bottleneck and is a big part of why inference runs so much faster on Cerebras's Wafer-Scale Engines. We will go much deeper into exactly how this works in a future lesson. The goal of this lesson was to help you experience what ultrafast inference actually looks like in real workflows. As inference speed increases, AI applications become more iterative, more responsive and more capable of operating in real time. In the next video, Seb will give you hands-on experience with the Cerebras client so you can see this difference for yourself. You'll run fast inference yourself and directly compare the latency metrics between a model served through a GPU and another served through Cerebras.
