Sign in to continue learning
Fast LLM Inference with Cerebras
Intermediate
1h42m
Topics
LLM Serving
GenAI Applications
Agents
AI Coding
Inference
With fast inference, both the user experience and the way you write code change. In this lesson, you look into the engineering shifts in software applications that fast inference creates. Let's go. Before we dive into the code, Let's look at one concept, which is the 500ms threshold. Sometimes people say 400 milliseconds, and sometimes 800 milliseconds, but the basic idea is that once a large language model or AI system responds in under about 500ms, the whole development pattern essentially changes. If you have traditional inference, then each request takes about 2-10 secs or longer, depending on how long the response is. You usually mask that with loading spinners and you need some sort of architecture in the background that lets you put a job on a queue. and then you wait for a callback until the job is done. Or you use streaming to return results early, but still not the full response at once. This might lead to users context switching while they are waiting for the actual answer. If you have fast inference, then your requests can be served in under 500 milliseconds. Again, it depends a little on the exact length. But for most of what humans can actually process anyway, which is a couple hundred tokens or words, fast inference feels like a function call. Which is why you can just run it synchronously as a simple function. In this situation, you get the full response, not a partial streaming response, before the user even notices it. This means the user stays in the flow and focuses on the conversation, not pauses in between. Inference responses in under a second do not just make the interaction feel real in a conversation. It also changes how software is built. So let us take a look at the concepts. and then also explore them in actual code. The first three shifts show up immediately in product and code. The two other are a little bit more abstract. but they matter just as much. In this lesson's notebook, this is phrased as five ways fast inference changes your code. And the core idea is that speed does not only improve the output, it changes the shape of the implementation around the model call. First, you can run synchronous functions instead of asynchronous functions. Synchronous means tasks run one after another, whereas asynchronous means tasks can run in parallel. An agent can find personalize, and act inside one conversational turn and you do not need to manage an asynchronous state. At the same time, you do not need to cover up loading. Sometimes, I have seen interfaces giving me little jokes for entertainment while I wait, or just a generic generating label or a loading spinner to signal that the response is still being processed on the back end. Another trick people usually use is streaming, where you get a word or a fraction of a word at a time, so you can start reading before the full answer is finished. And that does work because you can keep the user more engaged. But a real downside, particularly in agent use cases is that the agent often needs the full context to be actionable. That comes up, for instance, when you use tool calling, which you will do later in this course. With fast inference, even Multi-step agents can respond in seconds. So if you have, for instance, five steps, you will look at a sample of that in a moment and you will see that you still get the response in just a few seconds or at least with very low inference time. You can do search in the middle of the conversation or do other backend calls that are easier. if everything happens fast enough that your user never loses context. In general, this leads to simpler code, less streaming, and state management, and fewer abstractions overall. Fast inference does not just speed up your app. It can simplify the codebase too. It often makes the implementation smaller and more direct. You end up deleting machinery around the model call instead of adding more of it. And that is one of the most practical engineering advantages. Let's now switch to the notebook and see some of these engineering shifts in action. You'll compare a slower asynchronous path against a faster synchronous one, enabled by fast inference and finish with a multi-step workflow. We start with the notebook dependencies. It helps to see what each one is for as we go. os reads the API keys from the environment. threading lets you run the slow model call on a separate background thread, so the main program does not freeze while it waits. This is how you can imitate an asynchronous backend inside a single notebook. The request runs in the backend while the code keeps checking whether it has finished. time, lets you measure how long things take. pandas for tabular data, dotenv for loading secrets, pydantic for typed data, and rich for pretty printing. We then import the helper functions call_cerebras and call_openai. Each of these functions wraps a single model call and returns the response together. with the timing information we care about. That way, the notebook can focus on the shape of the application code instead of repeating request boilerplate. Finally, we load the environment variables to get the Cerebras and OpenAI API keys. and set our two model constants. The main idea of this lesson is that slower inference usually forces you into asynchronous patterns. The classic flow is enqueue, poll, and fetch. In a real product, you might hide some of this behind framework code, but the pattern is still there, underneath whenever the model is slow enough that you cannot comfortably wait for a direct response. On the left of this graph, a user submits a prompt and that request gets queued. Processing starts on the server and while the model is working, the interface shows some form of loading state. Then, every second, the client asks whether the result is ready. Once the job finishes, the application fetches the final answer and renders it. That pattern is familiar from many early LLM products. You type a message, see a spinner, and wait while the system periodically checks whether the long-running request is done. Imagine a user asks for a long answer, something like summarize the history of Alaska in approximately 800 words. If the model takes long enough, you do not want the interface to block silently. Instead, the request gets queued, the server begins processing it. The UI shows a waiting state and the client periodically asks whether the result is ready yet. To simulate that flow, we define a small JobState with a status and a result, where the status moves through queued, running and done. We then define a worker function that calls OpenAI in the background. As soon as it starts, it marks the job as running, and once the result comes back, it stores the result and marks the job as done. The prompt we're going to use is intentionally longer than the short examples from earlier lessons. Summarize the history of Alaska in approximately 800 words. because longer outputs make the waiting pattern easier to see. You could use a shorter prompt, but the effect would be less obvious. We launched a worker in a background thread marked with daemon=True, which means Python will shut down if the main notebook process ends. That avoids leaving a background worker hanging on the notebook if you interrupt execution. or leave the cell mid-run. In a normal script or a more structured server process, you would usually manage this differently. But for a teaching notebook, it keeps things predictable. Once the worker is running, the foreground loop polls the job status every couple of seconds. Each time through the loop, we record how much time has passed and append a poll record so we can inspect it later. The notebook prints those polling events so you can see the pattern directly. status, time waited, sleep, then poll again. That polling loop is deliberately simple. It is not meant to be production infrastructure. It is meant to make the hidden complexity visible. Even in this stripped down example, we already need a background worker, mutable job states, a timing loop, and front-end style polling behavior just to deliver one answer. That is exactly the sort of engineering overhead the lesson is trying to highlight. After the background job finishes, we print a summary with the number of polls, the total wall clock time, and the throughput. Depending on the run, the exact number of polls will vary. But the important observation is that the user visible workflow now stretches across many seconds. That is long enough that the product has to manage waiting explicitly. So let's run it. And now every two seconds we get the polling. The status is still running and then the time waited is 2, 4, 6, 8, 10 seconds. It's the time we're waiting for the LLM. to return the response. And here it is. We had seven polls, it took 14 seconds, and we got 75 approximately tokens per second. Notice the distinction between model duration and wall clock time. The model is busy for one amount of time, but the total time you wait is longer because the polling interval, thread scheduling, and the basic cadence of the app checking back for completion. That distinction matters in product design because what the user feels is wall clock time, not only the provider's internal completion metric. Now let's compare that with the synchronous path. We send the same prompt to Cerebras, but here the code shape is just request and answer. There's no background worker, no queue, no poll loop, and no fetch step. We call the model and get the result back directly. For the same long answer, this finishes in well under a second. The throughput is dramatically higher and the response comes back quickly enough that the product can simply wait for it in line. That is the first engineering shift. Fast inference removes orchestration that slower inference forces you to build. This difference is about more than benchmarking. It is not just that one bar on a chart is lower than another. The code you have to write is different. On the slower path, the application has to mask waiting. On the faster path, the application can stay direct. Now we move from one long prompt to a multi-step workflow where the difference becomes even more obvious. We define a small workflow with four tasks. classify the issue, summarize the context, draft a reply, and write a follow-up task. The scenario is customer support. A customer has reported a duplicate charge after a plan change. And we want the model to help process that issue step-by-step. Each step is short and intentionally narrow. But chained together, they represent the kind of multi-step logic many real applications run behind a single user action. First, we run the workflow synchronously with Cerebras. We iterate through the four steps, send each prompt to the model, collect the result and record the timing. Because the calls are fast, each step returns in a fraction of a second, and the whole workflow still feels immediate. We print each step with its per step timing. and the total workflow duration. So you can see both the micro level and the overall path. Even for consecutive model calls complete in a window where the application can stay simple. There's no need to push the workflow into a job system, send progress events or ask the user to come back later. As you see, this workflow took about one and a half seconds. Now we run the same logical workflow with OpenAI. focusing on the four API calls and measuring how long the workflow takes end to end. The same steps take materially longer and the difference compounds. A workflow with one slower step may be fine, but four slower steps start to push the user into waiting. Remember that users do not experience model calls in isolation. They experience full product flows. So once you chain several calls together, the wait adds up quickly. And we see it took about six seconds. This is why we called this an engineering shift rather than a simple performance improvement. Faster inference does not only make an existing workflow nicer, it changes what architecture is reasonable. You can keep more logic synchronous, show results in line, and avoid building support systems around waiting. Slow paths push you towards asynchronous patterns like queuing, polling, and background orchestration. Fast paths let the code stay direct, synchronous, and easier to reason about. And that difference only grows as you move from a single call to a chained workflow. In the next lesson, we illustrate a use case that makes decisions in real time with fast inference.
