Sign in to continue learning
Fast LLM Inference with Cerebras
Intermediate
1h42m
Topics
LLM Serving
GenAI Applications
Agents
AI Coding
Inference
Now it's time to put together what you've learned so far to build a real-time multi-step workflow that does live analysis of market signals. Let's dive in. Let's quickly recap what you have learned in the previous lessons. First, you built an understanding of the KPIs of fast inference. Then, you took a look Under the Hood and saw side-by-side comparisons of a fast versus a traditional inference use case where you create, for example, a software tool. After that, you took an engineering or code-first approach. to understand what changes when you use fast inference. The short version is that the code actually becomes easier. And then you looked at a real-time use case. Now, you're going to stack those concepts together and look at another real-time use case. This is the capstone lesson, a speed-first application. You're going to combine the different aspects from the earlier lessons. In other words, you're taking the speed argument from lesson four and the real-time product framing from lesson five, the direct synchronous style and the structured outputs, and use them inside one agentic workflow. In this scenario, You're going to look at the architecture of a speed-first application. It's an investment use case. The input is a stream of market signals, and a signal might be a press release, a news article, or an analyst update. Once a signal arrives, you want to decide in real time whether it matters, what it means and whether it should change prior decisions. So now let's take a look at the workflow you'll build. We will have different market signals come in and process them in real time. For that, as a first step, we are going to classify the signal and then pick a processing path and recommended tools. to identify how to deal with the signal. The second step, once we get our signal, is to identify whether it's relevant or not, and if we are tracking this actual company. Let's go for the negative path. If it's not relevant at all, we decide to ignore the signal and just go to the result summary. If it is still though relevant, then we load our company database with all of our prior facts that we have collected in the past and the previous decisions. Decision being buy, sell, or watch. Then we take the market signal, we extract the fresh facts that are coming from it, and we either quickly update our decision, or we compare it to the prior decisions because we see that there's a shift happening. So let's say we switch from buy to sell or from buy to watch. And then based on what our agent has decided, which path to take. We then return the result summary too. This table summarizes these different processing paths with the ignore, the quick_update, fact_refresh, and full_review process. When the router chooses it and the steps that run. So now, let's dive into the code. We start with our imports, os and .env for configuration, rich for nicer printing, and Literal for the typed fields we use throughout. We then import the call_cerebras helper for the capstone. Next, we load the environment variables and set the Cerebras API key. The Cerebras model we have used throughout the course and a path to the database, which is the company memory. Additionally, we are also going to list the available companies that are in our database. It's a chipmaker, an automotive company, a media company, and other. As in the previous lesson, we define a small structured model base that forbids extra fields. So every schema in the workflow stays strict. And now we are going to look at one incoming market signal. We represent each incoming signal with a small MarketSignal structure. Its source_type, title, source, and text. And let's take a look at a few example signals. The first one is a news article. Chipmaker shares rise after new cloud deal points to faster backlog conversion. from a market news outlet, describing a multi-year cloud infrastructure deal. At first glance, that looks positive. But the important question is not only whether the article is good news in general, it is whether it is relevant to one of the companies we track. The other two signals a city marathon and an automotive recall update. Let us see later how the workflow handles an irrelevant item and a negative one. Let's start with the first step, the classifier, which decides which track company the input signal is about, whether it matters, and which tools are worth using next. The classifier itself returns a structured result. The relevance, the target company, the processing path. The list of recommended tools and a short reason. The processing path is how the model decision is being made and how much work to do next. The classifier itself is a single model call with a router system prompt. It picks the target company from the tracked list only, decides whether the signal matters and selects the lightest useful path and set of tools. Let's take a quick look at the prompt. You are a router. Classify the signal and choose the lightest useful workflow. So here we instruct the model. And then we also have certain routing rules. we pass in our available companies at the top, the signal, and then go through those aforementioned routing rules. If the signal is not about the tracked company, set relevance='ignore', target_company='other', processing_path='ignore', and recommended_tools to zero. And then we also can go through these different paths. quick_update, fact_refresh, full_review that we've looked at earlier. These clear instructions allow the tool to make a dynamic decision based on the market signal it's getting. We begin by classifying the chipmaker article. The model marks it as highly relevant and selects a quick update path. The reasoning is straightforward. The article is about a target company we track and the news is positive. Now we take the second step. Once the workflow has chosen that path, we load the database context for the target company. The database already contains a prior decision and previously recorded facts. We can represent the database context in a structured object, which records whether the company was found. The previous decision, when it was last reviewed, and the stored facts. This step is a plain database lookup, not a model call. It reads the most recent rows for the target company from a SQLite database using a SQL query and packs them into the structured context. Now we are going to execute this and load the context for the chipmaker. The prior stance here is already buy. So the task is not to make a decision from scratch. It is to decide whether this new information strengthens, weakens, or reverses what we already believed. The next step is fact extraction. Here's the structure of the output returned. The fact extraction tool pulls out the concrete facts, the decision depends on the overall_signal, the announced_event, the key facts, the upside_signals, the risk_flags, and a short why those facts matter. That trace is useful later if we want to understand how the model interpreted the event. This extraction is another StructuredModel call, focused only on the facts that matter for the investment decision. Running it on the chipmaker article. The model pulls out the new cloud deal, the backlog implications, the upside indicators, and any risk flags, as you see in this print. As the fourth step in our workflow, we compare the output records to the previous decision. What changed, what stayed consistent, the effect on the previous decision, and whether confidence shifted. This is the model call that performs the comparison, lining up the store database context against the new facts. We do it with the compare_with_prior_decision function. And now we run it. The previous decision was buy. Here we see what changed, what stayed consistent, the effect and the confidence shift, which is now higher. Last but not least, we write the updated decision. For that, we define another type, the MarketSignalDecision with the target_company, the action, the summary, and the reasoning on it. And then we define the update_decision function, which takes this MarketSignal the database context, the fresh facts, and the comparison, and returns this nicely formatted MarketSignalDecision. Let's define the function and now run it. And because the new facts reinforce the existing thesis, the action stays at buy, but the supporting facts are refreshed. This is an important detail. The workflow is not only making binary choices like buy or sell. It is also deciding when no change is the right outcome. If the signal confirms the current thesis, the correct behavior may be to refresh the context rather than flip the action. Now we wrap the individual steps into one workflow function. First, we define the result structures. A workflow result summary captures the signal title, the processing path, the decision, the step count, and the wall clock time, so we can see both the chosen path and how much work it took. Then the class SignalWorkflowResult pairs that summary with the list of steps that actually executed. So now we're introducing the run_signal_workflow function. It's a slightly longer function, but it does exactly what we saw earlier on the diagram. First, we start with the workflow_start timing to make sure we measure how long steps take. Then we classify our market signal, which is the first API call we make. And if here in this first part the classification is not ignore, then we will load the entire complex workflow. If it is ignore, then we just jump down and return the signal workflow result. So if the classification of the relevance is not ignore and the classification of the target company is not other, meaning it is relevant to us and we want to classify it, then we will execute all these steps dependent on what the LLM chooses. Otherwise, we quickly jump down here and return the SignalWorkflowResult. If we take a closer look, at what happens if it is relevant and we track the target company for instance our chipmaker, then we load the database, we extract the fresh facts And then if the fresh_facts are not None, and we want to compare it with the prior decision, then we want to append the compare_with_prior_decision step and work through the list that we've seen earlier, dependent on the path that the LLM has chosen. Eventually, we make sure we track the entire workflow_wall_time, meaning the time it took from start to finish to come up with a decision, even if ignored. And we return the SignalWorkflowResult, as we said, in both cases, even if the company wasn't relevant to us or we wanted to ignore this market signal. Earlier in this notebook, we loaded three signals. So now we want to work through these other signals as well to make you understand how fast we can process them. Let's run the workflow on the second signal. This one was a city marathon project and it is not relevant to the track companies. So the workflow classified it as irrelevant and stopped immediately. in a single step. The third signal is more serious. An automotive company raises its recall reserve and reports lower margins. That signal is relevant and negative, so it triggers the full review path rather than the shorter update. It runs more steps and ends in a sell decision. For contrast, the positive chipmaker article takes the quick_update path. Let's run it. Keeps the decision at buy and completes in only a few steps. Printing the processing_path and the step_count side by side shows the point. An irrelevant signal exits early, a straightforward relevant signal takes a few steps, and a signal that could change the thesis takes the deepest path. Even with this branching, each workflow finishes in only a couple of seconds, fast enough to process signals in real time. The takeaway here is that fast inference lets you build selective, tool-driven workflows that stay responsive. Instead of forcing every input through the same extensive process, you can classify first. Choose a path and only spend extra reasoning where it matters. That makes the system faster, simpler to operate, and easier to explain. In the next lesson, you learn with Sarah how to work with an ultra-fast coding agent.
