Sign in to continue learning
Build Interactive Agents with Generative UI
Beginner
1h28m
Topics
AI in Software Development
In the previous lessons, you built agents that present rich UIs, but only inside of the chat window. In this lesson, you'll go beyond the chat window to build an app where the agent and the frontend share live state. We will build an agentic take on the classic to-do app, where the agent creates to-dos from the backend, users edit or complete those in the front end, and both sides stay in sync automatically. This state synchronization is the foundation for building your own Claude code or Cursor style application across different verticals. from legal to accounting to marketing to, you name it. So, let's get to it. We are now venturing beyond the isolated chat interaction to explore fully featured full-stack agents, which are designed to feel like true co-workers. It turns out that a small set of agent user interaction primitives are used by virtually all successful full-stack agents. Before we explore a couple of those technical primitives, it's worth taking a step back and briefly asking, what makes for a great Fullstack Agentic App from a product standpoint? Of course, a great intuitive user experience is the foundation. As we mentioned before, there is no reason to run away from the chat modality which has taken the world by storm. And remember that chat also does not mean text only interaction. You want to build augmented chat, which leverages Generative UI, voice interactions, and perhaps even video. A great chat experience also gives users visibility into what the agent is doing. To keep them engaged, but more importantly, to earn user's trust and to give users the opportunity to steer the agent to better address their needs. So this is a good place to experiment. Just as critically, chat as the anchor of a agentic interaction does not mean that chat exists in isolation. The agent should feel like a native part of a larger application. It should be able to leverage any data, context, connectors, or actions associated with the application at large. A good rule of thumb is that if an end user can take some action or learn some fact using the application at large, they should also be able to take that same action or learn the same fact via the chat. The agent must also have deep access to the application's real-time context, meaning to what the user is currently looking at and to the actions currently available to the user. We can tease out these qualities with a few examples. In Cursor and in GitHub Copilot, the agent isn't a chat window you switch to, it's integrated into the workspace itself. It reads your code, understands your project, edits files, runs commands, all without you leaving the editor. The agent and the tool are a single entity. In Notion AI, the agent is automatically aware of what's on the page. You don't need to paste context or explain what you're working on. It already knows because it's embedded in the document, not sitting beside it. In Harvey, a legal AI assistant, the agent doesn't just answer questions, it takes actions alongside the user, drafting documents, analyzing contracts, operating inside the same workflow the lawyer is using. Again, it's a collaborator, not a chatbot. It turns out that agentic applications across very different domains end up being built atop the same small set of core primitives. One of those building blocks is the Generative UI spectrum that we explored in the previous lessons. In this lesson, we'll look at two additional technical building blocks. Frontend Tool calling and Shared State synchronization. We will explore them from two perspectives. First, from the standpoint of developers, the cases you should know about in day-to-day development. And second, conceptually, so you can understand how these paradigms work under the hood. First, frontend tool calls. Frontend tool calls are directly analogous to the standard backend tool calls you're likely already familiar with. The difference is that they allow an agent to execute an action directly in the frontend application, not just where it's running. In React, you register frontend tools using the useFrontendTool hook, specifying the arguments expected by the tool and the handler to be executed when the tool is called. If your handler returns a result, that result is passed back to the agent as a standard tool call result. You can register frontend tool calls in a centralized fashion when your application initializes, or you can disperse their definitions across different parts of the app, so that they load and unload automatically as users traverse the application. Under the hood, these tool calls are passed to the agentic backend in a standard format. When the agent then decides to execute a frontend tool, the AG-UI recognizes that, pauses the backend execution, hands off control to the frontend, executes that tool, and then transfers control back to the agentic backend once execution is complete. Once again, you don't have to worry about all this complexity when developing, you simply call the useFrontendTool hook. Next, shared state synchronization. By default, the agent and the front end are disconnected. The agent does not know what the user sees, and the front end does not know what the agent knows. Shared state is the primitive that solves this. To use shared state synchronization, use your agent's standard state abstractions. In this lesson, we will use LangChain's agent state. On the frontend, use the useAgent hook, which exposes a reactively updated, typed state. object. Under the hood, here's how this works. As the agent run, it can update its state. When it does so, it can then update the frontend state by emitting state delta events, meaning it communicates the changes that needs to be applied to the frontend state so that it remains in sync with the backend state. These state delta events are themselves streamed because they're often generated by an LLM. The front end can also emit state delta events to support the state modifications that are coming from the user. Finally, when needed, conflict resolution between the two states is handled via AG-UI middleware. Once again, these are implementation details you do not need to be aware of in day-to-day development. Simply use the useAgent hook. In this lesson, you will build a canvas agentic to-do app that combines a chat interface with a traditional to-do board. You will expose a open to-do canvas tool to the agent to allow the agent to interact with the application on the user's behalf. And you will use state synchronization to keep the front-end representation of to-dos synced bidirectionally with the agent's representation of to-dos. Let's look at the code. We're going to be building an agentic to-do app that will venture outside of the chat window to a fully featured full-stack agentic application. Just like before, we'll start with our reset lesson cell. to make sure we're on a clean state. We'll set up our project dependencies. Once again, starting with our Python dependencies, and followed by our frontend dependencies. Before we move on, we'll also load up our API keys that are provided by the DeepLearning platform. We're now ready to load back our familiar starting point by starting the frontend and the agent server. We'll once again use a standard LangChain deep agent and serve it behind a FastAPI server. Now we're ready to start our frontend. Once again, we have our basic chat setup and ready for modification and augmentation. We're going to amplify this basic chat with frontend tool calling as well as state synchronization between the frontend and the agent. We'll begin by implementing shared state between the agent and the frontend. First, we're going to define a schema for our agent state. Our agent state consists of an array of to-dos, where each to-do has an ID, a title, and a flag indicating whether the to-do has been completed or not. We've defined our agent state. We're now going to define tools that will allow our agent to interact with its own state. The first tool we define is the manage_todos tool. This tool lets the agent update its own state using the Command function provided by LangChain. The second tool we provide is the get_todos tool, which allows the agent to read its own state. Finally, we put both of these tools in an array so that we can easily populate them into our agent later. All right. Now that we have the agent state and the tools defined, we're going to be updating our agent to take advantage of those. This code replaces the previous agent we defined with a new LangChain deep agent. This LangChain deep agent is still very basic with a few key changes. First, it specifies a state schema. using the AgentState type we created earlier. Second, it specifies the todo_tools that are now made available to the agent. Lastly, just like before, we're specifying CopilotKitMiddleware to allow the CopilotKit and the UI stacks to easily interact with the LangChain agent. Now, that we set up of the agent, it's time to set up the frontend to take advantage of this new functionality. Now we're going to introduce frontend tool calls to our agent, as well as bind the agent state we just defined to our frontend state bidirectionally. Once again, because we're editing TypeScript code inside of this Jupyter notebook, we must fully overwrite the file that was defined before. But let's walk through the changes we made. First, we're going to define a simple local variable that our application is going to take advantage of. todosOpen and setTodosOpen initially sets to false. We're then going to define a frontend tool that's going to be made available to the agent so that the agent can interact with this local variable. We are using the useFrontendTool hook provided by CopilotKit, which connects to AG-UI's frontend tool concept. And just like every other tool, this frontend tool can provide a name, a description, parameters, specified once again as a Zod schema. As a reminder, Zod is the TypeScript equivalent of Pydantic. And finally, our frontend tool can have a handler that will be executed when the agent decides to call this tool on the frontend. Notice that the frontend tool handler can be asynchronous. Under the hood, whenever an agent decides to call a frontend tool, the AG-UI middleware hands off execution from the agentic backend to the agentic frontend. Once a result is provided by the frontend, it is propagated back to the agentic backend. and the agentic loop continues. To recap what we did here, we created a local variable that allows the application to open a to-dos panel. Initially this to-do's panel is set to false. And we give the agent the ability to interact with this variable via a dedicated frontend tool call. Now we're going to subscribe to our agent state using the useAgent hook. useAgent is an incredibly powerful primitive for the agentic age on the frontend. It gives you everything you need on the frontend in order to interact with agents with all of the messy details automatically taken care of for you. In this case, we're going to be using useAgent's state variable in order to bidirectionally tie our frontend state to the agent state. Finally, we're returning the React component associated with this application now. The most important part I would like to point out here is the TodoList component, which accepts a state getter and a state setter for its own to-dos. The state getter is simply reading agent.state.todos. The state setter is simply setting agent.state with new todos. This is quite profound. We see here that the useAgent hook provides us with full-featured state synchronization between our front end and our back end. Just like local standard React state, the state is reactive, which we can take advantage of in our application. And every act of state update streaming and context resolution is automatically handled for us. As application developers, we can simply treat it as standard state as if it was just defined in our local application. Under the hood, it's automatically kept in sync with our agentic backend. All right, let's give it a try. Once again, we're going to use the display_app helper to display our application in line. All right, you see here we have now a chat with a new Todos button. Let's click it to see what happens. When we click it, we see a panel opening from the side with Todos. Right now, there are no Todos defined as of yet. If we wanted to, we could interact with this locally. Under the hood, remember, every time we do this, the agent.state is updated. and we can also close it and allow our agent to edit our to-dos for us. Remember, the agent can also set the to-dos open variable that decides whether the to-dos panel is open or closed. Let's ask our agent to set the Todos to open and to create a number of interesting Todos. All right, there you have it. We now have our agent open our Todos panel and populate it with brand new data. This state synchronization works in both directions. We can edit our Todos and ask our agent How many to-dos are still open and get a correct answer. Congratulations. You've now mastered frontend tool calling and state synchronization in addition to the full Generative UI spectrum you've mastered before. I'm extremely excited to see what you all be building. See you in the conclusion section for a quick recap.
