Sign in to continue learning
Build Interactive Agents with Generative UI
1h28m
In the previous lesson, you saw how controlled generative UI gives you full control and customizability, but it requires a dedicated component for every interaction. In this lesson, you will take a different approach called declarative generative UI. We will use a2 UI, which is an open spec for declarative generative UI led by Google, in which copilot kid has deeply collaborated. Instead of building a component for every single use case, you'll define a catalog of Lego like building blocks and let the agent assemble those on demand. This is an exciting part of the generative UI spectrum, so let's dive in. In the previous lesson, we defined a dedicated react component for each item that we wanted our agents to support. The benefit was great control over each and every item shown. The problem is that every new capability requires a new dedicated component built from scratch, which is fine at 20 components, but quickly becomes painful at 200 components. Declarative generic UI flips this equation rather than defining each component individually. We defined an upfront catalog, which we then let the agent assemble together on its own. Let's dive into what this means specifically. First, as we mentioned, developers declare a catalog of components which should be thought of as Lego like building blocks. The catalog is defined in two parts. First, there are definitions for each atom in the catalog, which include the items, name, description, and its required props. And separately, there are renderers defined for each item which take a Json payload describing each component and return an actual rendered component for the relevant platform. In this lesson, we will use react, but the same pattern extends elsewhere. When a user asks some question, the agent may decide to answer that question by assembly together a declarative generative UI component. The agent first emits a schema, which is a structural assembly of items from the Components Catalog, which is not yet populated with data. Then, the agent emits the data bindings that populate that schema with specific values. Finally, the schema. The data bindings are passed on to the front end, where they are served as inputs to a renderer, which returns the fully assembled components for the native platform. Again, in this case we will be using react. The end result is a component which can vary considerably, but within the confines of a bounded menu that you control. Declare a general UI has a number of distinct advantages. First, it offers flexibility, but with guardrails, your agent can render custom components for a job, but will drop from a fixed menu of pre-created components, which can be made, for example, to conform to your design guidelines. Additionally, because declarative UI is rooted in a structured schema, typically Json, it is suitable for any frontend environment not just for web, but also for mobile, slack, and other environments. At the same time, the cloud native UI is less predictable and customizable than fully control the generative UI. This unique profile of strengths and weaknesses makes declarative UI most suited for the long tail of product services. In consumer facing applications, where flexibility is generally more important than pixel perfect designs and predictability. It's also suitable for internal applications where functionality and simplicity of implementation often take precedence over a optimized experience. For example, if you're an airline, you will likely want your flight card to be implemented with control. The generative UI for maximal predictability as well as control of your applications. Most use surface by far, but if you think about features like tracking a lost laptop or getting refunded for a trip, it's more important that the agent can talk and interact with those abstractions at all, then that it offers a pixel perfect experience. And so these services are great matches for declarative generative UI. In this course we will implement declarative generative UI using a two UI. A UI is an open declarative UI spec that is led by Google, in which callback it has closely collaborate on. A UI implements the declarative of UI approach we outlined earlier with a component catalog, a schema data bindings, and renders using a UI. Operations e.g. middleware is fully integrated with 80 y, which means you can bring 80 UI declarative UI to any agent in the stack. Declarative. Generative UI comes in two flavors dynamic and fixed schema. With a dynamic schema variant, the agent is responsible both for assembling the schema on the fly and for populating the schema with data bindings. With the fixed schema variant. You, as the programmer, are responsible for defining the schema in advance, and the agent is only responsible for populating that schema with data bindings. Now let's jump into some code and take a look at how this works. First, just like in the previous modules, we're going to copy the reset session. So and run it to make sure we're starting from a fresh starting point. Next, just like before we're going to set up our dependencies. Starting by installing the Python dependencies followed by the front end dependencies. And finally we'll load up our API keys using the helper. Now that those are installed, we'll load up our API keys using our helper function. Remember, those are set up for you by the deep learning platform. Now we're going to build the agent. We'll start the server that's going to serve the agent. Now just like before we're going to start with our scaffold server. Initially we're populating an empty length agent behind it and we're going to populated very soon. That's it. Our API scaffold server is running. We're now going to actually serve a real agent from it. After we define it this time around, we want the agent to query for and use data before it responds with any declarative UI components. To simulate this, we're going to define a get sales data tool that the agent can call in order to fetch sales data. That given time in a real application, this would query some API or database. In this example here, we're just hardcoding some sales data for the agent to use. This is not our core focus. So if you're not sure what we're doing here you can come back to this point later. Now that we've created the tool, we're going to create an agent. Just like before, this is a standard long chain deep agent powered by GPT 4.1 model. We're giving this agent now the Get Sales data tool. And just like before we're passing it the Copia kit's middleware. You can read up the system prompt of this agent. As you see here, the agent is simply instructed to use the Generate Data UI tool to visualize data for the user. Now we're going to set up our front end with a two UI rendering. Just like before we're setting up our Copilot runtime. Most of the code here is very familiar. We're setting up a standard connector to our lane change agent that we set up in port 8004, and we're initializing Koba Runtime with that agent as the default agent. You'll notice the only real addition that we did not have before, which is the API configuration with a flag of inject a two UI tool set to true. Under the hood, the UI agent can talk to UI by calling a specialized tool. This tool is propagated by the callback. It's an UI stacks by default. It will be included if for any reason you do not want to include this tool automatically. You can always excluded by passing false to this argument. We're now going to define our declarative jovial AI component catalog. Remember, the component catalog consists of two parts the component definitions and the component renders. We'll begin by populating the component definitions. Now don't get scared. We're going to piece a big chunk of code, but we're going to scroll right to the top and explain what it is that we pasted in a moment. All right. This is our demonstration catalog definitions. You see here there is a dictionary of components with their definitions. The first component here is the title component. It has a name title a description in this case a heading. Use for section titles and petitioners and props. Typed schema is using Zod schemas like before for communicating the types of arguments this component is expecting. You see here we have a very long list of components that we're defining title, text, icon, image, and so on and so on. This is one of the inherent costs of declarative generative UI. You have to set up all of your Lego like building blocks up front for your agent to be able to assemble them together. We won't go through each one of the components here, because conceptually, they're all the same. We're now going to define the renders of our components. Once again, we're going to paste the long file first and then scroll up and walk through it. The beginning of the file consists of imports. They're going to be used by our renders and some helper functions. Once again use by renders. If we scroll down we'll see the actual catalog renders. First you see that the catalog renders is a generic type that specifies the catalog definition as its type parameter. This ensures that the renders perfectly match the types of the components that we defined earlier. Now you'll see that each one of the components we previously defined has a render. The renders are simple react functions. For example, let's look at the title render. You see that it takes a props argument. Props contains the types we defined earlier and then it simply returns a react component using these arguments. As you see here, there's a long list of component renders one render for each component we defined earlier. These custom renders is how you can get your own custom stylings when using declarative generative UI. Simply make your component renders aware of your design style guides. Now let's scroll down to the bottom of the render file. Quite a few renders in there. All right. You now see the demonstration catalog definition. We use the create catalog function call and we pass to it as arguments. The demonstration catalog definitions and the demonstration catalog renders which we created just now. Finally we're giving it to the catalog ID of app Dashboard catalog. All right. Now we have our catalog definitions in place and it's time to wire it up. We're now looking application code that you look very familiar. Once again we're wrapping our application with the callback provider so we can connect to agents anywhere within it. The only difference is that this time we're passing the callback provider. Another argument, the a two UI argument in the HPR argument. We're passing the demonstration catalog that we just created. This tells the agent it can use this component catalog when composing components together. Now we'll wire the copilot component back into our application along with some example suggestions. Then once again simply hardcode a set of prompts into the chat. Having defined the front end will now start it. We'll use our start frontend helper method to start the front end on port 3004. Now that it's running, will once again embed its iframe inside of a Jupyter notebook for convenience. All right, our chat is now live. Let's test out some declarative Genevieve UI rendering. First of all, click Sales Dashboard. You see here when we click the suggestion, we simply got a pre canned prompt. All right. The agents dynamically combine a bunch of the components we defined in our catalog together to show us this great sales dashboard. We see here the total revenue with a title and then a text. New customers conversion rate and even a pie chart and a bar chart. This is an example of dynamic schema declarative general UI. The agent first assembled a schema from scratch, putting together a bunch of the components from our component catalog. They then populated that schema with data bindings. We're now going to see an example of fixed schema declarative generative UI, where we as programmers hardcode a predetermined schema in advance, which the agent will simply populate with data. The first step in fixed schema declared unique UI is for us as programmers to define the schema. Now, schemas are a big messy Json file. The first step in fix schema declare origin of UI is of course defining the schema. Where is this schema going to come from? Well, you're not going to want to handwrite it. The schema file is a big messy Json schema that is designed to be written by machines. There's a great tool in the UI ecosystem called the API composer, which essentially includes a copilot that helps you assemble a schema to match some parameters. We're going to drop screenshots for this API composer into the Jupyter notebook, and you can find it on your own and then click on the composer. Fun fact the composer was me by copilot. This picture is what you will see when you go to the UI composer. There's a simple chat input box that lets you specify the type of composer you want to build. After some iterations, you'll end up with a schema. The composer shows side by side the schema sample data bindings that you can edit if you'd like, and a preview of what the rendered schema plus data bindings would look like. After some iterations, you'll end up with a schema to your liking. When you've done that, simply copy to Json and prepare for the next step. We're now going to add a tool that returns a fixed schema declarative generative UI component. You will want to paste the schema definition you got from the composer into a variable somewhere in your application. If you look here in the detail, you will see the rough shape of assembly of components together. But once again, this is not code that you want it to be. Handwriting. Another detail you notice here is the catalog ID referencing the component catalog we defined on the front end, and a surface ID. The surface ID identifies a specific UI component. It's useful because it allows components to be updated, not just appended to. We're going to create a dummy search flights tool for the agent to find relevant flights for this user. In real application, this would be hitting an API or database, but for the purposes of example, we're simply going to hard core some data here. Now we're ready to define the tool. It will return a fixed schema declarative generative UI component. As you will see, any component whatsoever can return such components. First let's look at what this tool returns. This tool uses the ATP Render Helper function to return an array of UI operations. The first operation creates a new service using our service ID. The second operation updates components for that service using our flight schema. And finally, the third operation populates the data for the components using the flights arguments passed to this tool. Remember in fixed schema declare generate UI, we as programmers predefined the schema, but it's still the responsibility of the agent to populate the data for that schema. It's critical that the shape of the data we pass the component matches the shape of the data expected by the schema in order to ensure this happens. We have our tool take an array of flight arguments whose shape exactly matches the shape defined in our schema. Now that we've defined our tools, it's time to create our agent. Once again, we're going to create a simple lane change agent. If you look at our list of tools, you will see the tool we defined previously. Get sales data with mock sales data, as well as two additional tools. The first one is the Search Flight tool, which provides mock data for flight information. Once again, in a real application, this would be collecting to an API or to a database, as well as the Display Flights tool. The agent can call the Display Flights tool in order to show the UI component to the user. Once again, we specify the callback ID middleware, and we include a system problem that tells the agent how and when to display components. If you're curious, you can read this in more detail. We're now ready to display the application. Now that we've defined our fixed schema tool, let's take a look at our application. The front end is exactly identical to what you saw before. But now that we've given our agent the ability to search and display flights, let's see what happens when we ask it for some flights information. Good job. The schema you created in the composer is now live on your screen, having been populated by the agent. We're now done with the section. Great job. You now have experience working with declarative generative UI. You first define the component catalog, which is specified by a combination of components, definitions, and component renders, and then exercise that component with both dynamic schema and fixed schema. Declarative generative UI. As a reminder, each declarative GI component is specified by a combination of a schema relative to component. A catalog, and data bindings relative to that schema. We first got our hands dirty with the dynamic schema declarative generator UI. When we gave the agent the ability to both specify the schema and populated with data. We then exercised fixed schema declared JPY, where we predefined a schema with the help of a UI composer, and then let the agent populate that schema with data. Great job!