Sign in to continue learning
AI Agents for Image and Video Generation
Intermediate
1h24m
Topics
Agents
In the previous lesson, you built your first generative media image agent. In this lesson, you will extend those capabilities to video. You will build an agent that plans multi-scene explainers, generates reference frames, animates them with audio, and evaluates temporal consistency across scenes. Let's have some fun. In the previous lesson, you built your first generative media image agent with agent development kit. Now we go further and build a video agent. You will start with a short and simple prompt and end up with a multi-scene explainer video with animated diagrams and synchronized audio. The agent uses tools to automate the process for you, from planning to generation to evaluation. Depending on the results, it can also take different paths. For instance, if a scene doesn't pass evaluation, the agent can regenerate it with feedback rather than just moving on. You will see how building a video agent introduces more complexity than an image agent. And also you will learn how to create long videos while maintaining visual and audio consistency. You can apply the same approach to many other scenarios such as product demos, architecture walkthroughs, or educational content. Let's take a look at the tools. The first one is plan_scenes. You give it a text brief, tell it how many scenes you want, and define a voice profile, and that's it. Gemini will take everything and generate a structured plan for every scene. Every plan has three fields that will be used as a prompt inputs for the next steps. The visual_description for image generation, the script narration for audio, and the camera_motion to tell Veo how to animate the scene. You might be wondering why a voice profile is needed. This is a voice style description that gets appended to every scene prompt, and this is how you keep your audio style consistent across the full video. The second tool generates a reference image for every scene using Gemini image generation. You use image to video rather than text to video because it gives you more control over the first frame. A style prefix is also appended to every image prompt to make sure all scenes maintain the same visual style. Once you have generated the reference image, the next tool uses Veo to animate them into a video clip with audio. It takes the reference frames from the previous step alongside the visual description, camera motion, audio, and voice profile, and internally it combines all of these into a single prompt that is passed to Veo together with the image. Finally, the agent has a specific tool for evaluation. It scores the generated videos across multiple criteria such as video consistency or visual quality. This is very important especially for long videos. Without this step, you can end up with scenes that feel disconnected from each other. The evaluation catches that and retries with specific feedback. You might be thinking, how does the agent know what to do at every step? The good news is that you don't need to worry about orchestration. The agent development kit takes care of everything, as you saw in the previous lesson. The LLM is the reasoning brain of your agent. It reads the results from every tool call and automatically decides what to do next. This logic lives in the system prompt. And if you want to change the behavior, you just need to change the prompt. Now it's time to build your video agent. As we just discussed, you will start from a short text brief and end up with a multi-scene explainer video with animated diagrams and audio. First, you import the same libraries as before, plus the video specific types for Veo. You will also have to authenticate in the same way as the previous lesson. Here you will use three models, Gemini Pro for planning and evaluation, Nano Banana for reference frames, and Veo for video with synchronized audio. Here you define the voice profile that controls how the audio sounds. And the STYLE_PREFIX does the same thing for visuals. Both are defined once and added to the prompts for every scene. Every detail you put here affects the consistency of the visuals and audio across scenes. For instance, Here the voice will be a male voice with a neutral American accent, as well as other characteristics. You can also define your own voice profile if you want. Now we will start defining the tools for the agent. The first one, plan_scenes, take your brief and uses Gemini to break it into structured scene plans. Take a look at the prompt. You ask for a JSON output with a visual_description, the audio script, and the camera_motion for every scene. Okay, let's start by defining the brief. Then you can run this and check the outputs. Here you have the three generated scenes with the visual layout, the spoken audio, and the camera motion that you have defined earlier. These three fields feed directly into the next tools, and the quality of your plan determine the quality of your final video. Once you have defined the scenes, the second tool will generate a reference frame for every scene using Nano Banana. Notice the STYLE_PREFIX is prepended to the visual description. This is what ensures all scenes share the same visual style. Then the next tool is generate_scene_video. It animates the reference frame into an 8-second video with audio using Veo. Take a look at how the full_prompt is constructed. You have here the visual description plus the audio script plus the voice profile that you have defined earlier. Veo also handles audio generation natively. You don't need a separate text-to-speech step. You just need to define generate_audio as true in the configuration. This is supposed to take about one minute per scene. And the final tool you will need to define is evaluate_scene for evaluation. This tool scores the video using the same LLM as a judge approach from lesson four, but applied to video. The criteria here include temporal consistency, motion coherence, prompt adherence, and others. There is one key addition here, which is the failure type. If the overall score is below the threshold, the evaluator classifies the failure as visual or audio. This classification is what tells the agent what to retry. Let's run this cell. Now, in the next step, you wire everything together with the agent development kit. Before running the agents, you inject the corrupted audio for the first scene. This is an intended simulation for demo purposes. You will be able to see how the agent evaluates the outputs, detects the audio mismatch and decides to retry the video only, reusing the same reference image. This is exactly what makes this an agent and not a deterministic pipeline. And then you will just import some display tools that you will need in the next steps. Now it's time to define the SYSTEM_PROMPT. This is a very important step. It's exactly what tells the agent what to do. Make sure to write detailed instructions here because the agents will rely on them to take decisions. Also take a look at the failure handling. Audio failure means retry the video, while visual failure means retry both images and videos. All right. Now you are ready. You define the tools and the instructions. You can go ahead and create the agent with these tools, and the agent development kit will handle everything for you from the tool calling, the orchestration, and the conversation loop. And similarly to what you have done in the previous lesson, here you will need to define a runner as well as a session to run the agents with the agent development kit. And then we go ahead and run the agents. This step takes typically about 10 minutes for three scenes. Okay, so here are the results for the first scene. The reference image looks correct, but listen to the video. The quantum blockchain leverages distributed photon matrices to synergize cross-dimensional neural pathways. The audio doesn't match the expected script because this is not a video about quantum computing. This is the corrupted audio you injected earlier. Now, take a look at the evaluation. You can see the audio alignment is low, and the failure type is audio. The agent successfully detected an audio problem and decided to retry the video. It will keep the same reference image while retrying. Now here is the retry. First, we take your initial user query and convert it into a numerical vector, which we typically call an embedding. This time the audio matches. And if we take a look at the evaluation again, it will pass, and the agent will move on. And now here is the second scene, image, video, evaluation. All of them pass on the first attempt. Same for scene three. The agent processes every scene and only retries when something fails. And we are done, all three scenes are completed. Now we'll use ffmpeg to join the three videos together. You will see how the final video feels like one continuous piece even though it's three separate videos. The style prefix and voice profile are what made the final video feels consistent. Let's take a look. First, we take your initial user query and convert it into a numerical vector, which we typically call an embedding. Next, we securely compare this embedding against our system's entire vector database to efficiently retrieve the most relevant document chunks. Finally, we feed those retrieved documents alongside your original query into the language model to generate a strictly grounded answer. And that was your final explainer video. Three scenes, consistent visual styles, audio, built entirely by the agents from one sentence brief. And you can use the same approach to build even longer videos with multiple scenes.
