Sign in to continue learning
Design, Develop, and Deploy Multi-Agent Systems with CrewAI
Beginner
13h8m
Topics
Agents
Now, let's get everything that we learned and put that into use. Let's get our deep research crew that we built earlier, and now we're going to make it better. We're going to make sure that we include memory and execution hooks into that, so we can do even more than what it was doing before. Let's dive into that. All right, so now we're going to take our deep research crew into the next level. I'm so excited about this one, because we're not only going to get the things that we built before and make them better, but we're going to use some of the new things that we just learned in the previous lessons. So this was kind of like the last state that we had of your crew, where we basically had the research plan, the gather research, the verify information, the writer, and the report writing, all working sequentially one after the other. Now we're going to start to apply a guardrail strategy into this to make sure that we get a better report, and we're also going to make sure that we're saving the report as a callback so that we can see it later. We can then make sure that we're enabling memory so that these agents get better overall, but not only sharing information with each other, but storing learnings across multiple runs. So let's jump into the Jupyter Notebook so that we can start to get our hands dirty. All right, so the first thing that we would do is our usual loading of modules and classes. Again, same thing that we have done before, loading the agent, the task, and the crew. We're loading the AXA search tool and the scrape web search tool, and we're also loading all of our keys. We're using Jupyter for our mini, and we're also getting our AXA and OpenAI key out of this one. And we can just run this cell and get things going. Now next, we're going to make sure that we're creating instances of our tools, basically the same way that we had the last time that we were working on this crew, where we're going to basically load the AXA tool and the scrape website tool, and we can execute that just to make sure that we get that done. Now, because we're moving away from declaring our agents and tasks all the way into here and mixing prompts with the code, to actually have these agents and tasks as YAML files, we can just load that content straight from the YAML file so that we can create our agents based off that. And I'm going to show that YAML file for you in a second. But in here, you can see how easy it is to load it. And now we can reference that when creating all of our agents. So creating our agents now, it's way more straightforward, because we can all reference that agent config in those specific agents configurations and prompts that are in that YAML file. The other thing that you will see is that both the Internet Researcher and the Fact Checker are both still with their AXA search tool and their scrape website tool. All right, so before we jump into these YAML files, let's go over why we're using YAML files in the first place. And the reason here is that you don't necessarily want your agents and tasks definitions and a bunch of that prompting to just leave in the middle of your code. That makes it harder for you to parse and read, and also don't incentivize other people to contribute on those. By moving that text content into a YAML file, you actually get to have some separation of concerns, so that you make sure that if you ever want to change those definitions, you can do that straight in the YAML files, and that is separate from your code. And that is as simple as just having this config folder with both agents and a tasks YAML file. And if you open these YAML files, what you find is a regular YAML that describes the content name in the top, and inside that, the same attributes that you would see over the code. So the role, the go, and the backstory, and for the tasks, the description, and the expected output. The cool thing about this is that again, now this is completely separate from your actual code, and you can version this so much easily. And if there is non-technical people that want to take a stab at updating this, they can do it in a very easy manner, and your entire code looks clean as a byproduct of doing that. So moving forward on your other examples, and as we work on this crew, we're going to stick with the agents YAML and tasks YAML file, just to make sure that we keep that a separation of concern, and make our lives a little easier. Now you might notice that we're also having this verbose equals true. And just to be vocal, what this is doing is just plotting out those outputs that we have been tagging along across the entire notebook. So as we run this agents, and we're able to tag along their executions straight in here, that verbose equals true is what is actually enable us to do that. Now let's move on and start to creating our tasks. But before we do that, I want to make sure that we create our first guard rail. It's a code guard rail, and this code guard rail is extremely interesting. So let me actually bring this on so that you can see it. If I paste it in here, you can see that this guard rail, it's extremely straightforward. What it's doing is, first of all, is getting the entire output and make it lowercase. So we can now use Regasps to search the entire output. And the main thing that you're checking for is to make sure that this report includes the summaries mentioned, and insights or recommendations mentioned, and also citations or reference mentions. So it's basically making sure that our final report, in fact, include the things that we care about and the sections that we care about. So that are referencing not only a summary, but insights and citations as well. And this is just one way to reinforce that behavior into our agents. And you're going to see as we run this crew, that if it fails into any of these checks, it sends this information back into the agent so the agent can actually get the work done. So you can see that if at any point in time it fails any of these checks, it actually returns false, indicating that this output is not ready and the test should not be done, and gives a reason on why that's the case. And the reasoning here is important because, again, this is injected back into the agent when we ask it to do better and to redo the work. Now, if doesn't fail in any of these checks, then we return true and the new output back. So we don't need to worry about that. Now let's make sure that we create this guardrail and we move on to creating our tasks, which will also be loading from the YAML file. So let me just copy that and paste in here in our cell. If you see in here, the tasks are extremely straightforward to create. In the same way, we're loading the YAML file and then we're creating all the tasks reference their main keys in that YAML file, as we just saw earlier. But now there is a major difference in here. You can see that we are defining a guardrail, and this is a coding guardrail, is the function that we set up earlier. That is basically making sure that the final output of this report passes that check before it goes on. Now that we have created our guardrail and our tasks and have properly assigned our guardrail to our task, I want to make sure that we're creating our first hook. It's going to be your post or after execution hook. So this hook is going to be extremely simple, it's just to showcase how it actually works and how this could be useful for you. It's going to be a simple save file hook so that we can get the final output. As you can see in here, we're taking the final task output in here and just saving that into a markdown file. So we're just opening that file and writing the content in there. You can do this by default with Crew AI as well by using the output file function, but this is just to highlight how easy it is for you to create these functions and set them up as hooks, either prior the execution or after the execution. So let's create this function and then jump into defining our crew. So for a crew definition, this is very straightforward. We're bringing all our agents in here, all our tasks, and we are enabling memory for the first time. Remember, we didn't have memory before, but now we're enabling memory for this crew and we are also adding the after kickoff callback to basically execute that hook for saving the file that we have. So let's create our crew and I'm very excited to set the input and then run it. Now let's create the inputs that we're going to be using for this and the input is going to be the same one that we used the last time where we're basically researching the top five emerging AI tools for automating competitive market analysis. We can just paste it in here and now we have the input properly defined and with this input, we're ready to just to execute our crew and tag along the process to see the results. So let me just kick off our crew with the user query that we just defined and see what we get in the end. And don't forget to play around with this. Change the file save hook, change some of the guardrails, make sure that you feel comfortable playing around with these concepts, changing the LLM and the user query to see what kind of outputs you're going to get from this. This is a very exciting moment for us. So let's kick this off. All right, so now let's tag along the execution of our agents. The research planner kick things off and finds out what should be the plan for us to get the best result from our report. So it decides to identify what are those tools, look at their features, look at their limitations, do some cost analysis and check the ideal use cases for these marketing firms. So it has all the plan laid down. As we keep scrolling, we're going to see the different agents, the research nerd to figure out how to find this information and put this report together. So now in our next agent, you can see that it's again using the access search tool and searching again for the top emerging AI tools and finding all the different tools that are available in there. It's going to then scrape some of these websites to understand what is the best tool around. So let's keep tagging along the execution. So we found out five tools for us to focus on, and that is now going to the fact checker agent that will also use the access search tool, but to steady check all the data that we have about these tools and make sure that that's factually correct. So let's go into that. All right. So now we get to our final report agent with its final answer, and you can see that it's given an entire executive summary, all details, findings about all these tools. It's actually mentioned the citations and even mentioned insights in a conclusion. So in matter of fact, this passes all the checks from the guardrails from the first try. Guardrail or guardrail never even mentioned that our agent should retry running this because it got the whole thing from the get go. But you can actually play around with this. You can update our tasks definitions or agents definitions so it doesn't mention a conclusion or didn't mention recommendations or doesn't include a summary, and you can see how that works. Matter of fact, why don't we do that real quick right now? So I'm going to go here in our final task, and I'm going to say that I don't care about this implementing a summary. So just say that it needs a detailed findings, that it needs a support evidence, and that it needs to have complete citations with key insights. So I'm not mentioning a summary anymore. All right. So now that we have changed our task, you can see that this time the final answer does not mention a summary. What is a hard requirement on our guardrail? So if we scroll this down a little bit, we can actually see that the guardrail blocked and it's retrying due to the fact that the report must include a summary section with a header. And now it takes that message back into the reporter agent saying like, hey, do this again, but this time you need to make sure that you include that summary in there. And what it does, it actually includes an executive summary on item A here that didn't have before. And this actually allows the agent to call the task done and keep moving. Now make sure that you play around with this, change the prompts a little more, change the models a little more. There's a lot of room for you to play around with this concept and see how they work in the real world. But it's definitely very exciting. And I hope that you stick around because in the next lesson, we're going to talk about some even more interesting aspects that will then feed back into how we improve this agent. And this is going to get insane down the road and you don't want to miss that out. So see you in the next lesson in a second.
