Learn how to systematically evaluate, improve, and iterate on AI agents using structured assessments.

Learn how to add observability to your agent to gain insights into its steps and know how to debug it. Learn how to add observability to your agent to gain insights into its steps and know how to debug it.Learn how to set up evaluations for the agent components by preparing testing examples, choosing the appropriate evaluator (code-based or LLM-as-a-Judge), and identifying the right metrics. Learn how to set up evaluations for the agent components by preparing testing examples, choosing the appropriate evaluator (code-based or LLM-as-a-Judge), and identifying the right metrics.Learn how to structure your evaluation into experiments to iterate on and improve the output quality and the path taken by your agent. Learn how to structure your evaluation into experiments to iterate on and improve the output quality and the path taken by your agent._Learn how to systematically assess and improve your AI agent’s performance in Evaluating AI Agents, a short course built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and Aman Khan, Director of Product.When you’re building an AI Agent, an important part of the development process is evaluations or evals. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically—rather than relying on trial and error. With a systematic approach, you structure your evaluations to assess the performance of each component of the agent, as well as its end-to-end performance. For each component, you select the appropriate evaluators, testing examples, and metrics. This process helps you identify any areas of improvement so you can iterate on your agent during development and in production. In this course, you’ll build an AI agent, add observability to visualize and debug its steps, and evaluate its performance component-wise.In detail, you’ll: Distinguish between evaluating LLM-based systems and traditional software testing.Explore the basic structure of AI agents – routers, skills, and memory – and implement an AI  agent from scratch.Add observability to the agent by collecting traces of the steps taken by the agent and visualizing the traces.Choose the appropriate evaluator – code-based, LLM-as-a-Judge, and human annotations – for each component of the agent. Set up evaluations for the skills and router decisions of the agent example using code-based and LLM-as-a-judge evaluators, by creating testing examples from collected traces and preparing detailed prompts for the LLM-as-a-judge.Compute a convergence score to evaluate if the example agent can respond to a query in an efficient number of steps.Run structured experiments to improve the performance of the agent by exploring changes to the prompt, LLM model, or the agent’s logic.Understand how to deploy these evaluation techniques to monitor the agent’s performance in production.By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. _Anyone who has basic Python knowledge and wants to learn to evaluate, troubleshoot, and improve AI agents effectively—both during development and in production. Familiarity with prompting an LLM model would be helpful but not required.

Evaluating AI Agents

Automate business workflows with multi-AI agent systems. Exceed the performance of prompting a single LLM by designing and prompting a team of AI agents through natural language.

Exceed the performance of prompting a single LLM by designing and prompting a team of AI agents through natural language. Exceed the performance of prompting a single LLM by designing and prompting a team of AI agents through natural language.Use an open source library, crewAI, to automate repeatable, multi-step tasks like tailoring a resume to a job description; and automate business processes that are typically done by a group of people, like event planning. Use an open source library, crewAI, to automate repeatable, multi-step tasks like tailoring a resume to a job description; and automate business processes that are typically done by a group of people, like event planning.By creating a team of AI agents, you can define a specific role, goal, and backstory for each agent, which breaks down complex multi-step tasks and assigns them to agents that are customized to perform those tasks. By creating a team of AI agents, you can define a specific role, goal, and backstory for each agent, which breaks down complex multi-step tasks and assigns them to agents that are customized to perform those tasks._Learn key principles of designing effective AI agents, and organizing a team of AI agents to perform complex, multi-step tasks. Apply these concepts to automate 6 common business processes.Learn from João Moura, founder and CEO of crewAI, and explore key components of multi-agent systems: Role-playing: Assign specialized roles to agents Memory: Provide agents with short-term, long-term, and shared memoryTools: Assign pre-built and custom tools to each agent (e.g. for web search)Focus: Break down the tasks, goals, and tools and assign to multiple AI agents for better performanceGuardrails: Effectively handle errors, hallucinations, and infinite loopsCooperation: Perform tasks in series, in parallel, and hierarchicallyThroughout the course, you’ll work with crewAI, an open source library designed for building multi-agent systems. You’ll learn to build agent crews that execute common business processes, such as: Tailor resumes and interview prep for job applications Research, write and edit technical articles Automate customer support inquiries Conduct customer outreach campaigns Plan and execute events Perform financial analysisBy the end of the course, you will have designed several multi-agent systems to assist you in common business processes, and also studied the key principles of AI agent systems._If you’ve taken some prompt engineering courses, have some familiarity with basic coding, and want to incorporate LLMs in your professional work, then this course is designed for you!

Multi AI Agent Systems with crewAI

Use the AutoGen framework to build multi-agent systems with diverse roles and capabilities for implementing complex AI applications.

Use the AutoGen framework to build multi-agent systems with diverse roles and capabilities for implementing complex AI applications. Use the AutoGen framework to build multi-agent systems with diverse roles and capabilities for implementing complex AI applications.Implement agentic design patterns: Reflection, Tool use, Planning, and Multi-agent collaboration using AutoGen. Implement agentic design patterns: Reflection, Tool use, Planning, and Multi-agent collaboration using AutoGen.Learn directly from the creators of AutoGen, Chi Wang and Qingyun Wu.  Learn directly from the creators of AutoGen, Chi Wang and Qingyun Wu. _In AI Agentic Design Patterns with AutoGen you’ll learn how to build and customize multi-agent systems, enabling agents to take on different roles and collaborate to accomplish complex tasks using AutoGen, a framework that enables development of LLM applications using multi-agents.In this course you’ll create: A two-agent chat that shows a conversation between two standup comedians, using “ConversableAgent,” a built-in agent class of AutoGen for constructing multi-agent conversations. A sequence of chats between agents to provide a fun customer onboarding experience for a product, using the multi-agent collaboration design pattern.A high-quality blog post by using the agent reflection framework. You’ll use the “nested chat” structure to develop a system where reviewer agents, nested within a critic agent, reflect on the blog post written by another agent.A conversational chess game where two agent players can call a tool and make legal moves on the chessboard, by implementing the tool use design pattern.A coding agent capable of generating the necessary code to plot stock gains for financial analysis. This agent can also integrate user-defined functions into the code.Agents with coding capabilities to complete a financial analysis task. You’ll create two systems where agents collaborate and seek human feedback. The first system will generate code from scratch using an LLM, and the second will use user-provided code.A custom group chat with multiple agents that collaborate to generate a detailed stock performance report, incorporating a planning agent and customizing how the conversation flows between different agents.You can use the AutoGen framework with any model via API call or locally within your own environment.By the end of the course, you’ll have hands-on experience with AutoGen’s core components and a solid understanding of agentic design patterns. You’ll be ready to effectively implement multi-agent systems in your workflows._If you have basic Python coding experience and you’re interested in automating complex workflows using AI agents, this course will provide the practical skills and knowledge you need to leverage AutoGen effectively.

AI Agentic Design Patterns with AutoGen

Learn a flexible framework to build a variety of complex AI applications.

Learn a flexible framework to Learn how Haystack components, pipelines, and document stores can be used to build custom AI applications.build custom AI applications with components, pipelines, and document stores Learn how Haystack components, pipelines, and document stores can be used to build custom AI applications.Build a RAG pipeline and extend its ability by creating custom components. Build a RAG pipeline and extend its ability by creating custom components.Use conditional branching, self-reflecting agents, and function calling to build complex pipelines and apps. Use conditional branching, self-reflecting agents, and function calling to build complex pipelines and apps._In Building AI Applications with Haystack you will learn a high-level orchestration framework that helps ensure your applications are flexible, extendible, and maintainable, even as the technology stack changes, user needs arise, and new features are added.Using a framework can provide common features out of the box that significantly speeds up the development process. Haystack offers robust and flexible architecture and framework for building AI applications. It manages complexity and helps you focus more on developing your application at a higher level of abstraction.Throughout the course, you will develop several projects, including a RAG app, a news summarization app, a chat agent with function calling, and a self-reflecting agent with loops.What you’ll do: Learn about the core abstractions and unique building blocks of the Haystack framework and see how these elements can be combined for various AI use cases.Build a RAG pipeline by using Haystack components, pipelines, and document stores.Create custom components in your pipeline by building a Hacker News summarizer that extends your app’s ability to access APIs.Use conditional routing to create a branching pipeline with a fall back to web-search when the LLM does not have the context needed to fully respond to the user’s query.Build a self-reflecting agent for named entity recognition with a Haystack pipeline that is able to loop using an output validator custom component.Create a chat agent using OpenAI’s function-calling capabilities which allow you to provide Haystack pipelines as tools to the LLM, enhancing that agent’s capabilities.Start building exciting LLM applications and optimizing your development workflow using Haystack._Anyone who has basic Python knowledge and wants to learn to build and customize complex apps using Haystack!

Building AI Applications With Haystack

Efficiently handle time-varying workloads with serverless agentic workflows and responsible agents built on Amazon Bedrock.

Build and deploy scalable serverless agentic applications with Amazon Bedrock. Build and deploy scalable serverless agentic applications with Amazon Bedrock.Integrate tools, code execution, and guardrails to manage agentic actions effectively. Integrate tools, code execution, and guardrails to manage agentic actions effectively.Design responsible agents with safeguards to prevent malicious prompts and unintended outputs. Design responsible agents with safeguards to prevent malicious prompts and unintended outputs._Agentic workflows handle unpredictable tasks based on user input, like making API calls. A serverless architecture efficiently manages these tasks and varying workloads without maintaining servers, enabling faster deployment.You will learn to protect sensitive information and shield customers from harmful content by employing agents with guardrails. This course teaches you to build and deploy a serverless agentic application. You’ll learn to create agents with tools, code execution, and guardrails. The serverless setup is ideal for agents that might need to access many tools or APIs on demand. You’ll explore this through hands-on examples where you’ll: Build a customer service bot for a fictional tea mug business that can handle tasks like answering queries, retrieving information, and processing orders.Connect multiple types of agent actions, and implement guardrails for responsible operation.Use Amazon Bedrock’s fully managed services to deploy and scale the bot efficiently.The course will implement two elements essential to the deployment of business applications:Serverless deployment to achieve rapid scaling and seamless operation without the need to manage infrastructure.Responsible agent to protect your application from malicious prompts and unintended outputs by configuring guardrails.In detail, here’s what you’ll do: Use Amazon Bedrock to create an AI agent, explore how you invoke the agent, and see the trace to review the agent’s thought process and observation loop until it reaches its final output.Connect your customer service agent to services like a CRM to get customer details and log support tickets in real time.Attach a code interpreter to your agent, giving it the ability to perform accurate calculations, where it writes and runs its own Python code to support its response.Implement and configure guardrails to prevent your agent from revealing sensitive information and using inappropriate language.Connect your agent to a repository of customer support documents that discuss many issues that the agent can resolve directly or choose to escalate, if necessary, to a human workflow.Get a walkthrough of the Amazon Bedrock interface in the AWS console to configure agents, set the guardrails, and connect to knowledge databases, all in an easy-to-configure graphical interface.By the end, you will have built a sophisticated AI agent capable of handling real-world customer support scenarios, fully serverless, and ready to scale._Anyone who has basic Python knowledge and wants to build an agentic application for business or personal use. It’s useful to understand serverless deployment and cloud-based applications.

Serverless Agentic Workflows with Amazon Bedrock

Build agents that collaborate to solve complex business tasks.

Build agents that collaborate in complex workflows, integrating external tools and using different models to handle specific tasks efficiently. Build agents that collaborate in complex workflows, integrating external tools and using different models to handle specific tasks efficiently.Learn to evaluate and enhance your agents by conducting performance tests and providing human feedback to train and optimize their performance. Learn to evaluate and enhance your agents by conducting performance tests and providing human feedback to train and optimize their performance.Create multi-agent systems to automate tasks, such as project planning, lead scoring, data reporting, and large-scale content creation. Create multi-agent systems to automate tasks, such as project planning, lead scoring, data reporting, and large-scale content creation._Join our new short course, Practical Multi AI Agents and Advanced Use Cases with crewAI, taught by João Moura, Founder of CrewAI, and learn how to build and deploy agent-based apps for advanced real applications in the industry.In this course, you will build several practical apps like an automated project planning system, lead-scoring and engagement automation, support data analysis, and content creation at scale.Throughout this course, you’ll learn: The main building blocks—tasks, agents, crews—that go into creating these multi-agent systems, and all the different things that make them work such as caching, memory, and guardrails.How to integrate your multi-agent application with internal and external systems.How to connect multiple agents in complex setups including parallel, sequential, and hybrid, and how to create flows that involve multiple crews working together.How to test your crew by measuring key metrics and train it using human feedback to optimize its performance for better and more consistent results.How to work with multiple LLMs in your multi-agent system, using the appropriate model sizes and providers to fit each agent’s specific task.How to start a project from scratch in your environment, and prepare it for deployment.Bonus Interview with Jacob Wilson, the Commercial GenAI CTO at PWC: Learn about deploying agentic workflows in real industry use cases.Use cases in the courseYou will work on many exciting use cases such as building:A crew for automated project planning, breaking a project into tasks, creating time estimates, and allocating resources to them.A project progress report with an example of interacting with a project management system such as Trello.An agentic sales pipeline that takes in lead information, enriches and scores it, and writes personalized emails for qualified leads.A customer support data insights analysis pipeline that generates issue reports and visualizations.Content creation crew that researches the web, uses RAG as a tool on specific web pages to write content, reviews and modifies the content for quality, and generates social copies for different platforms._Anyone who has basic Python knowledge and wants to build complex and practical multi-agentic systems.

Practical Multi AI Agents and Advanced Use Cases with crewAI

Learn how an AI Assistant is built to use and accomplish tasks on computers.