Learn to build AI agents with long-term memory with LangGraph, using LangMem for memory management.

Learn how memory works and how the three types of memory - semantic, episodic, and procedural - are used in agentic workflows. Learn how memory works and how the three types of memory – semantic, episodic, and procedural – are used in agentic workflows.Build a personal email agent with routing, writing, scheduling tools to automatically ignore, respond to, or notify about incoming emails. Build a personal email agent with routing, writing, scheduling tools to automatically ignore, respond to, or notify about incoming emails.Add long-term memory to your agent by adding facts, user preferences, and evolving system prompts to a memory store that can be searched for on-going interactions. Add long-term memory to your agent by adding facts, user preferences, and evolving system prompts to a memory store that can be searched for on-going interactions._Learn to build an agent with long-term memory in Long-Term Agentic Memory with LangGraph! Created in partnership with LangChain, and taught by its Co-Founder and CEO, Harrison Chase.Imagine you had a human personal assistant who forgot about their previous conversation with you—not ideal. The same goes for Agents.The main use cases for agents include personal assistance and productivity tasks. One core aspect of these tasks is long-term memory.In this course, you will learn how to build an agent with long-term memory by creating a personal email agent that can respond, ignore, and notify the user using writing, scheduling, and memory tools. You’ll develop your agent’s memory by adding facts to its memory store, providing examples to learn the user’s preferences, and optimizing system prompts to evolve instructions based on previous responses.By the end of this course, you will have the foundational mental framework to build an agent with long-term memory using LangGraph.In detail, you’ll: Learn how the three types of memory–semantic, episodic, and procedural–and the two mechanisms–via hot path and in the background– apply to your agent.  Build an email agent with writing, scheduling, and availability tools, along with a router that triages incoming email and handles it accordingly by ignoring, responding, or notifying the user.Add tools to your email agent that allow it to operate on semantic memory by learning facts about the user, storing them in a long-term memory store, and searching over them in future interactions.Incorporate episodic memory in the form of few-shot examples in the triage step of your agents to help them learn and update user preferences.  Add procedural memory as system prompts, optimized with feedback to improve the instructions the agent follows.Start building agents with long-term memory with LangGraph._It’s helpful to be familiar with Python and a basic understanding of LLM prompting and LLM application development.

Long-Term Agentic Memory With LangGraph

Learn MLOps tools for managing, versioning, debugging, and experimenting in your ML workflow.

Learn to evaluate programs utilizing LLMs as well as generative image models using platform-independent tools Learn to evaluate programs utilizing LLMs as well as generative image models using platform-independent toolsInstrument a training notebook, and add tracking, versioning, and logging Instrument a training notebook, and add tracking, versioning, and loggingImplement monitoring and tracing of LLMs over time in complex interactions Implement monitoring and tracing of LLMs over time in complex interactions_Machine learning and AI projects require managing diverse data sources, vast data volumes, model and parameter development, and conducting numerous test and evaluation experiments. Overseeing and tracking these aspects of a program can quickly become an overwhelming task.This course will introduce you to Machine Learning Operations tools that manage this workload. You will learn to use the Weights & Biases platform which makes it easy to track your experiments, run and version your data, and collaborate with your team.This course will teach you to:Instrument a Jupyter notebookManage hyperparameter configLog run metricsCollect artifacts for dataset and model versioningLog experiment resultsTrace prompts and responses to LLMs over time in complex interactionsWhen you complete this course, you will have a systematic workflow at your disposal to boost your productivity and accelerate your journey toward breakthrough results._Anyone who has familiarity with Python and PyTorch or similar framework and an interest in managing, versioning, and debugging their machine learning workflow.

Evaluating and Debugging Generative AI

Learn how to evaluate the safety and security of your LLM applications and protect against risks. Monitor and enhance security measures to safeguard your apps.

Monitor and enhance security measures over time to safeguard your LLM applications. Monitor and enhance security measures over time to safeguard your LLM applications.Detect and prevent critical security threats like hallucinations, jailbreaks, and data leakage. Detect and prevent critical security threats like hallucinations, jailbreaks, and data leakage.Explore real-world scenarios to prepare for potential risks and vulnerabilities. Explore real-world scenarios to prepare for potential risks and vulnerabilities._It’s always crucial to address and monitor safety and quality concerns in your applications. Building LLM applications poses special challenges.In this course, you’ll explore new metrics and best practices to monitor your LLM systems and ensure safety and quality. You’ll learn how to: Identify hallucinations with methods like SelfCheckGPT Detect jailbreaks (prompts that attempt to manipulate LLM responses) using sentiment analysis and implicit toxicity detection models.Identify data leakage using entity recognition and vector similarity analysis.Build your own monitoring system to evaluate app safety and security over time.Upon completing the course, you’ll have the ability to identify common security concerns in LLM-based applications, and be able to customize your safety and security evaluation tools to the LLM that you’re using for your application. _Anyone with basic Python knowledge interested in mitigating issues like hallucinations, prompt injections, and toxic outputs. 

Quality and Safety for LLM Applications

Learn advanced RAG retrieval methods like sentence-window and auto-merging that outperform baselines, and evaluate and iterate on your pipeline's performance. 

Learn methods like sentence-window retrieval and auto-merging retrieval, improving your RAG pipeline's performance beyond the baseline. Learn methods like sentence-window retrieval and auto-merging retrieval, improving your RAG pipeline’s performance beyond the baseline.Learn evaluation best practices to streamline your process, and iteratively build a robust system. Learn evaluation best practices to streamline your process, and iteratively build a robust system.Dive into the RAG triad for evaluating the relevance and truthfulness of an LLM's response:Context Relevance, Groundedness, and Answer Relevance. Dive into the RAG triad for evaluating the relevance and truthfulness of an LLM’s response:Context Relevance, Groundedness, and Answer Relevance._Retrieval Augmented Generation (RAG) stands out as one of the most popular use cases of large language models (LLMs). This method facilitates the integration of an LLM with an organization’s proprietary data.To successfully implement RAG, it is essential to enhance retrieval techniques for obtaining coherent contexts and employ effective evaluation metrics.In this course, we’ll explore:Two advanced retrieval methods: Sentence-window retrieval and auto-merging retrieval that perform better compared to the baseline RAG pipeline. Evaluation and experiment tracking: A way evaluate and iteratively improve your RAG pipeline’s performance. The RAG triad: Context Relevance, Groundedness, and Answer Relevance, which are methods to evaluate the relevance and truthfulness of your LLM’s response. _Anyone with basic Python knowledge interested in how to effectively employ the latest methods in Retrieval Augmented Generation (RAG).

Building and Evaluating Advanced RAG

Learn LLMOps best practices as you design and automate steps to fine-tune and deploy an LLM for a specific task.

Adapt an open source pipeline that applies supervised fine-tuning on an LLM to better answer user questions. Adapt an open source pipeline that applies supervised fine-tuning on an LLM to better answer user questions.Learn best practices, including versioning your data and your models, and pre-process large datasets inside a data warehouse. Learn best practices, including versioning your data and your models, and pre-process large datasets inside a data warehouse.Learn responsible AI by outputting safety scores on sub-categories of harmful content. Learn responsible AI by outputting safety scores on sub-categories of harmful content._In this course, you’ll go through the LLMOps pipeline of pre-processing training data for supervised instruction tuning, and adapt a supervised tuning pipeline to train and deploy a custom LLM. This is useful in creating an LLM workflow for your specific application. For example, creating a question-answer chatbot tailored to answer Python coding questions, which you’ll do in this course.Through the course, you’ll go through key steps of creating the LLMOps pipeline:Retrieve and transform training data for supervised fine-tuning of an LLM.Version your data and tuned models to track your tuning experiments. Configure an open-source supervised tuning pipeline and then execute that pipeline to train and then deploy a tuned LLM.Output and study safety scores to responsibly monitor and filter your LLM application’s behavior.Try out the tuned and deployed LLM yourself in the classroom!Tools you’ll practice with include BigQuery data warehouse, the open-source Kubeflow Pipelines, and Google Cloud. _Anyone who wants to learn to tune an LLM, and learn to work with and build an LLMOps pipeline.

LLMOps

Learn how to create an automated CI pipeline to evaluate your LLM applications on every change, for faster and safer development.

Learn how LLM-based testing differs from traditional software testing and implement rules-based testing to assess your LLM application. Learn how LLM-based testing differs from traditional software testing and implement rules-based testing to assess your LLM application.Build model-graded evaluations to test your LLM application using an evaluation LLM. Build model-graded evaluations to test your LLM application using an evaluation LLM.Automate your evals (rules-based and model-graded) using continuous integration tools from CircleCI. Automate your evals (rules-based and model-graded) using continuous integration tools from CircleCI._In this course, you will learn how to create a continuous integration (CI) workflow to evaluate your LLM applications at every change for faster, safer, and more efficient application development.When building applications with generative AI, model behavior is less predictable than traditional software. That’s why systematic testing can make an even bigger difference in saving you development time and cost. Continuous integration, a key part of LLMOps, is the practice of making small changes to software in development and thoroughly testing them to catch issues early when they are easier to fix. With a robust automated testing pipeline, you’ll be able to isolate bugs before they accumulate – when they’re easier and less costly to fix. Automated testing lets your team focus on building new features, so that you can iterate and ship products faster.After completing this course, you will be able to:Write robust LLM evaluations to cover common problems like hallucinations, data drift, and harmful or offensive output.Build a continuous integration (CI) workflow to automatically evaluate every change to your application.Orchestrate your CI workflow to run specific evaluations at different stages of development._Anyone with basic Python knowledge and familiarity with building LLM-based applications.

Automated Testing for LLMOps

Systematically improve the accuracy of LLM applications with evaluation, prompting, and memory tuning.

Understand development steps, from evaluation, through prompting, self-reflection, and fine-tuning, to improve your model’s reliability and accuracy. Understand development steps, from evaluation, through prompting, self-reflection, and fine-tuning, to improve your model’s reliability and accuracy.Learn how memory tuning can increase your model performance by embedding facts into your model to reduce hallucination. Learn how memory tuning can increase your model performance by embedding facts into your model to reduce hallucination.Use the Llama 3-8b model to build an LLM application that converts text to SQL with a custom schema. Use the Llama 3-8b model to build an LLM application that converts text to SQL with a custom schema._Join our new short course, Improving Accuracy of LLM Applications with Lamini and Meta. Learn from Sharon Zhou, Co-founder & CEO of Lamini, and Amit Sangani, Senior Director of Partner Engineering, Meta.Many developers have experienced frustration with inconsistent results when working with LLM applications. This course offers a systematic approach to enhance the accuracy and reliability of your LLM applications.You will build an SQL agent, add evaluation metrics to measure performance, and use prompt engineering and self-reflection to make the model perform better. Finally, you will fine-tune the model with techniques like LoRA and memory tuning that embeds facts in model weights to reduce hallucinations.In this course, you’ll use Llama’s family of open-source models.What you’ll do: Build a text to SQL agent and simulate situations where it hallucinates to begin the evaluation process.Build an evaluation framework to systematically measure performance, including criteria for good evaluations, best practices, and how to develop an evaluation score.Learn how instruction fine-tuning enhances pre-trained LLMs to follow instructions, and how memory fine-tuning embeds facts to reduce hallucinations. Break fine-tuning myths and see how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation(LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further. Go through an iterative process of generating training data and fine-tuning, learning practical tips such as adding examples, generating variations, and filtering generated data to increase model accuracy.Start improving the accuracy of LLM applications today! _This course is ideal for anyone with intermediate Python knowledge and familiarity with large language models (LLMs) looking to build more factual and precise LLM applications. 

Improving Accuracy of LLM Applications

Move your LLM-powered applications beyond proof-of-concept and into production with the added control of guardrails.

Learn the common failure modes of LLM-powered applications that guardrails can help mitigate, including hallucinations and revealing sensitive information. Learn the common failure modes of LLM-powered applications that guardrails can help mitigate, including hallucinations and revealing sensitive information.Understand how AI guardrails validate and verify your applications with input and output guards, ensuring reliable and controlled interactions. Understand how AI guardrails validate and verify your applications with input and output guards, ensuring reliable and controlled interactions.Add guardrails to a RAG-powered customer service chatbot to create a new layer of control of the applications behavior. Add guardrails to a RAG-powered customer service chatbot to create a new layer of control of the applications behavior._Join our new short course, Safe and Reliable AI via Guardrails, and learn how to build production-ready applications with Shreya Rajpal, co-founder & CEO of GuardrailsAI.The output of LLMs is fundamentally probabilistic, making it impossible to know in advance, or to guarantee the same response twice. This makes it difficult to put LLM-powered applications into production for industries with strict regulations or clients who require high consistency in application behavior. Fortunately, installing guardrails on your system gives you an additional layer of control in creating safe and reliable applications. Guardrails are safety mechanisms and validation tools built into AI applications, acting as a protective framework that prevents your application from revealing incorrect, irrelevant, or sensitive information. This course will show you how to build robust guardrails from scratch that mitigate common failure modes of LLM-powered applications like hallucinations or revealing personally identifiable information (PII). You’ll also learn how to access a variety of pre-built guardrails on the GuardrailsAI hub and are ready to integrate into your projects.You’ll implement these guardrails in the context of a RAG-powered customer service chatbot for a small pizzeria.In detail, you’ll: Explore the common failure modes of LLM-powered applications including hallucinations, going off-topic, revealing sensitive information, and generating responses that can harm your reputation.Learn to mitigate these failure modes with input and output guards that validate and verify your application’s responses.Create a guardrail to prevent the chatbot from discussing sensitive topics, such as a confidential project at the pizza shop.Develop a guardrail to detect hallucinations using a Natural Language Inference (NLI) model, ensuring responses are grounded in the truth of your trusted documents.Add a Personal Identifiable Information (PII) guardrail to detect and redact sensitive information in user prompts and LLM outputs using tools from the Guardrails hub.Set up a guardrail to limit the chatbot’s responses to topics relevant to the pizza shop, keeping interactions focused and appropriate.Configure a guardrail that prevents your chatbot from mentioning any competitors using a name detection pipeline consisting of conditional logic that routes to an exact match or a threshold check with a named entity recognition.The tools in this course will unlock more opportunities for you to build and deploy safe, reliable LLM-powered applications ready for real-world use._Anyone who has basic Python knowledge looking to enhance the safety and reliability of LLM-powered applications with practical, hands-on guardrail techniques.

Safe and reliable AI via guardrails

Learn how to systematically evaluate, improve, and iterate on AI agents using structured assessments.

Learn how to add observability to your agent to gain insights into its steps and know how to debug it. Learn how to add observability to your agent to gain insights into its steps and know how to debug it.Learn how to set up evaluations for the agent components by preparing testing examples, choosing the appropriate evaluator (code-based or LLM-as-a-Judge), and identifying the right metrics. Learn how to set up evaluations for the agent components by preparing testing examples, choosing the appropriate evaluator (code-based or LLM-as-a-Judge), and identifying the right metrics.Learn how to structure your evaluation into experiments to iterate on and improve the output quality and the path taken by your agent. Learn how to structure your evaluation into experiments to iterate on and improve the output quality and the path taken by your agent._Learn how to systematically assess and improve your AI agent’s performance in Evaluating AI Agents, a short course built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and Aman Khan, Director of Product.When you’re building an AI Agent, an important part of the development process is evaluations or evals. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically—rather than relying on trial and error. With a systematic approach, you structure your evaluations to assess the performance of each component of the agent, as well as its end-to-end performance. For each component, you select the appropriate evaluators, testing examples, and metrics. This process helps you identify any areas of improvement so you can iterate on your agent during development and in production. In this course, you’ll build an AI agent, add observability to visualize and debug its steps, and evaluate its performance component-wise.In detail, you’ll: Distinguish between evaluating LLM-based systems and traditional software testing.Explore the basic structure of AI agents – routers, skills, and memory – and implement an AI  agent from scratch.Add observability to the agent by collecting traces of the steps taken by the agent and visualizing the traces.Choose the appropriate evaluator – code-based, LLM-as-a-Judge, and human annotations – for each component of the agent. Set up evaluations for the skills and router decisions of the agent example using code-based and LLM-as-a-judge evaluators, by creating testing examples from collected traces and preparing detailed prompts for the LLM-as-a-judge.Compute a convergence score to evaluate if the example agent can respond to a query in an efficient number of steps.Run structured experiments to improve the performance of the agent by exploring changes to the prompt, LLM model, or the agent’s logic.Understand how to deploy these evaluation techniques to monitor the agent’s performance in production.By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance. _Anyone who has basic Python knowledge and wants to learn to evaluate, troubleshoot, and improve AI agents effectively—both during development and in production. Familiarity with prompting an LLM model would be helpful but not required.