Orchestrate a RAG prototype using Airflow: transform your code into pipelines consisting of modular tasks and schedule them using time-based and data-aware scheduling.
We'd like to know you better so we can create more relevant courses. What do you do for work?
Instructors: Kenten Danas, Tamara Fingerlin
Orchestrate a RAG prototype using Airflow: transform your code into pipelines consisting of modular tasks and schedule them using time-based and data-aware scheduling.
Apply dynamic task mapping to run tasks efficiently in parallel and automatically adapt to new data sources.
Build robust pipelines by adding automatic retries and failure notifications to handle errors gracefully.
Learn to build and orchestrate a RAG pipeline in Orchestrating Workflows for GenAI Applications, built in partnership with Astronomer and taught by Kenten Danas (DevRel Senior Manager) and Tamara Fingerlin (Developer Advocate).
When building generative AI applications, itâs common to start in a Jupyter notebook or a Python script but to move into production, your AI workflows need to run reliably, adapt to changing data, and gracefully recover from failures.
In this course, youâll learn how to turn a Retrieval Augmented Generation (RAG) prototype into a robust, automated pipeline using Airflow 3.0, a leading open-source orchestration tool.
Youâll build two workflows for a typical RAG application: one that ingests and embeds book description texts into a vector database, and another that queries that database to recommend books.
Along the way, youâll learn how to break workflows into discrete tasks, schedule pipelines using both time-based and data-aware triggers, and process tasks in parallel with dynamic task mapping. Youâll also add retries, alerts, and backfills to handle failure scenarios. Lastly, youâll explore how you can apply these best practices to other real-world GenAI applications such as batch inference. All of this is done using Airflow dags, which are pipelines, made up of tasks that run in a specific order, each with clear code logic and task dependencies.
In detail, youâll:
By the end of this course, youâll be able to design, build, and automate GenAI workflows using Airflow 3.0, ready for production.
This course is for AI builders who want to automate and deploy their GenAI prototypes more reliably. No Airflow experience is required, just familiarity with Python and an interest in moving to production-ready workflows.
Introduction
From Notebook To Pipeline
Your RAG Prototype
Building a Simple Pipeline
Turning your Prototype into a Pipeline
Scheduling and Dag Parameters
Make the Pipeline Adaptable
Prepare to Fail
GenAI pipelines in Real Life
Conclusion
Optional: How to Set up a Local Airflow Environment
Appendix - Resources, Help, and Downloads
Course access is free for a limited time during the DeepLearning.AI learning platform beta!
Keep learning with updates on curated AI news, courses, and events, as well as Andrewâs thoughts from DeepLearning.AI!