method for getting LLMs to answer questions over a user's own data. But to actually build and productionize a high-quality RAG system, it costs a lot to have effective retrieval techniques, to give the LLM highly relevant context to generate his answer, and also to have an effective evaluation framework to help you efficiently iterate and improve your RAG system, both during initial development and during post-deployment maintenance. This course covers two advanced retrieval methods, sentence window retrieval and auto-merging retrieval, that deliver a significantly better context of the LLM than simpler methods. It also covers how to evaluate your LLM question-answering system with three evaluation metrics, context relevance, groundedness, and answer relevance. I'm excited to introduce Jerry Liu, co-founder and CEO of LlamaIndex, and Anupam Datta, co-founder and Chief Scientist of TruEra. For a long time, I've enjoyed following Jerry and LLamaIndex on social media and getting tips on evolving RAG practices. So I'm looking forward to him teaching this body of knowledge more systematically here. And Anupam has been a professor at CMU and has done research for over a decade on trustworthy AI and how to monitor, evaluate, and optimize AI app effectiveness. Thanks, Andrew. It's great to be here. Great to be with you, Andrew. Sentence window retrieval gives an LLM better context by retrieving not just the most relevant sentence, but the window of sentences that occur before and after it in the document. Auto-merging retrieval organizes the document into a tree-like structure where each parent node's text is divided among its child nodes. When meta child nodes are identified as relevant to a user's question, then the entire text of the parent node is provided as context for the LLM. I know this sounds like a lot of steps, but don't worry, we'll go over it in detail on code later. But the main takeaway is that this provides a way to dynamically retrieve more coherent chunks of text than simpler methods. To evaluate RAG-based LLM apps, the RAG triad, a triad of metrics for the three main steps of a RAG's execution, is quite effective. For example, we'll cover in detail how to compute context relevance, which measures how relevant the retrieved chunks of text are to the user's question. This helps you identify and debug possible issues with how your system is retrieving context for the LLM in the QA system. But that's only part of the overall QA system. We'll also cover additional evaluation metrics such as groundedness and answer relevance that let you systematically analyze what parts of your system are or are not yet working well so that you can go in in a targeted way to improve whatever part needs the most work. If you're familiar with the concept of error analysis and machine learning, this has similarities. And I've found that taking this sort of systematic approach helps you be much more efficient in building a reliable QA system. The goal of this course is to help you build production-ready, write-based LLM apps. And important parts of getting production ready is to iterate in a systematic way on the system. In the later half of this course, you gain hands-on practice iterating using these retrieval methods and evaluation methods. And you also see how to use systematic experiment tracking to establish a baseline and then quickly improve on that. We'll also share some suggestions for tuning these two retrieval methods based on our experience assisting partners who are building RAG apps. Many people have worked to create this course. I'd like to thank, on the LlamaIndex side, Logan Markehwich, and on the TruEra side, Shayak Sen, Joshua Reini, and Barbara Lewis. From DeepLearning.ai, Eddie Shyu and Dialla Ezzeddine also contributed to this course. The next lesson will give you an overview of what you'll see in the rest of the course. You'll try out question-answering systems that use sentence window retrieval or auto-merging retrieval and compare their performance on the RAG triad, context relevance, groundedness, and answer relevance. Sounds great. Let's get started. And I think you people are really clean up with this RAG stuff. Laugh on it.
Week 1: Building and Evaluating Advanced RAG