In the previous video, we discussed some tips for evaluating agentic search. Using those evaluations, you can improve your agent through more reliable tool calling. But there are other issues related to the data sources, like chunks that split apart blocks of code in the middle. Up to this point, we haven't talked too much about the data sources themselves. Sometimes those sources are ready-made for us, and we use a prescribed set of tools to reach them. In those cases, we don't have much say over those data sources, particularly the preparation of the data, for example, chunking. This may all sound like data engineering. But when you build up your data sources, that's an important lever to improving your search system, making it an important part of context engineering. So, in this video, we will discuss three techniques for data preparation: chunking, parsing, and metadata. Then, we will use two metrics, precision and recall, to evaluate these techniques and measure their improvement on search. Let's start with a recap of chunking. Because embedding models can accept only a limited amount of input, we have to break long text or code into smaller chunks. That Chunk Size is defined by you. If consecutive chunks have no overlap, a boundary can split a sentence or even a word. Instead, we can create chunks with a rolling window of text so neighboring chunks share some content. The amount of overlap is also user-defined as a number of characters. Depending on the setting, it might correspond to a few words or a few sentences. Even when a harsh boundary occurs, the shared text can preserve the missing word or sentences in a neighboring chunk. That creates continuity between consecutive pieces and helps the model connect their context. Overlap is useful, but it's another engineering choice that we need to measure. Too much overlap stores and may retrieve the same information repeatedly. Another way to prevent unnatural splits is to use the structure of the text itself. For plain text, we can look for natural boundaries: newline, newline, between paragraphs, or newline between lines, punctuations such as periods, question marks, or exclamation marks at the end of sentences, and lastly, whitespace between words. This is commonly called recursive text splitting. First, try to split at paragraph boundaries. If a piece is still too large, split at line boundaries. If it remains too large, split at sentence boundaries, and finally, at word boundaries. This approach is useful especially for plain text But when text has more structure, we can use boundaries specific to its format. Markdown, for example, uses headings such as H1, H2, and H3 to identify sections and subsections. It also uses formatting markers, such as underscores for emphasis. The headings reveal natural relationships and hierarchy in the document. A heading tells us the text beneath it belongs to a named section, while a subheading places it in a more specific part of that section. Splitting along that structure can produce more meaningful chunks than relying only on generic new lines or punctuation. Code also has structure. Building blocks like functions and classes are natural chunks of code. But code is tricky because those can be nested or linked together. A class may contain several methods, and a function may reference a global variable elsewhere in the file. If we chunk code only by character count, we ignore those relationships. We might split a class in half or separate functions that belong to the same class. Instead, we can use a parser. A parser understands the syntax of a programming language, such as Python, and produces a data structure called an abstract syntax tree, or AST. The tree captures the code's nesting from top to bottom. It represents recognizable constructs, such as import statements, classes, functions, and the relationships between them. A class can appear as a parent with its methods nested beneath it. Like recursive text splitting, the chunk can move from larger constructs to smaller ones until each piece fits within the character limit. But now it follows known code boundaries. A parser can expose useful source information, such as the start and end positions of each construct. Such metadata is important because, as we saw in the previous video, chunks can be difficult to reassemble without it. We try to keep chunks small, so retrieval stays targeted and avoids distracting information. But when chunks become too small, we may lose where they fit within a file or across files. Metadata, such as the file path and line number, can help. Once we add those labels, we can also use them for filtering. Metadata filtering complements ranked methods, such as lexical and semantic search, by reducing the candidate set before ranking. Filtering can remove large amounts of unrelated data, so search runs over a smaller, more relevant subset and returns more targeted results. We will talk more about metadata filtering in the next module, where document ID will help us with deduplication. Metadata can sit outside the chunk as tags, or it can be added to the text of the chunk, often as front matter. One interesting use is to augment a small chunk with surrounding information. Beyond direct overlap, we might add a short summary of the preceding chunk, the following chunk, or even the whole file. So the chunk can carry all this extra information of its surroundings, rather than just the limited information from the overlap, making it more informative and potentially easier to retrieve for a relevant query. Now that we've discussed three important techniques for data preparation, let's talk about how to evaluate their impact on search. For evaluations, we can use different metrics, and here we're going to focus on precision and recall. Maybe you've heard of these metrics in different settings, so let's discuss them through an example to quickly remind ourselves how they work in this setting. To calculate precision and recall, start by looking at the data in building an evaluation set. Identify tasks that are representative of how people will use the agent, especially tasks that are both common and impactful. For each task, create a representative query and identify the source material that the system must retrieve. That becomes the Ground Truth for the query. Let me show you what I mean. Suppose that after looking at our data, we decide that a representative task is understanding a feature of the feedback form. while it processes the submitted data, and before it shows the confirmation page, does that spinner have a timeout?" We inspect the data and manually identify the source material that is relevant to that query. After the data has been prepared, we map that material to the subset of chunks that search should return for this question. Let's say we identify 10 Ground Truth chunks. They are snippets from several files, and can include both documentation and code. When the agent searches the vector store, it decides how many chunks to retrieve. For evaluations, we want to fix that number in calculation of the metric. Here let's fix k at 12. Because precision and recall depend on that cutoff, we report them as Precision@k and Recall@k. In this case, Precision@12 and Recall@12. Precision asks, of the chunks we retrieved, what fraction are relevant according to our ground truth? Suppose semantic search returns 12 chunks, and 8 of those 12 are in the ground truth. Precision @12 is 8/12, or about 67%. Recall asks the question from the other direction. Of all the chunks in ground truth, what fraction did we retrieve? We have 10 ground truth chunks, and retrieval found 8 of them, so Recall @12 is 8/10, or 80%. So precision tells us how much of what we retrieved was relevant, and recall tells us how much of the relevant ground truth we successfully retrieved. Remember, higher is better for both precision and recall, but improving both can be tough because there are trade-offs between them. Let's wrap up this video by discussing some practical ways to increase them. Suppose Precision@k is low. Maybe the chunks are too large, and relevant information is mixed up with irrelevant information. We can use smaller chunk size, but we can't take it too small because smaller chunks can lose the surrounding context, making it harder to retrieve associated chunks among the ground truth, which would lead to lower Recall. Some overlap can connect associated chunks that come one after the other. But we can't have too much overlap, because that fills up the top-k with redundant information, leading to low Recall. So best to keep the overlap small, and instead use metadata to identify chunks in their surrounding context. There's a trade-off between precision and recall, and depending on the situation, you might want to focus more on one than the other. To make informed decisions, the really important habit is to develop the practice of identifying errors, building evals to measure the performance on a set of reference cases, and then using the eval results to implement changes. This is critical to rigorously improving your agents. To offer more perspective on handling data in context engineering, particularly huge amounts of data, join me in the next video to hear from our guest speaker, Drew Breunig.
