Sign in to continue learning
Context Engineering
7h55m
Throughout Module 2, we're building a coding assistant to answer questions about our repository. In this lesson, we will study chunking text, which we can use to process the documentation. And then in the next lesson, we will learn to chunk code, which will involve different tools and techniques. Remember, the repository contains two kinds of data, the code that runs the application, its front-end dashboard, and its back-end database, along with text in the documentation, like the README file you see here. Our dataset just contains code and text. In particular, the documentation doesn't have any images or other data readable by you or me, but not a language model. Converting different modalities of data into text for a language model is called parsing, a step that comes before chunking. In this lesson, we don't need to use parsing, because we already have text in the documentation. However, in the next lesson, we will need to use parsing to break down code into components. That lend themselves to chunks. When your coding assistant searches across the text in the documentation, it won't be through bash commands like the grep or jq commands from Module 1. Instead, the harness will have tools to query a database for chunks retrieving parts of files like paragraphs or sentences, as opposed to whole files like the entire README. Chunking will help us better use the embedding model and the language model. Both models have context length. The embedding model, like the language model, has a limited context window constrained by context length, so a chunk can only hold so many tokens. Otherwise, the embedding model can't convert characters and strings into numbers and arrays. Additionally, the language model has effective context length, as mentioned in Module 1. If the harness retrieves an entire file, like the whole README file, but only some paragraphs or sentences are relevant to a user prompt, then the context window of the language model fills up with lots of tokens leading to issues like slower responses, higher costs, and degraded attention. So, chunking allows us to stay within constraints on context length for both the embedding model and language model. Additionally, chunking helps us manage context for the language model, but the size of each chunk is a tradeoff between small and large. We can't make chunks too large, otherwise a chunk might not fit into the embedding model or lead to context rot in the language model, but too small, and a chunk loses surrounding context. Commonly, the answer to a question can sit in one chunk, but the words matching the user prompt sit nearby in another chunk. Let's look at an example from the documentation. Note that for some methods of search, like regular expressions, the application will time out after 5 seconds. Suppose we split the paragraph into 4 sentences. Now let's imagine the user asks, does the search feature in the dashboard have a time out? The harness will search the database for chunks related to the user prompt. Chunk 2 contains, the default value is 5 seconds, but it never mentions rejects time out. That phrase only appears in chunk 1, so there's no guarantee that both chunks will be retrieved, meaning the model might not get all the necessary information to answer 5 seconds. With 4 sentences, as opposed to 1 paragraph, the query returns partial matches searching for time out might rank chunk 1 highest because it mentions redex time out. Even though chunk 2 has the actual answer, it's relevant, but the retrieved content doesn't answer the question. So the model might not provide a response like answering I don't know, or not provide a confident response like answering the default value appears to be 5 seconds, but I'm not certain what setting this refers to, or might provide hallucinated details like answering the default value is 30 seconds. We nicknamed this context poisoning in module 1. To troubleshoot, we could retrieve more chunks. We might retrieve 10 or 20 chunks instead of 3 or 4. That number, how many chunks we return, is called top K. It's a choice about querying, so we'll come back to it in a later lesson, but we're learning that decisions about chunking impact those later decisions in querying. The trouble is, more and more chunks could lead to issues, like duplicate or nearly duplicate data. For example, suppose the documentation additionally contains the sentence, the semantic timeout sets how long a semantic search runs before the app stops it. The default value is 1 second. That makes sense. We will learn more about indexing in databases later in module 2. In particular, we will learn that semantic search over databases can be much faster than regular expression search over a dataset, so 1 second, as opposed to 5 seconds, is a reasonable default for timeout. You might again split the paragraph into sentences, so in addition to the 4 chunks from before, including the 2 about timeout for regex search, there are 2 chunks in the database about timeout for semantic search. When the user asks that original question, does the search feature in the dashboard have a timeout, the model might answer, the default value for semantic search is 5 seconds and for regex search is 1 second by matching chunk 1 to chunk 6 and chunk 2 to chunk 5 in its response to the user. We nicknamed this context confusion in module 1. We could have avoided possible issues, like context poisoning or context confusion, chunking by paragraphs as opposed to sentences. That way, the harness can retrieve the value 5 seconds in the same chunk as the phrases regex search and timeout, namely the answer 5 seconds, together with its surrounding context regex search and timeout. Commonly, we use fixed-length chunking or recursive chunking, where fixed-length splits by a count of tokens and recursive splits at special characters, like new lines or periods. There are other methods as well, where we might employ embedding models to check the similarity or dissimilarity between blocks of text. You can learn more about alternatives in the reading items for module 2. Chunking by length involves splitting the text by token length or character length. It's a simple way to break text into chunks of a fixed size, like 400 characters, which would be around 100 tokens, while each chunk has the same size. Or even words could split in half. Chunking recursively at special characters involves splitting at natural breaks in the text, like a blank line between paragraphs or a period between sentences, and you keep splitting until each chunk fits within the limit. The breakpoints you choose depend on the document. Our readme is marked down, with headers marking each section, so we can split on those headers, as opposed to just new lines. We will discuss chunking markdown files in more detail next lesson. For both fixed length and recursive chunking, we have to set the chunk size and chunk overlap. With overlap, neighboring chunks share a little text, usually 5-10%, or about 50-100 characters. So when regex timeout and 5 seconds fall on either side of a split, they still come back together, despite their small size. Yet too much overlap will return the same sentences twice, and that redundancy means you're storing the shared text twice in the database. To help us make decisions about the method, size, and overlap of chunks, we can use metrics, particularly precision and recall. We have to choose a collection queries about our dataset, like, does the search feature have a timeout? We mark the chunks that hold the answer to each. That's the ground truth. Then we search the database to retrieve chunks, and compare the retrieved chunks. Add a cutoff like the top 5 or top 10 to the ground truth. Precision is the fraction of retrieved chunks marked as ground truth for the query. Say 8 of 12 retrieved chunks are marked. That's about 67% precision. Recall is the fraction of ground truth chunks retrieved for the query. Say we retrieved 8 chunks, but there were 10 marked in the ground truth. That's 80% recall. So precision and recall tell us different things. Precision how reliable things are. Recall how thorough things are. Remember we calculate each metric at a cutoff. Our top K. Think of it like how many guesses we give ourselves. To keep track of the choice of K, we'll write precision at K and recall at K, as opposed to just precision or recall. With more guesses we might have higher recall, but lower precision. That's one of many decisions we can compare, through metrics. For example, larger chunks allow us to keep blocks of text, like paragraphs, together. So regex timeout and 5 seconds sit in one chunk, meaning precision goes up. But as chunks get too large, the paragraph that holds the answer is more likely to get buried among other paragraphs, and recall goes down. Because at most K can be retrieved, usually there's a tradeoff between precision and recall. After all they measure different things. But sometimes overlap between chunks can improve. Both chunks can remain small without losing surrounding context, provided by the overlapping text. Precision and recall help you decide the size of the chunks, the overlap between them, and where you split. But there's no one-size-fits-all approach to chunking. You have to pick an approach, and calculate metrics. The right approach will depend on the data. That's why you have to look at your data to see for yourself that the documentation has sections separated by double hash symbols, and the code has functions between the def and return symbols. Let's talk more in the next lesson about code, where we'll revisit recursive chunking.