Sign in to continue learning
Context Engineering
7h55m
Last lesson, we chunked text from our documentation. By looking at our data, we could make more informed decisions about the method of chunking, the size of chunks, and the amount of overlap between chunks. Remember the method of recursively chunking at special characters, particularly punctuation like backslash n, backslash n for paragraphs, then backslash n for lines, then period for sentences, and lastly whitespace for words. With plain text, these characters make sense because they divide the text into smaller and smaller parts. But our README is Markdown. It has a particular structure, like section headers marked with pound symbols, one symbol for H1 headers, two for H2 headers, and so on. This allows us to recursively chunk at hash for sections, then hash hash for subsections, down to smaller and smaller parts. Headers provide additional context. If we split at a particular header, then we can record the name of the section and subsection. Compare this to overlap between consecutive chunks of text. But instead of overlap, which leads to redundancy across chunks, we're adding metadata like section and subsection into the chunk to capture surrounding context. For example, take a chunk from our README file, like from under how search works, then searching the collection. We can write both phrases at the top of the chunk. Now the chunk contains more context, indicating its place in the file. So when the model reads it later, it knows these lines are about how search works and not about installation, configuration, or other things regarding our application. We can apply the same approach to code, keeping related blocks together, along with a record of where the block sits among the rest of the code. But chunking code is more challenging than chunking text. There are different places to split the code, maybe around small statements or large modules. Language models can understand code, because code contains the same characters as text. But code and text have different structures. We can take advantage of that structure by parsing the code into recognizable components like functions and classes, and then forming chunks from combinations of components like all functions in a particular class. Code contains many components, imports, loops, assignments. These components depend on the programming language. Here, we used Python for our app. The Python interpreter breaks down code into components to make sense of them as executable commands. There's a useful data structure called an abstract syntax tree, abbreviated AST, that maps out code into these components, here, nicknamed nodes. Tree refers to the nesting of nodes within nodes. There's a helpful tool called TreeSitter that we'll use in the lab. Like the Python interpreter, TreeSitter can parse a .py file to build an AST from code. Comments and whitespace and indentation, none of these will appear in the AST. They're not target components for the nodes of the tree. We get to choose target components, ranging from lower-level components like assignment of variables to higher-level components like classes, and everything in between, like if statement, try statements, so on. For example, take our semantic search strategy class. It has a validate underscore query method that checks the query, a search method that runs the search, and a helper that builds the metadata filter. A parser like TreeSitter will recursively build the AST, from parent nodes down to child nodes, through each line of code, until there are no more target components. Actually, there are two orders to parsing, preorder and postorder. Different orders for parsing lead to different sorts of chunks. In preorder, we order the nodes from largest to smallest, meaning semantic search strategy comes before validate underscore query and the other methods. In postorder, we go the other way. We order the nodes from smallest to largest, meaning semantic search strategy comes after validate underscore query and the other methods. Having parsed the classes, methods, and so on into nodes of the tree, we have to decide which nodes to include in chunks, and where, with the intent of putting related nodes together in the same chunk. Preorder is helpful to make chunks from the larger nodes, but not the smaller nodes. So we would have a chunk for semantic search strategy, but not validate underscore query and other methods. Postorder is helpful to make chunks from the smaller nodes, along with the larger nodes. So we would have chunks for child nodes, like validate underscore query, and also a chunk for semantic search strategy. Note validate underscore query would appear in two chunks, its own chunk, and the chunk for the class around it. Remember this to overlap between different chunks of text in the last lesson. Postorder can help glue together chunks of code. In the same way, overlap glues together chunks of text. Suppose a target component is too large for a chunk, meaning the number of tokens in the node is not within our limit, maybe even the context length of our models. This might happen with a function that spans hundreds of lines, or a class that spans thousands of lines. We have to split the node apart to avoid splitting in an improper way, like down the middle of a line of code. We can use a buffer, meaning we read the code one line at a time, adding each line to temporary storage nicknamed the buffer. And when the next line would push it over the model's token limit, we flush the buffer into a chunk and start a fresh one. Note there's a class called RegexSearchStrategy. That's closely related to semantic search strategy. In particular, it has a search method as well. So if we chunked the code in postorder, then there would be two chunks with same name and similar code. But in two different classes, corresponding to separate methods for search. This is like the context confusion we discussed in the last lesson. To avoid context confusion, we can tag each chunk with metadata, indicating file name and line number, and anything that might help useReassembleTheCode from the chunks of code. For example, semantic search strategy. Its file path is services slash search underscore service dot py, and its line range is lines 47 to 85. So the model can tell the two search methods apart. A chunk can carry different metadata tags, its language, like Python or TypeScript, the name of the repository, and the branch in it, the symbol type, like function or class, and timestamps. When it was created or updated, we could include the metadata in the chunk itself. That was the approach to Markdown, where we added identifiers like section and subsection alongside the text in the chunk, but we don't want to add text to code outside of a comment. Instead, we could tag the chunk with metadata. Databases like Chroma allow users to tag the data with their own fields, here language, path, timestamp, and so on, rather than add context to the chunk to help semantic search. We tag the chunk to allow for filtering on metadata. Filtering keeps chunks that match particular criteria. For example, all chunks with path equal to services slash search underscore service dot py and line number less than 300. We could even use a regex pattern to find every place a function like validate underscore query is called, and line number less than 300. We could even use a regex pattern to find every place a function like validate underscore query is called. Faceting goes further by counting the number of occurrences. It groups the chunks and counts them by a metadata tag. Our app has a stats view that does this. It counts how many chunks are functions, how many are modules, and so on. The count of these occurrences is more informative because it tells the agent not only what matched, but to what extent. Remember that we are studying three methods of search in Module 2. Already we are seeing different ways these methods can come together with the right preprocessing of the data, namely parsing and chunking. Now that we have learned to preprocess text and code, we move on to embedding, indexing, and querying. And that starts with lexical search, matching code, not just metadata, by exact phrases.