In this lesson, you'll learn several techniques to make your cache more accurate. like threshold tuning, cross-encoders, LLM checks, and fuzzy matching. Let's get coding. We'll use four different strategies to improve the cache quality. The first one is called threshold sweep. In this strategy, we evaluate the cache with different thresholds to find the one that maximizes the F1 Score. Looking at the graphics, we can see that increasing the threshold leads to better Recall, but it decreases the Precision. We see that F1, because it's a function of both of them, it maximizes at an intermediate threshold value. This is usually the threshold we want to pick. The second strategy is using a cross-encoder as a reranker. Cross-encoder is a model that can capture more nuances between the two sentences. because it processes them simultaneously. In this regard, it is more expressive, but it cannot be used in place of the embedding model because it does not produce vectors that are quick to search. In this specific example, we can see the Cross-Encoder Score and compare it to the embedding distance. As similarity increases, so does the Cross-Encoder Score, or its confidence in the similarity of the pair. The way we usually use the Cross Encoder effectively in the semantic caching system is by first retrieving a set of closest cache hits for a specific user query, and then running them through the Cross Encoder. We proceed by picking the cache hit with the highest similarity score. So in summary, we can expect a sizable boost in precision, especially if we fine-tune the model on our own data. But of course, this setup also increases the cache latency of our system. Our third strategy is to use an LLM and simply ask if a pair of sentences are semantically similar. To make this work effectively, we can use a cheaper LLM to do the validation. And also, validating similarity requires less tokens than generating a full user query response. and therefore is cheaper. In practice, you can also set it up as a fallback mechanism to query cache hits which have an ambiguous distance value. This validation schema can drastically increase precision, but it comes with increased latency and token usage compared to a system without an LLM validator. Lastly, let's look at fuzzy matching, an algorithmic strategy to matching strings. Fuzzy matching usually measures the edit distance between strings, or how quickly you can transform one string to the other using deletions, substitutions, and insertions. As you can see in the example on the slide, the two sentences with small difference produce a high similarity score. And the two sentences that are completely different have a very low score. This technique can be very helpful in keyword style search domains. where we can filter easily for typos. To set this up properly, we usually place the fuzzy matching in front of our semantic caching system, and with high similarity threshold, we can match typos to corresponding previous queries, short-circuiting the whole embedding generation and vector search mechanism after that. Setting this usually leads to decreased latency and increased precision. But it can lead to false positives if not set up correctly. Let's see all of these techniques in code. Let's start again by setting up our environment. and wait for this to load. And as previously, let's also again introduce the cache wrapper, which will help us hydrate and use our cache. Let's also instantiate our data container and get an instance of all our test queries. And just as a reminder from last time, we'll hydrate our cache, get all of the matches for a distance of 0.3, and run this through the evaluator. We'll obtain the previously seen evaluation report and use this as a baseline for the metrics that we're going to obtain further. Let's now look at our first improvement technique, which is the threshold sweep. We will start by checking all of the test queries that we have with a distance_threshold of one, obtaining a full retrieval. Then, we'll run the evaluator as previously, being mindful to construct it with a special constructor from_full_retrieval. Finally, let's look at this new helper function which is called report_threshold_sweep. The evaluator already has all the information necessary to run a threshold sweep for us and report the results in this nicely formatted report. As you can see on the diagram here, we have our precision going down with higher distance thresholds and recall going up. respectively. We see that the F1 maximizes in this specific value here, which if we look at the table with metrics, maximizes at threshold 0.2828, which gives us 90% precision and 85% recall at 25% cache hit rate, 90% precision is pretty good, and we also have a decently sized cache hit rate. Looking at the confusion matrix, we can also see that most of our values lie on the diagonal, which is what we want. Now let's move on to the next technique that we're going to see. Running this code instantiates our cross encoder, which is based on Alibaba-NLP, which is a reranker based on modernbert-base. The cache wrapper also has another helper function called register_reranker, which when given a proper instance of a reranker, which can obtain from the cross encoder using this helper method. You could supply it here. and get a cache that has a reranking mechanism injected into it. But before that, let's look at an example of what the cross-encoder does, as we saw in the slides. The cross encoder is a model that can, given two pairs of sentences, give us a score of how similar they are or how relevant they are. In this case, we have What is the capital of China? versus Beijing, and we see that we have a very high similarity score. We also have sentence pairs of how to implement quick sort in python? which matches very well with Introduction of quick sort, and a bad match, which is again how to implement quick sort in python, matched against The weather is nice today, which is very dissimilar and has a very low score. Let's also see another example of a cross-encoder versus an embedding model. Here we're running the sentences both through the semantic cache and the cross-encoder. The sentences are as follows: The bank raised its interest rates. versus The river overflowed near the bank after heavy rain. They are intentionally confusing. And we can see that even though they are relatively far apart, the cross-encoder, rightfully, puts them very far apart, while the embedding model is relatively confused. It is also important to note that here a cross encoder distance is constructed by doing one minus the cross encoder score that we saw previously here. Okay. So let's now obtain results from the cross encoder and run it to our evaluator. We're running again now our cache wrapper in which we register the cross encoder. We're running it with the test queries. using a distance_threshold of one, which means that we're doing a full retrieval here. We're using num_results 10, which means that the cross-encoder is going to look into and evaluate 10 examples and rerank them. And again, we are instantiating the evaluator using the special constructor with the from_full_retrieval. Let's look at the results from the threshold sweep report. Interestingly, now we can obtain a much higher precision at 94% with relatively decent recall, 79%, and still good cache hit rate of 23%. And this we obtained at a threshold of 0.11. Moving on, let's now look at the system using an LLM validator in our cache. Because this is using an LLM, let's load our OpenAI key. And let's instantiate our LLMEvaluator. We can instantiate it similarly in the cache_wrapper by constructing it and passing it to the register_reranker method. We continue by running the cache_wrapper. that has the LLMEvaluator with a threshold that we already know it's relatively good. So this will potentially improve our results even further by filtering out false positive results. Let's run the cache results through the evaluator. As you can see, we obtained 100% precision. This means that we removed all of our false positives. Our recall dropped significantly, but we still have a relatively high cache hit rate. Now, let's run the semantic cache with the LLM validator through the evaluation that we've seen previously. We have 100% precision, which means that we filtered out all of our false positives. but we also acquired a lot of false negatives, so things that could have hit but did not because the LLM was too strict. This might be still valuable in a lot of domains, and we achieve a relatively high cache hit rate of 21%, while having 100% precision. Lastly, we arrived at the last section of our lesson. We're going to explore Fuzzy Matching. We'll start by introducing a simple function called fuzzify_string. Basically gets a string as input, a number of iterations where it introduces a random swaps within strings. So because Fuzzy Matching is a useful technique to use in keyword style searches, where the queries and the cache hits are relatively similar syntactically. We're going to augment our data set to make it more appropriate for evaluation in this specific setting. So we're going to construct a special data set. to validate and showcase this technique, where we're going to use the FAQ questions and we're going to generate pairs with varying degree of difficulty or fuzziness, which is introduced with this function, which does random swaps between neighboring characters. Let's take a look at the fuzzified queries that we've generated here. these queries at a varying degree of fuzziness. We start by a fuzziness of two, which does only two random swaps. Then we go to three, four, 10, and then a hundred where the string looks almost unrecognizable. After that, we're going to instantiate an abstraction called FuzzyCache. which has a very similar interface as the semantic cache wrapper that we have worked with so far. We can hydrate it using our FAQ data set. We can check multiple queries. Looking at the fuzzy query retrievals, we have again the query and the matches that it gets. As you can see for complex cases like these ones, they get matched to their corresponding queries rightfully. And even for this really difficult example, we still managed to map it to the correct cache entry. We'll continue by generating a valid fuzzy labeling, which basically captures all of the pairs that are valid query and cache hit matches. We'll use this to label our retrievals, and because we have this labeling, we can use it in the CacheEvaluator, where we're going to pass it here in the true_labels argument. and we're going to report the metrics for a specific distance threshold, where these retrievals, which in this case is 0.6. and we obtained 100% precision, 94% recall. and have a very high cache hit rate. But this example is very contrived in a way and rather small scale, and just serves to showcase that you can actually set up and use such a system in front of your cache. Let's also clean up our cache here. In the next lesson, you're going to implement your first AI agent that is going to use semantic caching to be fast and efficient.
