Semantic Caching for LLM Requests: Similarity Thresholds, Measured

The test

On 9 October 2026 I embedded 50 texts with each of two models through the Gemini API (embedContent): 10 base questions that play the role of the cache, 20 rephrasings of them (two per question), 10 near-misses (one per question, with most words shared but a different intent) and 10 unrelated questions. Embeddings were compared with cosine similarity in NumPy; both models returned 3,072-dimensional vectors. For each query I took the most similar base question and called it a hit if its similarity reached the threshold, and I counted it as a correct hit only if it was the right base question. Examples:

What the similarity scores looked like

Unrelated questions are easy: they stay well below everything else. The hard part is the overlap between rephrasings and near-misses. With gemini-embedding-2 the weakest rephrasing scored 0.802 while the strongest near-miss scored 0.887, so no threshold between them gets all 20 hits and all 10 rejections. The three highest-scoring near-misses were "How do I import my data from CSV?" (0.887, matched to the export question), "How do I remove a teammate from my workspace?" (0.878, matched to the add question) and "How do I disable two-factor authentication?" (0.850, matched to the enable question).

What each threshold would have done

Read the two models side by side: to avoid every wrong answer, gemini-embedding-2 needed 0.90, where it served only 7 of 20 rephrasings; gemini-embedding-001 needed 0.88, where it served 1 of 20. The same number means different things for different models. Unrelated questions were never served at any threshold from 0.70 up. No rephrasing was ever matched to the wrong base question.

The latency side

An embedding call is a network request. Over the 50 calls per model in this test it had a median of 569 ms (95th percentile 715 ms, slowest 760 ms) for gemini-embedding-2 and a median of 545 ms for gemini-embedding-001. For comparison I timed five short support-style answers from gemini-3.5-flash: 7.97 s for the first call and about 2.0 s for the other four (median 2.09 s). The arithmetic follows from those numbers: every request pays the 0.57 s embedding, a hit saves the 2.09 s generation, so the average latency improves only when the hit rate is above about 0.57 / 2.09, or 27%. The cache lookup itself, a matrix product over a few vectors, is negligible next to the embedding call. The figures depend on your network, region and models, so measure them yourself.

A minimal cache, with the threshold as a parameter

import numpy as np

class SemanticCache:
    """In-memory semantic cache: cosine similarity over unit vectors, one threshold, optional TTL."""
    def __init__(self, embed_fn, threshold, ttl_seconds=None):
        self.embed, self.threshold, self.ttl = embed_fn, threshold, ttl_seconds
        self.vectors, self.answers, self.stored_at = [], [], []

    def _unit(self, text):
        v = np.asarray(self.embed(text), dtype=float)
        return v / np.linalg.norm(v)

    def add(self, question, answer, now):
        self.vectors.append(self._unit(question)); self.answers.append(answer); self.stored_at.append(now)

    def lookup(self, question, now):
        if not self.vectors:
            return None, 0.0
        sims = np.vstack(self.vectors) @ self._unit(question)
        i = int(np.argmax(sims))
        expired = self.ttl is not None and now - self.stored_at[i] > self.ttl
        return (None if expired or sims[i] < self.threshold else self.answers[i]), float(sims[i])

I ran this class with threshold 0.80 on the stored embeddings. "How can another person join my workspace?" returned the cached answer (similarity 0.914), the unrelated capital-city question returned nothing (0.551), an expired entry returned nothing, and "How do I remove a teammate from my workspace?" also returned the answer cached for adding a teammate (0.878). That last line is the failure this article is about.

Practical rules

1. Calibrate with your own near-misses. Collect questions from your logs where one word flips the meaning (add and remove, enable and disable, import and export, free and paid) and make sure they fall below your threshold.
2. Prefer misses to wrong hits. A miss costs one model call; a wrong hit gives the user a confident answer to a different question. Start with a high threshold and lower it only while the near-miss list stays clean.
3. Only cache answers that stay valid. Add a time limit, and put the model name and system prompt in the cache key so a changed prompt does not serve old answers.
4. Track both rates. Log the hit rate and the share of hits that a human or a check flags as wrong. I did not test a second-stage check (for example asking a model whether two questions mean the same), so I cannot say how well it works.

What I did not test

One English support-style set of 40 questions, two Gemini embedding models, no vector database (the 10 cached vectors sit in a NumPy array), no OpenAI or open-source embedding models, and no non-English text. Qdrant, Redis and other stores do the same similarity search; the thresholds above come from the embeddings, not from the store. A real FAQ will have more cached questions and more overlap, which makes wrong hits likelier.

Sources and Further Reading

• Embeddings — Google AI for Developers
• Cosine similarity — Wikipedia