The Cost of Exact-Match Caching Failures
Traditional application caching relies on exact keys. If a user queries your system with an extra space, a minor typo, or a rearranged sentence structure, a standard Redis lookup misses. You send the payload to the language model anyway, paying full token price for an answer you essentially already generated five minutes ago. Exact-match caching is useless for natural language inputs because human typing is gloriously messy.
To fix this, you need semantic caching. Instead of hashing the raw prompt string, you generate a vector embedding of the input text and compare its cosine similarity against cached vectors stored in your database. If a stored prompt shares a high enough mathematical proximity to the incoming request, you skip the provider entirely and return the cached completion. You can combine this approach with techniques found in Caching Embeddings and LLM Completions in Production to keep database overhead manageable.
Most developers don't realize how quickly small test suites rack up massive bills during active feature development. You run your test runner ten times, and suddenly you've burned through thousands of input tokens for identical assertions. It's painful, and it's entirely avoidable. By intercepting these requests at the service container level, you save both time and money.
Scale compounds these issues rapidly in a production environment. A popular feature can easily generate millions of requests per day, and if half of those requests ask slightly phrased variations of the same question, you are wasting half your budget on redundant API calls. Standard string matching algorithms fail completely here because they require lexical overlap. Semantic caching bridges this gap by mapping meaning rather than characters, ensuring that different sentence structures pointing to the same intent resolve to the exact same cached response.
Building the Similarity Check Pipeline
You need a storage layer that handles vector indexing efficiently. PostgreSQL with the pgvector extension is usually the pragmatic choice if you are already storing relational data, avoiding the operational pain of managing a dedicated vector database for simple prompt caches. Here is how you can set up a basic semantic cache table in your database schema.
CREATE TABLE semantic_prompt_caches (
id SERIAL PRIMARY KEY,
prompt_text TEXT NOT NULL,
embedding VECTOR(1536) NOT NULL,
completion_text TEXT NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX ON semantic_prompt_caches USING hnsw (embedding vector_cosine_ops);
When an incoming request arrives, your application must first convert the prompt into an embedding vector using a cheap, fast embedding model. You then query the database for the nearest neighbor using the cosine operator. If the distance falls below your threshold, you bypass the external API call.
import os
from openai import OpenAI
import psycopg2
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
def get_embedding(text: str) -> list[float]:
response = client.embeddings.create(
input=text,
model="text-embedding-3-small"
)
return response.data[0].embedding
def find_cached_completion(prompt: str, threshold: float = 0.15) -> str | None:
vector = get_embedding(prompt)
connection = psycopg2.connect(os.environ.get("DATABASE_URL"))
cursor = connection.cursor()
cursor.execute(
"SELECT completion_text, embedding <=> %s::vector AS distance FROM semantic_prompt_caches ORDER BY distance ASC LIMIT 1",
(str(vector),)
)
row = cursor.fetchone()
cursor.close()
connection.close()
if row and row[1] <= threshold:
return row[0]
return None
Don't skip the index creation step. Without an HNSW or IVFFlat index, your database performs a sequential scan across every stored vector. That turns a sub-millisecond lookup into a multi-second bottleneck, completely defeating the purpose of caching. Building out this pipeline also requires careful monitoring of your embedding generation latency, because if your embedding model takes longer to run than a small local model completion, you might inadvertently slow down simple queries.
You should also implement a write path that stores new completions alongside their generated embeddings. Every time the external provider returns a valid response, your application calculates the vector representation of the prompt and inserts both strings into the cache table. This ensures the cache grows organically with actual usage patterns, targeting the exact topics your users care about most.
Handling Cache Invalidation and Edge Cases
Semantic caches are notorious for serving stale data when underlying context changes. If your prompt includes user-specific session data, application state, or time-sensitive instructions, caching the completion globally will cause severe bugs. You must namespace your cache keys or restrict semantic matching strictly to stateless, pure transformation prompts. For tasks that require precise outputs, you should also review LLM Token Counting Cost Control in Laravel and Next.js to monitor exact usage before caching policies trigger.
Another major gotcha is the distance threshold. Set it too strict, and your hit rate drops to zero because almost nothing matches closely enough. Set it too loose, and the model returns answers to completely different questions that happen to share a similar vector space. Tune your threshold against a historical test suite of real user prompts rather than arbitrary guesses. If your application handles high concurrency, wrap your insertion logic in unique constraints or background jobs so you don't thrash your database with duplicate embedding calculations for simultaneous identical requests.
Memory consumption scales linearly with your cache retention policy. You'll want to implement a routine pruning job that drops records older than a specific window, or entries that haven't registered a cache hit in weeks. Otherwise, your vector table bloats, disk I/O climbs, and query performance degrades right when your traffic spikes. Stale data accumulation is a silent killer of vector stores, and routine maintenance is the only defense against runaway disk consumption.
