Caching Embeddings and LLM Completions in Production

If you connect an LLM to your application without a caching layer, you're paying a tax on every request. Generating embeddings with OpenAI's text-embedding-3-small costs money, but the real pain is latency. A vector call takes anywhere from 80ms to 250ms. A gpt-4o completion takes 800ms to 3000ms. When users reload a dashboard or search the exact same query, waiting for roundtrips to external APIs makes your product feel slow.

The standard answer is to hash the prompt string with MD5 or SHA-256 and store the response in Redis. That works until a designer updates a system prompt to change bullet points to numbered lists, or an engineer adds a trailing newline to a prompt template. Instantly, your entire cache becomes useless. Every cached item misses, API latency spikes, and your billing dashboard explodes.

You don't have to choose between fresh prompts and high cache hit rates. You need a two-tier key strategy that separates text representation from prompt execution instructions.

Why Naive Hashing Fails in Production

Consider a retrieval-augmented generation (RAG) feature. When a user submits a query, two separate LLM API operations occur:

1. Converting the user's query into a 1536-dimension float vector (Embedding).
2. Passing the query, retrieved contexts, and system rules to a chat model (Completion).

If you bundle these into a single cache key based on the final combined prompt string, any small edit to your system instructions invalidates both steps. That's a huge mistake. The vector representation of the phrase "How do I reset my password?" never changes, regardless of whether your system prompt instructs the model to sound like a pirate or a corporate support agent. The embedding is deterministic and static based on the input text.

The solution is to decouple vector caching from completion caching completely.

Decoupling Vectors and Completions in PHP 8.3 and Laravel 12

In PHP 8.3 and Laravel 12, we can build a cache driver specifically optimized for float vectors. Instead of storing embeddings as bloated JSON arrays like [0.0023, -0.0124, ...] which take up roughly 28KB per vector, we can use binary packing.

A 1536-dimension vector consists of 32-bit single-precision floats. Using PHP's pack('f*', ...$vector), we compress that array into exactly 6,144 bytes of binary data. That cuts your Redis RAM footprint by nearly 80% and speeds up serialization.

Here is a complete EmbeddingCacheService implementation designed for Laravel 12 that normalizes text inputs, hashes only the raw payload, and stores packed binary vectors in Redis.

namespace App\Services; processText; use Illuminate\Support\Facades\Redis; use Normalizer;  class EmbeddingCacheService {     private string $model = 'text-embedding-3-small';      public function getEmbedding(string $text): array     {         $normalized = $this->normalizeText($text);         $hash = hash('sha256', $normalized);         $cacheKey = "embed:v1:{$this->model}:{$hash}";          $cached = Redis::get($cacheKey);         if ($cached !== null) {             // Unpack binary string back into array of floats             return array_values(unpack('f*', $cached));         }          $vector = $this->fetchFromOpenAi($normalized);          // Pack floats into 32-bit binary representation         $packed = pack('f*', ...$vector);         Redis::setex($cacheKey, 86400 * 30, $packed);          return $vector;     }      public function normalizeText(string $text): string     {         // Strip extra whitespace and convert to lowercase         $clean = trim(preg_replace('/\s+/', ' ', $text));         $clean = mb_strtolower($clean, 'UTF-8');                  // Normalize Unicode characters (Form C)         if (class_exists('Normalizer')) {             $clean = Normalizer::normalize($clean, Normalizer::FORM_C);         }                  return $clean;     }      private function fetchFromOpenAi(string $text): array     {         // Pretend API call here returning array of floats         return array_fill(0, 1536, 0.01234);     } }

Notice the text normalization step. Lowercasing, stripping redundant space, and running Unicode normalization ensures that variations in user copy-paste don't trigger unnecessary API calls. A user typing "Find invoice" and another typing "find invoice " will hit the exact same cached vector key.

Structuring Completion Cache Keys That Survive Prompt Edits

Now let's tackle the completion layer. A completion depends on four independent inputs:

• System Prompt Template
• User Payload / Context Documents
• Model Name (e.g. gpt-4o-mini)
• Hyperparameters (Temperature, Top-P)

If you update your system prompt template, you don't want to clear your vector cache, but you do want to update your completion cache. To manage this cleanly without manually purging Redis keys every time you deploy prompt changes, include a system prompt hash inside the completion cache key namespace.

Here is how you construct the key in Laravel 12:

$systemPrompt = config('prompts.support_bot'); $systemHash = substr(hash('sha256', $systemPrompt), 0, 12); $userContentHash = hash('sha256', $embeddingCacheService->normalizeText($userInput));  $completionKey = "comp:v1:{$model}:sys_{$systemHash}:t_{$temperature}:{$userContentHash}";

Look at what happens when you edit your system prompt template in your codebase:

1. The $systemHash changes from sys_a1b2c3d4e5f6 to sys_f9e8d7c6b5a4.
2. Old completions naturally age out according to your Redis TTL (e.g., 24 hours).
3. The new system prompt creates new completion cache keys on miss.
4. Crucial benefit: The underlying vector embeddings used during retrieval hit the embed:v1:... cache continuously because their key generation logic remains untouched!

Frontend Integration: Next.js 16 and React 19 Server Actions

On the web tier using Next.js 16 and React 19, you can mirror this logic or wrap server responses using React's cache() function alongside Next.js tag-based revalidation. This gives you instant responses on client transitions while giving developers precise cache invalidation controls.

Here is a Next.js 16 route handler using explicit tag invalidations for completion caches without touching vector data:

import { NextResponse } from 'next/server'; import { revalidateTag } from 'next/cache';  export async function POST(request: Request) {   const body = await request.json();   const { promptId, userInput, systemPrompt } = body;    // Compute deterministic prompt fingerprint   const promptFingerprint = await crypto.subtle     .digest('SHA-256', new TextEncoder().encode(systemPrompt))     .then((buf) => Array.from(new Uint8Array(buf)).map((b) => b.toString(16).padStart(2, '0')).join('').slice(0, 12));    const payloadHash = await crypto.subtle     .digest('SHA-256', new TextEncoder().encode(userInput.trim().toLowerCase()))     .then((buf) => Array.from(new Uint8Array(buf)).map((b) => b.toString(16).padStart(2, '0')).join(''));    const cacheTag = `prompt-${promptId}-${promptFingerprint}`;    // Call backend Laravel 12 API with cache metadata   const response = await fetch('https://api.internal/v1/completions', {     method: 'POST',     headers: { 'Content-Type': 'application/json' },     body: JSON.stringify({ userInput, systemPrompt, payloadHash }),     next: {       tags: [cacheTag, 'llm-responses'],       revalidate: 86400, // 24 hours     },   });    const data = await response.json();   return NextResponse.json(data); }

If you update a prompt in your CMS or database without changing application code, you can trigger a cache revalidation specifically for that prompt ID using revalidateTag('prompt-support_bot-a1b2c3d4e5f6'). Next.js purges only the rendering layer while your Laravel backend maintains vector hit rates in Redis.

Edge Cases That Will Hurt You in Production

Here are three common issues developers face when implementing LLM caching:

1. Floating Point Variations Across PHP Environments

If you pack floats into binary strings using pack('f*', ...) on an x86 server and unpack them on an ARM64 server (like Apple Silicon during local dev), endianness matters. Use standard IEEE 754 single-precision floats (`f`) which follow machine byte order, or force little-endian format if you share raw Redis dumps across different CPU architectures.

2. High-Temperature Non-Determinism

Never cache responses when temperature is set high (e.g., temperature >= 0.7) unless your product spec explicitly demands exact repetition for identical inputs. For creative tools, caching ruins the user experience. Limit completion caching strictly to low-temperature operations (temperature <= 0.2) like classification, extraction, or rigid support bots.

3. Unbounded Redis Memory Growth

Don't save completions without an explicit TTL. Text responses are long, and storing millions of historical responses can exhaust your memory fast. Use Redis maxmemory-policy volatile-lru combined with strict 7-day or 30-day TTL settings on all embedding and completion keys.