Third-party language model providers go down. They return HTTP five hundred errors, trigger rate limits during high traffic surges, and occasionally hang indefinitely until your worker processes time out. If you call these services directly inside your request lifecycle without protection, an upstream outage takes down your entire web application. You'll watch your database connection pool exhaust itself as waiting requests pile up behind stalled HTTP calls. That's where the llm api circuit breaker pattern saves your infrastructure from cascading failures.
A circuit breaker tracks failures across rolling time windows. When failures exceed a defined threshold, the breaker trips open. Subsequent requests fail immediately without hitting the network, saving precious execution time and giving the remote provider space to recover. After a cooling period, the breaker enters a half-open state, letting a single test request through to check if service has returned to normal. If it succeeds, the breaker closes again. If it fails, the countdown starts over.
Building the Circuit Breaker State Machine
You need a persistent store to track state across multiple worker processes. Cache drivers work well for this because they support atomic operations and automatic expiration times. Don't build an in-memory solution inside a PHP application unless you enjoy debugging state that resets on every request lifecycle. Here's a concrete implementation using the Laravel cache store and HTTP client.
<?php
namespace App\Services;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\Http;
use Illuminate\Http\Client\Response;
use RuntimeException;
class LlmCircuitBreaker
{
protected string $serviceKey;
protected int $failureThreshold;
protected int $cooldownSeconds;
public function __construct(string $serviceKey = 'openai', int $failureThreshold = 5, int $cooldownSeconds = 60)
{
$this->serviceKey = 'circuit_breaker:' . $serviceKey;
$this->failureThreshold = $failureThreshold;
$this->cooldownSeconds = $cooldownSeconds;
}
public function execute(callable $callback): mixed
{
if ($this->isOpen()) {
throw new RuntimeException('Circuit breaker is open. Upstream service is temporarily unavailable.');
}
try {
$result = $callback();
$this->reset();
return $result;
} catch (\Throwable $e) {
$this->recordFailure();
throw $e;
}
}
public function isOpen(): bool
{
$state = Cache::get($this->serviceKey . ':state', 'closed');
if ($state === 'open') {
$expiresAt = Cache::get($this->serviceKey . ':expires_at', 0);
if (time() >= $expiresAt) {
Cache::put($this->serviceKey . ':state', 'half-open', $this->cooldownSeconds);
return false;
}
return true;
}
return false;
}
protected function recordFailure(): void
{
$failureKey = $this->serviceKey . ':failures';
$failures = Cache::increment($failureKey);
if ($failures === 1) {
Cache::put($failureKey, 1, $this->cooldownSeconds);
}
if ($failures >= $this->failureThreshold) {
Cache::put($this->serviceKey . ':state', 'open', $this->cooldownSeconds);
Cache::put($this->serviceKey . ':expires_at', time() + $this->cooldownSeconds, $this->cooldownSeconds);
}
}
protected function reset(): void
{
Cache::forget($this->serviceKey . ':state');
Cache::forget($this->serviceKey . ':failures');
Cache::forget($this->serviceKey . ':expires_at');
}
}
That cache logic handles basic state transitions without external dependencies. If your application handles high concurrency, you'll want to wrap the cache increment calls in an atomic lock to prevent race conditions during failure counting. It's also worth checking out LLM API Rate Limit Caching and Quota Guarding for additional ways to protect your keys from getting banned by overzealous traffic.
Many developers make the mistake of catching every single exception type as a provider failure. Database timeout exceptions or validation errors inside your own application will mistakenly increment the upstream LLM failure counter if you aren't careful. You should explicitly filter for network timeouts, DNS resolution failures, and specific five hundred range status codes thrown by the remote client before recording a failure against the model provider.
Integrating Fallbacks and Secondary Providers
Failing fast is only half the battle. Your users want an answer, or at least a graceful degradation of features rather than a blank error page. When the primary provider trips the circuit breaker, your code should route the prompt to a secondary cheaper model or a local fallback. You might pair this strategy with the caching techniques discussed in Caching Embeddings and LLM Completions in Production to serve stale cached responses when everything else fails.
Here's how you wire the circuit breaker into a service class that handles model calls with a fallback chain.
<?php
namespace App\Services;
use Illuminate\Support\Facades\Http;
use RuntimeException;
class ResilientAiService
{
protected LlmCircuitBreaker $breaker;
public function __construct(LlmCircuitBreaker $breaker)
{
$this->breaker = $breaker;
}
public function generateCompletion(string $prompt): string
{
try {
return $this->breaker->execute(function () use ($prompt) {
$response = Http::withToken(config('services.openai.key'))
->timeout(10)
->post('https://api.openai.com/v1/chat/completions', [
'model' => 'gpt-4o',
'messages' => [
['role' => 'user', 'content' => $prompt]
],
]);
if ($response->failed()) {
throw new RuntimeException('Primary LLM returned status ' . $response->status());
}
return $response->json('choices.0.message.content');
});
} catch (RuntimeException $e) {
return $this->fallbackCompletion($prompt);
}
}
protected function fallbackCompletion(string $prompt): string
{
$response = Http::withToken(config('services.anthropic.key'))
->timeout(10)
->post('https://api.anthropic.com/v1/messages', [
'model' => 'claude-3-5-sonnet',
'max_tokens' => 1024,
'messages' => [
['role' => 'user', 'content' => $prompt]
],
]);
if ($response->failed()) {
throw new RuntimeException('All LLM providers are currently unavailable.');
}
return $response->json('content.0.text');
}
}
That structure isolates your application business logic from network volatility. If OpenAI starts throwing gateway errors, the primary breaker trips, subsequent requests bypass the primary block entirely, and your fallback provider absorbs the traffic until the cooldown expires. You won't block web worker threads waiting on dead sockets, and your application stays responsive even during major public cloud outages.
Testing these fallback paths in a local development environment requires deliberate mock configurations. Don't wait for a real outage to find out your secondary provider key has expired or your response parsing logic expects a different JSON structure. Write integration tests that force the primary circuit breaker open by manually populating the cache store, then assert that your service transparently calls the secondary endpoint without throwing an unhandled exception to the user interface layer.
Handling Rate Limits Without Tripping Breakers
HTTP four hundred twenty-nine responses require careful handling. A rate limit doesn't necessarily mean the provider is broken; it means your application is sending too many tokens too fast. If you treat a rate limit error the same way you treat a server crash, you'll trip your circuit breakers prematurely during standard traffic spikes.
You should inspect response codes inside your execution callback and decide whether to increment the failure counter. If the status code is four hundred twenty-nine, catch the exception, parse the retry header, and push the job onto a background queue instead of failing the request immediately. This keeps your web workers free and respects provider backoff headers.
Monitoring these state transitions gives you early visibility into degradation before users start filing support tickets. Hook into your application logging framework inside the state change methods to record every time a circuit opens or closes. Pair those logs with metric counters so you can alert your engineering team when a core AI provider starts flapping between open and closed states under heavy load.
