Relying on a single third-party API for text generation is an operational risk. Commercial LLM endpoints fail. They experience routing issues. They hit unexpected rate limits during peak hours. If your application sends requests directly to an endpoint without a backup plan, users see hanging requests or hard errors. You need a fallback strategy that catches transient exceptions, retries with exponential backoff, and switches to an alternative provider if the primary service stays down. This pattern keeps your application responsive when external infrastructure stumbles.
Building this properly takes more than a few try-except blocks wrapped around an HTTP client. It is easy to write code that works on your local machine, but production traffic exposes every edge case you didn't anticipate. Third-party APIs throw 429 status codes when you burn through your rate limits. They throw 503 errors when their internal worker pools get overwhelmed. They drop random connection resets when network paths fail. If you don't handle these specific failures correctly, your users experience broken UI states and lost data.
Designing a Resilient Retry Strategy
The tenacity library provides clean decorators to handle retries in Python. Instead of writing custom while loops and exception counters, you decorate your API calling function with rules that specify wait times, stop conditions, and exception filters. Most developers make the mistake of retrying every exception. You shouldn't retry errors that won't change on a subsequent attempt. Client-side validation errors and authentication failures will never succeed on a retry. Catching them and burning through your retry attempts wastes precious seconds and increases latency for the end user.
You should only retry errors that might succeed on a subsequent attempt, such as network timeouts, rate limit responses, and server errors. You also want to combine your retry logic with caching layers to minimize redundant network calls, especially when dealing with identical prompts or static reference data as outlined in caching embeddings and llm completions in production. If you cache successful completions, your fallback chain doesn't even need to execute for repeated queries. That saves both money and time.
Here is how you set up a basic tenacity retry block for a primary API call:
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type
import httpx
class ProviderAPIError(Exception):
pass
@retry(
retry=retry_if_exception_type((httpx.TimeoutException, httpx.NetworkError)),
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=2, max=10),
reraise=True
)
def call_primary_provider(prompt: str) -> str:
response = httpx.post(
"https://api.openai.com/v1/chat/completions",
json={"model": "gpt-4o", "messages": [{"role": "user", "content": prompt}]},
timeout=5.0
)
if response.status_code >= 500:
raise ProviderAPIError("Primary provider server error")
response.raise_for_status()
return response.json()["choices"][0]["message"]["content"]
That decorator handles the exponential backoff automatically. It waits two seconds after the first failure. It waits four seconds after the second. Then it stops entirely after three attempts. Don't set your max wait time too high. If your users are waiting thirty seconds for a response, they've already abandoned the page. They clicked refresh, which duplicates the broken request and compounds your server load.
Switching Providers When Failures Persist
Retrying a failing endpoint three times works for brief network hiccups. If the provider suffers a wider outage, those retries just add delay before the inevitable failure. Once your retry budget exhausts, your code needs to catch the exception and immediately route the exact same prompt to a secondary provider. You might fallback from an OpenAI model to an Anthropic model or a locally hosted alternative.
Combining provider switching with strict guardrails ensures that your fallback provider does not ingest malformed or dangerous inputs. You can inspect token counts and costs before executing the fallback call, keeping your operational expenses predictable. For more details on managing token limits and unexpected usage spikes, refer to llm token counting cost control in laravel and nextjs. Different providers have different tokenizers and rate limits, so your fallback function must account for those API payload discrepancies.
Here is a complete implementation that wraps both the retry logic and the secondary fallback provider:
from tenacity import RetryError
import httpx
def call_secondary_provider(prompt: str) -> str:
response = httpx.post(
"https://api.anthropic.com/v1/messages",
headers={
"x-api-key": "your-api-key",
"anthropic-version": "2023-06-01"
},
json={"model": "claude-3-5-sonnet", "max_tokens": 1024, "messages": [{"role": "user", "content": prompt}]},
timeout=10.0
)
response.raise_for_status()
return response.json()["content"][0]["text"]
def generate_completion(prompt: str) -> str:
try:
return call_primary_provider(prompt)
except (RetryError, ProviderAPIError):
return call_secondary_provider(prompt)
This structure isolates the failure domain. If the primary provider's servers go dark, the exception bubbles up past tenacity. It gets caught cleanly by the outer try-except block. Then it dispatches the payload to the secondary service. Your application logic doesn't crash. Your user gets their text response with only a minor delay.
Handling Edge Cases and Logging Failures
Silent failures make debugging production issues difficult. When your primary provider drops requests and your fallback kicks in, you need clear log entries so your monitoring tools can track availability. Configure your logging handlers to record which provider failed, the exact exception raised, and the duration of the fallback switch. If your primary provider fails for five minutes every morning, your logs will show the pattern immediately.
You should also implement circuit breakers if your fallback provider experiences strain. Sending high-volume traffic to a secondary service that is already degraded will take down your entire application flow. Keep your timeouts tight. Log every fallback event. Monitor your error tracking dashboards for anomalies in provider response codes. Don't assume your secondary provider has infinite capacity just because it acts as a backup.
Testing these fallback chains requires mocking network failures in your test suite. You don't want to burn real API credits during CI runs. You can use standard testing libraries to simulate httpx exceptions and verify that your fallback function invokes the secondary provider correctly when the primary endpoint returns server errors. If you don't test these failure paths explicitly, they will break the first time your provider goes down.
