Classify provider errors into retry, fallback, and fail-fast actions

Last reviewed: 2026-09-03

An LLM gateway should never treat “not 2xx” as a complete fallback policy. A response can be a temporary capacity signal, a permanent configuration error, a billing stop, or a response that started successfully and then failed. Those cases have different customer and safety consequences. The reliable pattern is to normalize the provider’s status, typed error, error code, retry hints, and request identifier into one internal record, then make a bounded decision from that record.

Direct answer

Retry the same provider only when the error is plausibly transient, the request can be replayed, and the remaining deadline and retry budget justify another attempt. Move to a fallback provider when that bounded retry budget is spent or the primary provider reports a provider-scoped overload that an alternate route can serve. Fail fast for invalid input, unsupported parameters, authentication, permission, billing, spend-cap, and other errors that another provider cannot repair. Keep an “unknown or ambiguous” state for transport and mid-stream cases until you know whether the provider may have processed the request.

The OpenAI error-code guide illustrates why a single status-code rule is unsafe: its 429 responses include rate limiting, exhausted credits, and spend-limit conditions, while its 503 model-overload response has separate Retry-After guidance. The Anthropic error reference likewise distinguishes authentication, billing, permissions, conflicts, request size, timeouts, rate limits, internal errors, and overload. Build the policy around the most specific error type or code first, and use the HTTP status as a fallback signal rather than the whole decision.

The table is a policy starting point, not a promise that every provider uses the same labels. For example, Google’s Gemini troubleshooting guidance recommends exponential backoff for transient 408, 429, and 5xx responses, explicitly says not to retry 400 or 403 client errors, and calls for a maximum retry count. It also warns operators to use the API version and model that support the requested feature. Put those provider-specific facts in configuration, while keeping the gateway’s three outcomes—retry, fallback, fail-fast—stable.

A concrete operator workflow

1. Start one user action. Create an action identifier, an absolute deadline, a replay-safety flag, and a single retry/fallback budget. Every provider attempt belongs to this action; a new provider must not reset the budget.
2. Send the primary request. Capture only sanitized metadata before sending: route, provider, model alias, operation, deadline, body digest, and attempt number. Keep the request body in a controlled replay buffer rather than in logs.
3. Take the happy path. If the response is successful, validate the output contract (for example, required fields or tool-call shape), record the provider request ID, and commit exactly once. A 2xx status alone is not enough if the application contract is invalid.
4. Take the error path. Parse the typed error and code, status, Retry-After, and request ID. Classify by code/type before status. If the result is transient or throttled and replay is safe, calculate a jittered delay that fits the deadline and consume one retry token. If the result is permanent, return a stable error without retrying.
5. Fail over deliberately. When the retry budget is exhausted or a provider-scoped overload is unlikely to clear before the deadline, choose an alternate route that has capacity and a compatible request/response contract. Increment a hop counter, retain the same action ID, and record why the decision changed providers.
6. Close the loop. On fallback success, validate the same output contract and tell the client only what it needs to know. On fallback failure, return the best safe degradation and preserve the complete decision chain for review. Never let a second SDK’s automatic retries silently add attempts outside the gateway budget.

The AWS SDK retry-behavior reference is useful design evidence even when the target provider is not AWS. It separates transient, throttling, and non-retryable errors; matches an error code before falling back to HTTP status; uses exponential backoff with full jitter; and stops when a retry quota is depleted. Its adaptive mode can delay initial requests, so a multi-tenant gateway should not accidentally share one adaptive limiter across unrelated resources. Apply the principle—one visible budget and one owner of retries—to your LLM gateway.

A sanitized decision record can look like this:

{
  "event": "provider_decision",
  "action_id": "[REDACTED]",
  "provider": "primary",
  "model_alias": "production-chat",
  "operation": "generate",
  "http_status": 503,
  "error_type": "service_unavailable_error",
  "error_code": "server_is_overloaded",
  "retry_after_ms": 1000,
  "attempt": 1,
  "max_attempts": 3,
  "decision": "retry",
  "fallback_eligible": true,
  "body_digest": "[REDACTED]",
  "provider_request_id": "[REDACTED]"
}

Do not put prompts, completions, user identifiers, authorization material, or full request bodies in this event. A digest, bounded classification, and provider request ID are enough to correlate a review. Anthropic specifically documents request IDs in both headers and error bodies, which makes that identifier more useful than copying an entire response into an incident log.

Who this is for

This guide is for platform engineers, SREs, and application owners who operate a gateway in front of two or more LLM endpoints. It assumes you already have a timeout, a route selector, and a way to observe provider responses. It is especially useful when an SDK retries automatically, providers expose different error taxonomies, or a fallback can change model capabilities, latency, cost, or data handling.

It is not a substitute for provider-specific terms, quota configuration, authentication setup, or a product decision about whether a degraded answer is acceptable. Those constraints belong in the route contract and should be checked before production traffic is enabled.

Key takeaways

• Normalize status, error type, error code, Retry-After, request ID, and stream state into one record.
• Give permanent request, access, billing, and spend failures a fail-fast path; do not hide them behind a different model.
• Treat rate limits as two classes: recoverable throttling and a budget or spend cap that will keep failing until an operator changes it.
• Use one deadline, one retry budget, and one retry owner across the SDK, gateway, and fallback route.
• Apply exponential backoff with jitter, and let a depleted retry quota fail fast instead of creating a retry storm.
• Validate the same response contract after a primary success and after fallback success.
• Treat a stream error after headers or partial output as incomplete work, not as a clean success.

Sources checked

The policy above is grounded in four public, independently useful references that were refetched for this article:

• OpenAI API error codes — distinguishes authentication, credits, rate limits, spend limits, server errors, and model overload, including Retry-After guidance.
• Claude API errors — documents typed HTTP errors, conflicts, request-size limits, timeouts, overload, request IDs, SDK retries, and errors that can arrive during SSE streaming.
• Gemini API troubleshooting — specifies transient retry classes, exponential backoff with jitter, maximum retries, and the instruction not to retry 400 or 403 client errors.
• AWS SDK retry behavior — explains standard and adaptive modes, error-code-first classification, retry quotas, maximum attempts, and full-jitter backoff.

Provider behavior changes. Recheck these pages, your selected model’s contract, and your route configuration whenever you change an endpoint or SDK version.

Contract details to verify

Before enabling a new fallback edge, write down the assumptions that make the decision safe:

1. Classification precedence: Which typed codes override a generic status? The AWS reference shows why a 5xx carrying a throttling code should remain a throttling decision, not be treated as a generic server error.
2. Retry hints: Does the provider send Retry-After, and is it absent for a spend cap? A missing hint is not permission to retry forever; use a bounded local delay.
3. Replay safety: Can the request body be reconstructed byte-for-byte, and can any tool or business side effect run twice? If not, mark the attempt ambiguous and require an idempotent application boundary before replay.
4. Deadline and budgets: What is the absolute user deadline, maximum attempts, token or spend allowance, and per-tenant fallback share? The maximum must include the initial request, not just retries.
5. Capability contract: Does the alternate model support the API version, input modalities, tools, structured output, and context size? Google’s troubleshooting page calls out API-version and model compatibility as a source of failures.
6. Response contract: Which fields, finish states, safety outcomes, and tool-call arguments must be present before the gateway commits a result?
7. Client behavior: What stable error or degradation message does the caller receive when all routes fail? Keep provider internals out of that message while retaining a correlation ID for support.

Run a controlled test for each row in the policy matrix. Assert the selected action, delay, attempt count, log redaction, and final client response. Then test two simultaneous tenants so a capacity event in one route cannot consume another tenant’s fallback budget.

Failure modes

Retrying every 429. A rate limit may clear, but an exhausted credit balance, spend cap, or acceleration limit may not. OpenAI and Anthropic expose multiple meanings for 429; classify the provider code and billing context first. If the condition is a cap, fail fast and alert the owner.

Using status alone. A generic 500 can be transient, while a 5xx with a throttling code deserves the slower throttling backoff. Keep the typed code, type, and status in the normalized record and define an explicit unknown branch.

Stacking retry loops. An SDK, gateway, queue, and client can each retry the same call. The resulting multiplication can exhaust capacity precisely during an incident. Set SDK attempts deliberately, expose the remaining budget to the gateway, and make the gateway the final authority. A token-bucket retry quota is a useful fail-fast guardrail.

Failing over caller mistakes. Sending malformed JSON, an unsupported parameter, or an unauthorized request to another provider adds latency and obscures the real fix. Return a stable 4xx-style application error and attach a remediation hint to the operator record.

Ignoring the deadline. A mathematically valid backoff can still exceed the user’s remaining time. Compute delay from the absolute deadline, provider hint, and local cap; if no useful attempt fits, move to the next safe action or degrade.

Committing partial streams. Anthropic notes that an SSE error can arrive after a 200 response, where ordinary HTTP error handling no longer applies. Mark the stream incomplete, retain only bounded partial state, and do not present it as a complete answer. A replay requires the same duplicate-work and side-effect checks as any other retry.

Logging secrets or content. Full prompts and headers make incident review risky. Log classifications, digests, bounded timings, and provider request IDs; redact content and authentication material before the event leaves the gateway.

Routing to an incompatible alternate. A fallback that cannot honor the input or output contract is an outage with extra steps. Check model, API version, context, tool, safety, and data-handling assumptions before putting the route in the candidate set.

FAQ

Should every 429 trigger fallback?

No. Retry a short-lived throttle with jitter and a provider hint when the deadline allows it. Treat credit exhaustion, spend caps, and other persistent budget signals as fail-fast conditions unless an operator has explicitly supplied a different funded route.

Is a 500 always safe to retry?

No. A bounded retry can be reasonable for an internal server error when the request is replay-safe, but the gateway must still enforce its deadline and attempt budget. If the response identifies throttling or a permanent application condition, use that more specific classification.

How many retries should the gateway allow?

There is no universal number. Choose a maximum from the user deadline, provider latency, cost, and duplicate-work risk. Count the initial request, reserve capacity for the alternate route, and prevent SDK retries from operating outside the same budget.

What if the primary returned some output before failing?

Treat it as incomplete or ambiguous. Do not silently concatenate a second answer to the first. Decide whether the product can discard the partial result, resume safely, or return a clear degradation message; record the stream state for review.

Why retain a provider request ID?

It lets an operator correlate a sanitized gateway decision with the provider’s support or incident records without storing sensitive payloads. Anthropic documents the ID in both a response header and an error body; other providers may expose equivalent identifiers.

Where does CometAPI fit in this matrix?

CometAPI can be an alternate route only after its endpoint, model capability, response shape, safety checks, and capacity budget satisfy the same contract as the primary. The matrix chooses when to use a route; it does not waive those compatibility checks.

Reader next step

Turn the table into a versioned configuration file and add one test per row: a happy 2xx response, a transient timeout, a throttled response with Retry-After, a persistent spend or credit cap, a permanent 400/403, an overload response, and a mid-stream failure. Verify that every test consumes the expected budget, emits only sanitized fields, and returns the same client contract after a successful fallback.

For the surrounding gateway controls, read Normalize provider rate-limit signals before fallback routing and Retry storm guardrails for gateway calls . Then run the fault-injection tests in a staging route before allowing a new provider to receive user traffic.