Last reviewed: 2026-08-08

Direct answer

Keep separate circuit-breaker state for every LLM route that can fail independently. In most gateways, the practical breaker key starts with provider, region, and model. Add an operation mode, such as streaming, only when it reaches a distinct backend or has meaningfully different failure behavior. Share policy templates across routes, but do not share failure counters, cooldown timers, or half-open probe results unless the routes truly share a failure domain.

A global breaker turns a partial provider problem into a gateway-wide outage. If one model in one region starts returning gateway failures, that evidence should not automatically block another model or region that remains healthy. Conversely, splitting state too finely produces tiny samples and unstable decisions. The right boundary is the smallest route for which recent failures are genuinely predictive of the next call.

The breaker also needs four explicit contracts: which outcomes count as failures, how much evidence is required to open, how recovery probes are admitted, and how much healthy capacity must remain. It should stop retries into an open route, allow only bounded traffic while half-open, and close only after the configured recovery evidence is satisfied. Circuit breaking is therefore a routing control, not a substitute for retry limits, concurrency isolation, or endpoint health checks.

Who this is for

This guide is for platform engineers, SREs, and on-call owners who operate an LLM gateway across multiple providers, regions, or models. It is especially relevant when fallback routing is automated and one route can degrade while adjacent routes remain usable.

You should already know the routes your gateway can select and the response contract each route must satisfy. The goal here is to decide where breaker state lives and how operators can explain every transition without storing prompts, generated content, or sensitive request data.

Key takeaways

  • Key breaker state by an independently failing route, normally provider plus region plus model.
  • Reuse configuration templates, but isolate counters, timers, and probe outcomes.
  • Count transport failures, selected upstream failures, and policy-defined slow calls; ignore request-specific errors that do not demonstrate route health.
  • Require a minimum sample before rate-based thresholds can open a breaker.
  • Limit half-open probes so a recovering route is not flooded immediately.
  • Preserve a healthy-capacity floor or ejection ceiling so the breaker system cannot remove every route at once.
  • Keep retries sensitive to breaker rejection. An open breaker should end attempts against that route, not create another retry signal.
  • Use a bulkhead when the requirement is to limit concurrent work. A breaker observes outcomes; it does not itself cap concurrent calls.

Sources checked

  • Microsoft’s Circuit Breaker pattern separates retry behavior from circuit breaking, describes closed, open, and half-open states, recommends observable transitions, and warns against one breaker covering independent providers or resources.
  • The Resilience4j CircuitBreaker guide documents count-based and time-based sliding windows, minimum-call requirements, failure and slow-call thresholds, configurable half-open calls, and explicit exception classification.
  • Envoy’s outlier-detection documentation distinguishes locally originated failures from upstream transaction failures and explains host ejection, maximum ejection limits, recovery timing, and the interaction between passive and active health checks.
  • The Istio Destination Rule reference shows how traffic policy and outlier detection can be applied at destination, subset, workload, and port boundaries, with controls for ejection time and retained healthy capacity.

Together, these sources support the state machine, evidence windows, error classification, ejection controls, and route-specific policy boundaries used below. The provider-region-model key is an LLM-gateway design recommendation derived from the shared principle that independent resources should not contaminate one another’s health state.

Contract details to verify

1. Define the breaker identity.

Start with a stable route key such as provider-a:region-1:model-family:stream. The key should describe infrastructure and capability, not a user, tenant, or individual request. For each component, ask whether a failure in that component predicts failure for the other values:

  • Provider separates independently operated upstreams.
  • Region separates deployments when health or connectivity can differ by location.
  • Model separates serving pools or availability contracts when one model can fail while another remains healthy.
  • Operation mode belongs in the key only when it maps to a distinct failure domain. Do not add dimensions merely because telemetry exposes them.

A provider-wide parent breaker can still be useful when evidence shows a shared dependency is failing, but its signals and recovery policy must be separate from route-level state. Document which layer has routing authority when parent and child breakers disagree.

2. Define the signal contract.

Classify each outcome before updating a breaker:

  • Locally originated failures include connection failures, timeouts, and resets observed before a usable upstream response. Count them against the affected route, and keep them distinguishable from upstream responses.
  • Upstream failures include the selected server and gateway error classes your policy treats as evidence of route unavailability.
  • Slow calls can contribute to a separate slow-call rate when latency already violates the route’s service objective. A successful but slow response is not automatically equivalent to an error.
  • Request-specific validation or contract failures should normally be ignored by the health breaker. They may make a route unsuitable for that request, but repeated malformed requests do not prove that the route is unhealthy.
  • A breaker rejection is a routing decision, not a new upstream failure. Counting it again creates a feedback loop that keeps the breaker open without fresh evidence.

Store the classification rule version with every transition. Operators should be able to tell whether an apparent change in health came from upstream behavior or a policy deployment.

3. Choose the evidence window.

A count-based window gives every route a comparable number of observations, while a time-based window emphasizes recent behavior. Either can work, but rate thresholds need a minimum number of calls. Without that floor, a low-volume model can open after one or two failures and remain effectively untestable.

Set failure-rate and slow-call thresholds from historical route data and user-facing objectives rather than copying a library default. Record the window type, window size, minimum calls, open duration, half-open allowance, and required recovery result as one versioned policy. Re-evaluate them when routing volume or model placement changes.

4. Protect recovery and remaining capacity.

When the open duration expires, admit only a bounded number of representative half-open calls. A probe should exercise the same data path that failed without causing an external side effect. If a separate health endpoint does not test real inference traffic, treat it as supporting evidence rather than proof of recovery.

Set an ejection ceiling or minimum healthy-capacity rule. If opening another breaker would remove the last compatible route, the gateway needs an explicit degradation decision instead of silently ejecting everything. That decision might reject new work quickly, constrain features, or preserve a known-safe route under a stricter admission limit.

5. Run the happy path as an operator workflow.

  1. A request enters with a capability requirement, and the router resolves the closed route provider-a:region-1:model-family:stream.
  2. The call succeeds. The gateway records latency and a successful route outcome without storing prompt or response content.
  3. Later calls produce enough route-local gateway failures to satisfy both the minimum sample and configured threshold. Only that route moves from closed to open.
  4. Retry logic stops targeting the open route. Any fallback attempt still obeys the per-action limit described in cap fallback attempts per user action .
  5. The router selects a contract-compatible route whose breaker is closed and whose capacity policy allows the call.
  6. After the open interval, the affected route admits a small configured set of half-open probes. Other production traffic continues using healthy routes.
  7. The probes satisfy the recovery rule, the breaker closes, and traffic returns according to the gateway’s rollout policy. Every state change is visible to operators.

6. Run the error path explicitly.

  1. If a half-open probe fails, reopen that route immediately and restart its recovery interval. Do not broadcast the probe result to unrelated provider, region, or model keys.
  2. If the same request produces a validation failure on several routes, stop treating it as a provider-health event. Surface the request or compatibility problem instead of opening breakers and spraying fallback traffic.
  3. If the next ejection would breach the healthy-capacity floor, enter the documented degradation mode and alert the operator.
  4. If gateway replicas disagree about state, use the declared authority and policy version. Do not let each replica conduct an unbounded set of half-open probes.
  5. If an active health check passes while inference traffic still fails, keep passive traffic evidence separate and investigate whether the probe exercises the failed path.

7. Log transition evidence without content.

A sanitized transition record can look like this:

event: breaker_transition
request_id: req-42
route_key: 'provider-a:region-1:model-family:stream'
provider: provider-a
region: region-1
model: model-family
operation_mode: stream
state_before: closed
state_after: open
signal_class: gateway_failure
status_code: 503
latency_ms: 8120
window_calls: 40
window_failures: 22
window_slow_calls: 25
policy_version: cb-v3
fallback_route: 'provider-b:region-2:model-family:stream'
content_preview: '[REDACTED]'

Also record the transition reason, open-until time, half-open allowance, ejection ceiling, and trace correlation identifier when available. Do not log prompts, generated text, request bodies, or transport headers merely to explain a breaker transition. Aggregate counts and stable route identifiers are enough for most reliability reviews.

Failure modes

One global breaker poisons healthy routes. A regional failure opens a provider-wide or gateway-wide circuit, so working models and regions disappear from routing. Fix this by isolating state at the verified failure boundary and reserving broader breakers for genuinely shared dependencies.

The key is too granular. Adding tenant, feature, request shape, and deployment identifiers creates many low-volume breakers. Their windows never gather enough representative evidence, and operators cannot reason about routing state. Remove dimensions that do not identify an independent upstream failure domain.

Low-volume routes open on noise. A percentage threshold without a minimum call count can react to a tiny sample. Require the minimum evidence before calculating a route’s failure or slow-call rate.

Request errors are mistaken for outages. Validation failures enter the same bucket as timeouts and upstream failures. A malformed workload then ejects every provider in sequence. Define record, ignore, and reject classes before enabling automatic fallback.

A successful health probe closes the wrong failure. A shallow endpoint passes while inference traffic still times out. Envoy’s documentation specifically cautions that a successful active check can uneject a host prematurely when the check does not validate the failing data path. Keep active and passive evidence visible separately.

Half-open becomes full traffic. Every gateway replica admits its own probe set or normal traffic resumes as soon as the timer expires. Bound probe concurrency at the route level and define which component owns state transitions.

Every route is ejected. Independent breakers behave correctly in isolation but collectively remove all compatible capacity. Apply a maximum-ejection or minimum-health rule and pair it with a deliberate degradation path.

A shared proxy cluster recreates global coupling. Envoy notes that when a cluster is shared across filter chains, one chain’s ejection can affect the others. Verify that proxy cluster boundaries align with the gateway’s provider-region-model breaker keys.

The breaker is used as a concurrency limiter. Sliding-window size does not limit calls in flight. Use the separate resource isolation described in bulkhead pools for provider saturation .

FAQ

Why not use one breaker per provider?

A provider-level key is sufficient only when its regions and models share the same practical failure domain. If one model or region can remain healthy while another fails, provider-only state is too broad. You can maintain a separate provider-level parent breaker for evidence of a shared outage without merging its counters with route-level breakers.

Should every timeout and server error have equal weight?

Not automatically. Keep locally originated failures, upstream transaction failures, and slow calls distinguishable. Then define which classes count and whether their thresholds differ. The important requirement is that the classification be deterministic, versioned, and visible in transition logs.

Does a circuit breaker replace retries?

No. Retries address faults expected to clear within the request’s attempt budget. A breaker blocks operations currently judged likely to fail. Retry logic should respect an open breaker and stop targeting that route. This separation also prevents the breaker from becoming a retry amplifier.

Can a health endpoint close a breaker?

Only if the health check is representative enough for the failed path and the recovery contract explicitly accepts it. Otherwise, use it to permit a limited half-open inference probe rather than closing the breaker directly. Keep endpoint readiness checks distinct as described in separate endpoint health checks before fallback routing .

How should traffic return after successful probes?

Closing the breaker establishes that the route may receive traffic; it does not require an immediate return to full share. Use a separate recovery policy for promotion, observation, and rollback. The related guide on choosing when fallback traffic can return covers that decision boundary.

Reader next step

Create a route inventory with one row for every provider-region-model combination your gateway can select. For each row, write down the breaker key, failure classes, ignored outcomes, window type, minimum sample, opening rule, open duration, half-open allowance, recovery rule, and healthy-capacity floor.

Then run two controlled walkthroughs. In the first, fail one route and verify that adjacent regions and models stay eligible. In the second, fail a half-open probe and verify that only its route reopens, fallback attempts remain bounded, and the operator can reconstruct the decision from sanitized logs. Confirm that the proxy, gateway, and application all use compatible route boundaries.

Do not enable automatic ejection until those workflows are observable end to end. The immediate deliverable is a versioned breaker policy and a transition dashboard that answers three questions: which route changed state, which classified evidence caused it, and which healthy route or degradation policy handled the next request.