Circuit-breaker state evidence for safer CometAPI fallback routing

Last reviewed: 2026-08-03

Direct answer

An LLM API circuit breaker fallback is safest when the breaker is a documented routing contract rather than a hidden boolean. Give each meaningful failure domain its own state, aggregate recent failures and slow calls, require a minimum sample before opening, and record every state transition with the evidence that caused it.

The normal state model is straightforward. Closed allows calls to the primary route while collecting outcomes. Open rejects primary calls immediately so they do not consume time and resources waiting for a dependency that is unlikely to respond. Half-Open admits only a limited probe set to determine whether recovery is real. A failed probe returns the breaker to Open; enough successful probes can justify a controlled return to Closed.

Opening the primary breaker is not, by itself, permission to send every request to CometAPI. Fallback admission needs a separate decision covering route compatibility, available capacity, safety policy, remaining latency, and the attempt budget for the user action. If those checks fail, the correct outcome is a controlled degradation, not an unbounded chain of alternate calls.

Keep the evidence reviewable. An operator should be able to answer which scope opened, which observations counted, which threshold version applied, whether the fallback was eligible, how many requests were short-circuited, what happened to Half-Open probes, and why the breaker eventually closed or stayed open.

Who this is for

This guide is for reliability engineers, platform teams, gateway owners, and on-call responders who route production LLM requests across a primary provider and a CometAPI fallback. It assumes the application already has request deadlines, normalized error classes, and a way to distinguish a new user action from a retry of an existing action.

It is especially useful when a gateway supports several model aliases, endpoints, streaming modes, or regions. Those dimensions may fail independently. A single global breaker can turn a narrow incident into a broad outage, while a breaker scoped too narrowly can produce fragmented state that never reaches a meaningful sample size. The contract must make that tradeoff explicit.

Key takeaways

• Scope breaker state to a real failure domain, such as provider, endpoint, model alias, operation mode, and region when it behaves independently.
• Classify outcomes before aggregation. Request-specific rejection, upstream timeout, connection failure, slow success, breaker rejection, and fallback failure should not be collapsed into one undifferentiated error count.
• Use both failure and slow-call evidence. Long waits can consume resources before hard failures dominate the metrics.
• Require a minimum number of observations so a tiny low-traffic sample does not open the route on noise alone.
• Emit a transition event for Closed to Open, Open to Half-Open, Half-Open to Open, and Half-Open to Closed.
• Treat Half-Open as a bounded experiment. It is not a signal to restore the full primary load.
• Keep retry limits, concurrency limits, and fallback capacity controls separate from the statistical breaker state.

Sources checked

The Microsoft Circuit Breaker pattern supports the Closed, Open, and Half-Open model, immediate rejection while Open, limited recovery probes, transition events, exception classification, manual overrides, and separate treatment for independent resources.

The Envoy circuit-breaking documentation documents network-level limits for connections, pending requests, active requests, retries, and connection pools. It also identifies overflow counters and explains that live breaker capacity is observable through statistics on a per-cluster and per-priority basis.

The Resilience4j CircuitBreaker guide describes count-based and time-based sliding windows, minimum call counts, failure-rate and slow-call-rate thresholds, configurable Open durations, and bounded Half-Open calls. It also clarifies that a sliding outcome window does not itself restrict concurrent calls.

The claimed AWS page on timeouts, retries, and backoff with jitter currently resolves to AWS Builder Center. Its accessible evidence here contains only the Builder Center label, so no threshold, retry, backoff, or jitter claim in this article depends on that page.

Contract details to verify

Write the breaker contract before choosing threshold values. The contract should define what is protected, what counts as evidence, and what every state permits.

The following is an illustrative contract, not a universal default. Derive production values from normal latency, traffic volume, recovery behavior, and tested fallback capacity.

breaker:
  scope: primary-a-chat-nonstreaming
  window_type: time
  window_seconds: 60
  minimum_calls: 40
  open_on_failure_rate: 0.50
  slow_call_milliseconds: 8000
  open_on_slow_call_rate: 0.60
  open_seconds: 30
  half_open_probe_calls: 3
  close_after_successful_probes: 3
fallback:
  route: cometapi-approved-chat
  max_attempts_per_action: 1

A statistical breaker and a gateway resource breaker answer related but different questions. The statistical breaker asks whether recent primary outcomes justify short-circuiting calls. Envoy-style limits ask whether connections, queued work, active requests, retries, or connection pools have reached configured bounds. If Envoy is in the request path, retain relevant deltas such as upstream_cx_overflow, upstream_rq_pending_overflow, upstream_rq_active_overflow, upstream_rq_retry_overflow, and upstream_cx_pool_overflow. Do not interpret one counter as a complete incident diagnosis.

Before enabling the route, compare the implementation against a fallback evidence checklist . At minimum, the routing layer should emit a sanitized decision record like this:

{
  "event": "circuit_breaker_transition",
  "observed_at": "2026-08-03T15:00:00Z",
  "breaker_scope": "primary-a-chat-nonstreaming",
  "model_alias": "approved-chat",
  "from_state": "closed",
  "to_state": "open",
  "reason_class": "upstream_timeout",
  "window_calls": 40,
  "failure_rate": 0.55,
  "slow_call_rate": 0.30,
  "threshold_version": "cb-v3",
  "fallback_route": "cometapi-approved-chat",
  "decision": "fallback_allowed",
  "attempt_ordinal": 1,
  "trace_ref": "tr-7f3a",
  "request_payload": "[REDACTED]"
}

Do not place raw prompts, response bodies, tool arguments, request headers, or user identity data in the breaker event. Keep a short trace reference for authorized investigation, and preserve aggregate evidence in the routing log.

Happy-path operator workflow

1. Normalize the incoming request into a breaker scope before contacting the primary route.
2. While Closed, record the primary outcome and duration in the selected window. Do not count a breaker rejection as a new upstream failure.
3. Once the minimum sample exists, compare the snapshot with the versioned thresholds. If an Open criterion is met, change state once and emit the transition event.
4. Reject subsequent calls to that primary scope without waiting for another upstream timeout. Evaluate CometAPI eligibility independently against compatibility, policy, capacity, deadline, and user-action state.
5. If fallback is eligible, make only the permitted attempt and record its result separately. Use the established guidance to cap fallback attempts per user action .
6. After the Open interval, admit only the configured primary probes. Keep ordinary primary traffic short-circuited while the probes run.
7. If every required probe succeeds within its deadline, move to Closed or a controlled restoration stage. Apply explicit return-to-primary checks instead of moving all traffic at once.

Error-path operator workflow

1. If a Half-Open probe fails or exceeds its deadline, return the breaker to Open, restart the recovery interval, and emit a new transition event.
2. If CometAPI fails an admission check, do not cascade into another unbounded fallback. Record the reason and select the documented degraded response.
3. If a fallback call has an ambiguous outcome, preserve that classification. Do not treat it as proof that the primary recovered, and consult the partial-success retry guidance before another action could repeat side effects.
4. Give the user a stable outcome using prepared degradation messages rather than exposing raw infrastructure errors.
5. Review the breaker scope, recent state history, primary evidence, fallback evidence, and overflow counters. Escalate when the failure domain or recovery mechanism remains unknown.
6. Use a manual override only through a recorded change with an owner and expiry. A forced reset must not erase the evidence that justified Open state.

Failure modes

One breaker for unrelated routes. A failure on one model or endpoint opens the circuit for healthy traffic elsewhere. Scope independent resources separately, while avoiding dimensions so granular that each breaker sees too few calls.

Request-specific failures poison provider health. A malformed or incompatible request may say nothing about the upstream service’s ability to serve valid traffic. Define which outcome classes count, which are ignored, and which require a separate client-facing response.

A tiny sample creates false confidence. One success can close a lightly used route, or two failures can open it, when no minimum sample exists. Preserve the sample count beside every rate so operators can judge its meaning.

Latency is ignored until hard failure. Slow calls can hold threads, connections, memory, and request deadlines even when they eventually succeed. Track slow-call rate as a distinct signal and keep its threshold tied to the protected operation’s normal behavior.

Half-Open becomes a traffic flood. Letting ordinary traffic through as soon as a timer expires can overload a recovering service. Admit a fixed probe cohort and reject or redirect other traffic until the result is known.

Retry and breaker policies disagree. Retry code that keeps running after the breaker rejects a call creates useless work. Conversely, retry overflow may be the first clear sign that a gateway is amplifying a sporadic failure. The breaker exception and retry budget must produce one deterministic decision.

Primary failure becomes a fallback avalanche. Opening a busy primary can transfer a step change in traffic to CometAPI. Check fallback capacity headroom before enabling automatic admission, and retain a degraded mode when capacity is unavailable.

Distributed counters are treated as perfectly synchronized. Envoy documents its breaker limits as distributed and eventually consistent, so races can briefly exceed configured values. Alerting and incident review should allow for that behavior rather than declaring every small overshoot a configuration failure.

Manual state hides the real incident. A forced Open or reset without an owner, reason, and expiry can persist after the incident or conceal recurring faults. Manual control needs the same transition evidence as automatic control.

FAQ

Does an Open primary breaker mean every request should go to CometAPI?

No. Open means the protected primary call should be rejected immediately. CometAPI admission is a separate policy decision. The request still needs an approved route, compatible behavior, adequate capacity, sufficient remaining deadline, and an unused fallback attempt.

Should a successful health check close the breaker?

A health check can justify moving from Open to a limited probe state, but it should not automatically restore full traffic. The Microsoft pattern allows either timed or explicit availability testing, while its Half-Open model still depends on limited operation attempts. Test the behavior that matters to the route, then restore traffic gradually.

Which failures should count toward opening?

Count failures that are evidence about the protected dependency and operation. Keep request-specific rejection, local cancellation, breaker rejection, and fallback failure in separate classes unless the written contract deliberately includes them. Resilience4j’s configurable recorded and ignored exception categories illustrate why that classification must be explicit.

How many Half-Open probes are enough?

There is no universal number in the checked evidence. Choose a small bounded cohort that can detect continued failure without flooding recovery, then validate it against traffic volume and historical incidents. Record both the configured count and the completed count.

Is a circuit breaker also a concurrency limiter?

Not necessarily. Resilience4j states that its sliding window aggregates outcomes but does not limit concurrent function calls. Envoy uses the circuit-breaking name for concrete resource limits as well. Document which mechanism is changing state so operators do not confuse outcome evidence with admission capacity.

When should the breaker close?

Close only after the documented probe criteria succeed and the primary can accept a controlled restoration. A single successful ping is weak evidence for an operation that previously failed under real load. Keep fallback and primary metrics separate during the return so improvement on one route does not hide deterioration on the other.

Reader next step

Choose one production route and write its breaker contract before changing gateway behavior. Define the scope key, outcome taxonomy, observation window, minimum sample, failure and slow-call criteria, Open behavior, fallback admission checks, probe cohort, close criteria, manual override process, and sanitized transition record.

Then replay recent failure traces or run a controlled fault test. Confirm that the primary is called while Closed, skipped while Open, probed only in bounded Half-Open traffic, and reopened after a failed probe. Verify that CometAPI receives only eligible actions, that attempt limits survive route changes, and that the user gets a controlled response when neither route is suitable.

Once the evidence and capacity checks are in place, Start with CometAPI and validate the fallback path with the same state-transition logging used for the primary.