Normalize provider rate-limit signals before CometAPI fallback routing

Last reviewed: 2026-08-08

Direct answer

Normalize a provider throttle response into a small, provider-neutral decision record before your gateway considers CometAPI. The record should distinguish a rate limit from an outage, preserve the scope that was limited, and carry a bounded wait estimate. At minimum, record status_class, scope, limit_kind, retry_after_ms, reset_at, remaining, source_confidence, and observed_at. Then make fallback a policy decision: the request may move only when the primary response is retryable, the CometAPI route is known healthy, the request fits its token and spend budget, and the remaining user deadline can accommodate one controlled attempt.

This distinction matters because HTTP 429 is a capacity signal, not proof that an upstream is unavailable. RFC 6585 defines 429 as Too Many Requests and says a response may include Retry-After; it also says 429 responses must not be stored by a cache. A gateway that treats every 429 as a dead provider can send a synchronized burst to its fallback. A gateway that treats every 429 as immediately retryable can create a loop. Normalize first, choose once, and record why.

Who this is for

This workflow is for platform engineers who own an LLM gateway, SREs who make failover decisions during an incident, and SDK or middleware maintainers who translate several provider contracts into one internal interface. It is useful when a product has a primary model route and CometAPI as a bridge or fallback route, especially when requests consume different amounts of input and output tokens.

It is not a provider purchasing guide and it does not assume that CometAPI offers a particular limit, header, model, or region. Those details belong in your endpoint contract and should be verified before rollout. The goal is to make the routing decision explainable even when providers expose different windows and names.

Key takeaways

  • Parse HTTP status, response headers, and provider error fields into a canonical signal; retain the original provider code for diagnosis.
  • Keep scope explicit. A project-wide or organization-wide limit is different from a model, tenant, or single-request limit.
  • Treat Retry-After as a lower-bound hint. Apply a maximum delay, the request deadline, and a single fallback-attempt budget.
  • Track request and token dimensions separately. A route with request headroom can still be exhausted on input or output tokens.
  • Mark unknown or contradictory metadata as low confidence and choose the conservative policy rather than guessing.
  • Probe or otherwise verify CometAPI capacity before routing. A primary 429 followed by a fallback 429 is one incident, not two independent retries.

Sources checked

The standards baseline is RFC 6585: Additional HTTP Status Codes . It defines the 429 meaning, describes Retry-After as optional, and explains why rate-limited responses must not be cached.

OpenAI rate limits describes several independent dimensions, including requests per minute or day and tokens per minute or day, and provides provider guidance for reading limit information and retrying safely. Use it as an example of why one integer called remaining is not enough.

Anthropic rate limits documents limit, remaining, and reset metadata, 429 responses with a retry-after value, token-bucket replenishment, acceleration limits, and model-specific pools. Those details show why a reset timestamp and a window type belong in the normalized record.

Gemini API rate limits and quotas measures requests and tokens across dimensions, applies limits per project rather than per API key, and documents daily, rolling spend, and model-dependent behavior. It also notes that specified limits are not guaranteed and that a 429 can be returned as RESOURCE_EXHAUSTED.

These sources are independent public contracts. They support the normalization method; they do not establish any undocumented CometAPI quota. Verify CometAPI behavior in your own service agreement and endpoint tests.

Contract details to verify

Start with a canonical schema. status_class should have at least rate_limited, transient_overload, auth_or_policy, invalid_request, and unknown. Map a 429 and an explicit provider quota error to rate_limited only when the provider says the request exceeded capacity. Keep the raw status and a sanitized error category alongside the class. Never copy an entire error body into logs because it can contain prompt fragments or identifying data.

Represent scope as an enum such as project, organization, model, tenant, or request. If a provider does not state scope, use unknown and lower source_confidence. Gemini’s project scope and Anthropic’s model-specific pools are concrete reminders that an API key is not a universal scope. Store limit_kind separately: rpm, rpd, tpm, itpm, otpm, spend_window, or unknown. A request can be below RPM and above TPM at the same time.

Normalize time into UTC milliseconds. Parse Retry-After as either a delay or an HTTP date, reject negative values, and cap an unreasonable value at your policy maximum. A provider reset value may describe a fixed boundary, while Anthropic documents continuously replenished token-bucket capacity. Preserve window_type as fixed, rolling, token_bucket, or unknown so an operator does not mistake a refill estimate for a midnight reset. If Retry-After and reset_at disagree, use the later safe time and set confidence to low; keep both raw fields only in a protected diagnostic store.

Before fallback, run a deterministic gate:

  1. Check that the original request has not been canceled and still has a user-visible deadline.
  2. Confirm status_class=rate_limited and that the primary retry budget is not already spent.
  3. Estimate the request’s input and output token demand, plus any spend guard, against the CometAPI route contract.
  4. Check a recent CometAPI health or capacity signal for the exact model class and region you will use.
  5. Compute effective_wait = max(retry_after_ms, policy_floor) and compare it with the deadline. If the wait leaves no useful time, return the normal degradation response instead of failing over.
  6. Send one fallback attempt with a new provider attempt identifier but the same user action identifier. Do not recursively invoke another fallback from the fallback response.

The happy path is a primary 429 with a clear two-second delay, a known project scope, and a healthy CometAPI route with headroom. The gateway records the decision, waits only if the deadline permits, sends one request, validates the response contract, and returns the result. The error path is a primary 429 with no usable delay metadata or a destination that also returns 429. In that case, bound the wait with jitter, mark the destination unavailable for the current decision, and return a controlled error or brownout response. Do not turn missing metadata into permission for unlimited retries.

Use sanitized fields that let an on-call engineer reconstruct the choice without exposing credentials or content. A compact event can look like this:

event: fallback_decision
trace_id: trace_7f3c
user_action_id: action_91b2
primary_provider: primary
primary_status: 429
status_class: rate_limited
scope: project
limit_kind: tpm
retry_after_ms: 2000
reset_at: 2026-08-08T12:05:00Z
window_type: token_bucket
remaining: 0
source_confidence: high
destination: cometapi
destination_health: healthy
fallback_eligible: true
decision_reason: primary_rate_limited_destination_healthy
attempt_index: 1

Keep identifiers short and synthetic, and omit authorization headers, API keys, full prompts, generated text, and raw provider bodies. Correlate this event with the existing CometAPI retry log fields and use the retry budget evidence checklist to check that every attempt has an owner and a deadline.

Failure modes

Scope collapse. A gateway sees a 429 on one model and opens a breaker for every model and tenant. Preserve provider, project or organization, model class, region, and tenant scope. Broaden the block only when the provider contract says the limit is shared.

Window mismatch. RPM, TPM, daily quotas, rolling spend windows, and token buckets refill differently. Store window_type and limit_kind; never compute a reset from a fixed minute boundary unless the provider documents one.

Retry-after parsing errors. Treating an HTTP date as seconds can produce a negative or multi-day delay. Parse both legal forms, clamp to policy, and log the parsed value and parser result, not the full header set.

Fallback amplification. Many workers receive the same 429 and all switch at once. Add jitter, a shared destination admission check, and a per-action attempt cap. A destination 429 should close the current path, not start a third provider attempt.

False outage classification. A 429 caused by acceleration or spend policy is not equivalent to a network failure. Keep status_class and limit_kind visible so capacity policy can differ from transport retry policy.

Stale capacity evidence. A health result from several minutes ago can be wrong during a burst. Attach observed_at and a freshness bound to the CometAPI signal. If it is stale, treat eligibility as unknown and fail closed for automatic routing.

Cache contamination. RFC 6585 says 429 responses must not be stored by a cache. Set explicit cache controls in the gateway and make sure a fallback response cannot be served as if it belonged to the primary route.

FAQ

Should every 429 trigger CometAPI? No. First classify the scope and limit kind, check the remaining deadline, and confirm destination capacity. A project-wide exhaustion or a spend block may make an immediate fallback useless.

What if a provider sends no Retry-After? Use a bounded exponential backoff with jitter for the primary retry policy, but do not invent a precise reset. Set confidence to low and let the deadline and attempt budget decide whether a single CometAPI attempt is still worthwhile.

Why keep request and token limits separate? Providers can enforce both. OpenAI lists request and token dimensions, while Gemini and Anthropic describe model or project-specific quotas. A request-count check alone can route a prompt that the destination cannot admit.

Can I use the API key as the scope? Only if the provider explicitly defines limits that way. Gemini documents project-level application, so an API key should not be assumed to represent an isolated pool. Store the provider’s declared scope instead.

How do I handle a fallback 429? Record a second attempt event linked to the same user action, set fallback_eligible=false for that action, and return the product’s documented degradation response. Do not recursively route the same request.

Reader next step

Implement the canonical decision record behind a feature flag, replay sanitized 429 fixtures from each provider, and assert that scope, window type, delay, deadline, and attempt index produce the expected route. Then verify the exact CometAPI model, region, quota, and health signal your gateway will use. When the contract is documented and the tests pass, Start with CometAPI as the controlled destination for eligible fallback traffic.