Last reviewed: 2026-08-28

Direct answer

A fallback request should leave your gateway only after the target route can accommodate every capacity dimension the request may consume. Reserve one request unit, estimated input-token capacity, a bounded local output-token allowance, and any local spend or concurrency units that apply. Treat those values as a vector rather than collapsing them into a request count. If any required reservation fails, do not dispatch the fallback.

That rule follows from the shape of current provider contracts. The OpenAI rate-limit guide describes separate request and token metrics and notes that some model families share a limit. The Anthropic rate-limit reference separates requests per minute, input tokens per minute, and output tokens per minute. The Gemini API rate-limit guide evaluates requests per minute, input tokens per minute, and requests per day, with limits scoped per project rather than per API key.

These sources describe provider contracts, not a universal CometAPI quota model. CometAPI token-aware fallback admission is therefore a gateway policy: translate the contract for each candidate route into local admission dimensions, check the correct quota scope, and reserve capacity atomically before sending work. Never assume that a green request counter means token capacity is also available.

After completion, settle the reservation against observed usage and release unused local capacity. If the provider rejects an admitted request, update the relevant local bucket from the response evidence and run any alternative route through admission again. A local pass reduces preventable overload; it does not guarantee acceptance, because configured limits and available service capacity can change.

Who this is for

This pattern is for platform engineers, SREs, and application teams that operate multi-model gateways, especially where a primary-route incident can move a burst of long prompts onto a smaller fallback pool. It is also useful for teams whose requests vary widely in context length or output size.

It assumes that routing, retry policy, and basic usage telemetry already exist. It complements fair per-tenant limits and bounded fallback attempts ; it does not replace either control.

Key takeaways

  • Admit against all applicable dimensions. A request can fit the request bucket while exceeding input-token or output-token capacity.
  • Key the reservation to the actual limiter scope. Model, model family, organization, workspace, project, and shared-pool boundaries are not interchangeable.
  • Keep provider accounting separate. Do not combine input and output into one token number when the target contract enforces them independently.
  • Estimate before dispatch and settle after completion. Estimates protect the route; observed usage improves the next estimate.
  • Make the reservation atomic. Parallel gateway workers must not spend the same remaining capacity.
  • Treat an upstream 429 as classification evidence, not an automatic instruction to spray the request across more routes.
  • Keep admission inside the user action’s attempt budget so a denial cannot turn into an unbounded fallback loop.

Sources checked

  • OpenAI API rate limits documents multiple limiter metrics, separate request and token limits, and shared limits for some model families.
  • Claude Platform rate limits documents organization-level enforcement, RPM, ITPM, OTPM, token-bucket replenishment, short-interval enforcement, and response details for several limit conditions.
  • Gemini API rate limits documents RPM, input TPM, RPD, per-project scope, model-dependent limits, and spend-based rate-limit responses.

Together, these sources support a multidimensional, route-specific admission design. They do not support hardcoding one provider’s counters, cache treatment, or reset behavior for every route.

Contract details to verify

Before enforcing admission, create a versioned contract record for every eligible fallback route. Verify these fields against the route and account configuration:

  • Limiter scope: Record whether each bucket belongs to an organization, workspace, project, model, model class, or shared model family. Use a stable internal scope name without logging provider account identifiers.
  • Dimensions: Record which of RPM, input TPM, output TPM, daily requests, spend, or other route-specific dimensions apply. Mark an unsupported dimension as not applicable rather than silently setting it to zero.
  • Token accounting: Capture whether cached input, cache creation, or other token classes count. Anthropic’s documentation, for example, describes cache-aware ITPM rules and model-specific exceptions; those rules should not be copied onto unrelated routes.
  • Output accounting: Separate the provider’s quota rule from your local reservation rule. Anthropic states that OTPM is evaluated from actual generated output and that max_tokens does not determine OTPM usage. A gateway may still reserve a bounded output allowance locally to prevent too many generations from starting together.
  • Replenishment: Record whether capacity replenishes continuously, resets on a schedule, or is reported only through current account controls. Anthropic documents token-bucket replenishment and warns that a nominal per-minute limit can still be enforced over shorter intervals.
  • Error contract: Record the status, provider error class, retry guidance, and whether a spend condition can resemble a transient rate limit.
  • Local policy: Add the safety margin, tenant share, concurrency cap, action deadline, and maximum fallback attempts that your gateway enforces independently of the provider.

Do not hardcode example quota values from public documentation. Read current route configuration into a control-plane snapshot, version that snapshot, and make the version visible in each admission event.

Admission rule

For each candidate route, build a need vector containing one request unit, the target model’s estimated input units, a local output reservation, and any other required units. Admit only if every applicable bucket has enough unreserved capacity after the configured safety margin. If a contract separates input and output, preserve both dimensions; adding them together can hide which bucket is exhausted.

Use a single atomic operation to reserve all dimensions. A partial reservation is unsafe: reserving request capacity and then failing to reserve input tokens can strand capacity or tempt the caller to dispatch anyway. Attach the successful reservation to the user action and the specific fallback attempt.

Happy path workflow

  1. Assign the user action a stable internal identifier and read its remaining attempt and deadline budgets.
  2. Render the request exactly as the candidate route would receive it, including system instructions, retained conversation, and tool definitions. Estimate input usage after this transformation.
  3. Choose a local output reservation from a configured workload class and cap. Keep the estimate conservative until observed error is well understood.
  4. Resolve the candidate’s current contract version and quota scope. Read all applicable remaining buckets from one consistent local snapshot.
  5. Atomically reserve the request, input, output, spend, and concurrency units. Record the remaining capacity before and after the decision.
  6. Dispatch one fallback attempt. Do not let another worker reuse the same reservation.
  7. On a complete response, settle estimated input and reserved output against the usage evidence available to the gateway. Release unused local capacity and record estimation error.
  8. Feed aggregate, sanitized estimation error back into the workload-class configuration. Do not train the estimator from raw prompts stored in admission logs.

Error path workflow

  1. If a local bucket cannot fit the request, make no upstream call. Return the configured action: queue, shed, shorten only where product policy permits, select an independently scoped eligible route, or present a controlled degradation.
  2. If an upstream rate-limit response arrives despite admission, capture the status, provider error class, and retry guidance. Mark the affected local scope as constrained.
  3. Respect explicit retry timing when present. Do not assume every rate-looking error is short-lived. Anthropic’s documentation shows that an enforced monthly spend cap can use a rate-limit error class without a retry-after header, while Gemini documents spend-based 429 RESOURCE_EXHAUSTED responses.
  4. Do not rotate API keys as a capacity strategy within the same Gemini project; the documented limit scope is the project, not the individual key.
  5. Before using another route, rebuild the need vector and run admission against that route’s own scope. Confirm that the user action still has time and an unused attempt.
  6. If a stream ends without trustworthy final usage, retain a conservative reservation until a bounded reconciliation timeout. Release it through an explicit settlement event, not an untracked background guess.

Sanitized logging fields

Log one admission event and one settlement event per attempt. Keep prompts, model output, request headers, credentials, user identifiers, and provider account identifiers out of these records. A compact event can look like this:

event: fallback_admission
observed_at: 2026-08-28T14:20:00Z
action_id: act-42
attempt: 1
route_class: fallback-a
quota_scope: project-model
contract_version: v3
estimated_input_tokens: 1800
reserved_output_tokens: 600
request_units: 1
remaining_before:
  request_units: 18
  input_token_units: 24000
  output_token_units: 9000
decision: admit
reason: all_buckets_available
response_status: 200
actual_input_tokens: 1740
actual_output_tokens: 412
retry_after_ms: null
sensitive_payload: '[REDACTED]'

The useful fields are the action and attempt, sanitized route class, quota-scope class, contract version, estimated and actual units, remaining local capacity, decision reason, response class, retry timing, and settlement result. Use [REDACTED] wherever a retained schema requires a sensitive placeholder.

Failure modes

  • Request-only admission: A burst of large prompts passes because RPM is available, then input-token capacity rejects or stalls the fallback work. The remedy is an admission vector, not a lower request ceiling alone.
  • Wrong scope key: Workers allocate separate local buckets per API key even though the provider enforces a project or organization pool. The local dashboard looks healthy while all workers consume the same upstream capacity.
  • Shared-pool double spending: Two model routes appear independent but draw from one shared limit. OpenAI explicitly documents shared limits for some model families, so the route contract must be able to point multiple routes at one bucket.
  • Blind cache discount: The estimator subtracts cached input without checking the candidate route’s accounting rules. A cache hit that improves one route’s effective throughput may not translate to another route.
  • Output oversubscription: The gateway admits many generations using only input estimates. Output is uncertain, so local reservations need a bounded workload-specific allowance and settlement rather than no allowance at all.
  • Fixed-window assumptions: A scheduler waits for a minute boundary even though the limiter replenishes continuously, or it sends a burst at the boundary and hits shorter-interval enforcement. The configured replenishment model must match the route contract.
  • Retry fan-out: Every 429 creates parallel attempts. That consumes the action budget and can pressure routes that share the same limiting scope. One denied route should produce one classified routing decision.
  • Spend condition treated as transient: A caller retries a spend-based refusal as though a small delay will restore capacity. Preserve provider error details and distinguish spend, acceleration, and ordinary rate dimensions where the source contract permits.
  • Estimator drift hidden by averages: A global average underestimates document-heavy or tool-heavy traffic. Track error by sanitized workload class and route contract version, including high-side misses.
  • Sensitive admission logs: Teams copy prompts or response bodies into quota events to debug estimates. Capacity debugging normally needs counts and classifications, not reader content.

FAQ

Why is request count insufficient?

Requests can consume very different amounts of input and output capacity. Current provider documentation exposes multiple dimensions precisely because passing one limit does not imply that the others pass. Gemini’s documentation states that exceeding any evaluated dimension triggers a rate-limit error.

Can API-key rotation create more fallback headroom?

Not when keys share the same upstream limiter scope. Gemini explicitly applies its documented rate limits per project rather than per API key. Model other scopes explicitly too; never infer independence from different credentials or client instances.

Should the gateway reserve the configured maximum output?

Not automatically. For Anthropic’s documented OTPM behavior, actual generated output counts and max_tokens does not determine OTPM usage. Your local admission control can reserve a conservative workload allowance, enforce a product output cap, and settle the difference. The correct allowance is a gateway policy, not a claim about provider accounting.

Do cached tokens count against input capacity?

It depends on the target contract. Anthropic documents cache-aware ITPM treatment and exceptions. Store those rules in the route adapter, observe the relevant usage classes, and do not apply one provider’s cache discount to every fallback.

Should every 429 trigger another fallback?

No. First classify the exhausted scope and determine whether the alternative is independent, eligible, within deadline, and within the user action’s remaining attempt budget. If those checks fail, queue or degrade according to product policy instead of adding load.

What if accurate token counting is unavailable before dispatch?

Use a conservative route-specific estimator, add an explicit uncertainty margin, and limit the request classes admitted while calibrating. Record estimated and actual units when available. An unknown estimate should never be silently treated as zero.

Does token-aware admission replace capacity planning or retry budgets?

No. Capacity planning decides how much fallback headroom should exist; admission decides whether this request fits now. A retry budget decides how much extra work one action may create. Connect the controls using the same action and route-scope identifiers.

Reader next step

Choose one high-volume fallback route and write down its limiter scope, dimensions, accounting rules, replenishment behavior, and error classes. Add observe-only admission events, compare estimates with settled usage by sanitized workload class, and confirm that parallel workers share the right local buckets. Then enable atomic rejection with an explicit queue or degradation path.

Review the design alongside the site’s retry-budget guidance , and keep every alternate attempt inside the original user action’s deadline and attempt cap.

If you are evaluating a gateway path for this control, Start with CometAPI , then validate the live route contracts before enabling enforcement.