Last reviewed: 2026-08-05
Direct answer
Put a fallback admission controller in front of the CometAPI route, and make it enforce two limits at the same time: a global ceiling for the shared route and a weighted allowance for each tenant. The practical form of CometAPI fallback per-tenant rate limits is not a single requests-per-minute number. It is a small contract that covers trusted tenant identity, estimated work, in-flight concurrency, queue time, and a distinct rejection reason.
The global ceiling must come from a load test or a measured operating envelope for the exact fallback configuration. Leave explicit headroom for control traffic and critical work. Inside that ceiling, give every tenant a floor or a documented weight, then allow only a bounded burst. A request is admitted only when the global budget, the tenant budget, and the request class all have room. If a request cannot start before its remaining deadline, reject it locally instead of parking it in an unbounded queue.
Use work units as well as arrival rate. The Google SRE chapter on Handling Overload warns that queries can have very different resource requirements and that queries per second can be a poor capacity metric. For an LLM gateway, a practical local proxy is an estimate based on input size, requested output ceiling, model class, and whether the request holds a streaming slot. Reconcile that estimate with observed usage after completion so the limiter gets better rather than silently drifting.
Fairness does not mean every tenant receives the same number. It means the policy is explicit, enforceable, and prevents one workload from consuming the entire emergency route. The Kubernetes API Priority and Fairness documentation is a useful design reference because it frames admission around priority, queues, and concurrency allocation. The Envoy local rate limit documentation shows a concrete gateway mechanism: token buckets can be attached to routes and descriptors, and an exhausted bucket can produce a configurable 429 response.
Those sources describe general overload controls, not CometAPI account limits. Do not infer a provider quota, model entitlement, or response header from them. Keep the policy in your gateway and verify the current CometAPI contract separately before choosing production numbers.
A useful decision order is:
- Resolve a trusted tenant reference and workload class.
- Estimate the request cost and reserve the maximum needed work.
- Check the global fallback ceiling.
- Check the tenant allowance and class concurrency cap.
- Admit, queue briefly, or reject with a local reason code.
- Release the reservation on completion, cancellation, or timeout.
This order makes a local capacity decision terminal for that attempt. A tenant-limit rejection must not be mistaken for an upstream outage and must not trigger another automatic fallback attempt.
Who this is for
This article is for platform engineers, gateway owners, and SREs operating a multi-tenant application that can move traffic to a constrained CometAPI route when a primary provider degrades. It is especially relevant when interactive requests, background jobs, and long streams share one fallback pool.
It is not a claim that CometAPI supplies native tenant scheduling. If the provider offers account or project limits, treat those as an additional upstream boundary. The tenant policy described here protects your own shared capacity before a request leaves your system.
Key takeaways
- Derive tenant identity from trusted server-side context; never use an unverified client label as the quota key.
- Enforce a global fallback ceiling before any tenant can consume a burst allowance.
- Count estimated work and in-flight seats, not only requests per minute.
- Give workload classes explicit weights, floors, and queue deadlines.
- Return a local, reason-coded rejection when the gateway is full, and keep it separate from a CometAPI response.
- Start in observe-only mode, test one noisy tenant against a quiet tenant, then enforce gradually.
- Keep retry limits and fallback scope attached to the same user action; the fallback-attempt limit guide covers that adjacent control.
Sources checked
- Kubernetes API Priority and Fairness documents priority levels, queues, concurrency limits, and fairness controls that can inform a gateway admission design. It is a conceptual reference, not a CometAPI configuration guide.
- Google SRE Handling Overload explains per-customer quotas, resource-based capacity, quick rejection, and client-side throttling during overload. Its examples are general serving-system guidance.
- Envoy Local Rate Limit documents route and descriptor token buckets, local scope, configurable 429 responses, statistics, and an enabled-versus-enforced rollout mode.
Together, these sources support the fairness and admission patterns in this article. They do not establish current CometAPI pricing, quota, model, or error semantics, so those details belong in the verification checklist below.
Contract details to verify
Capacity envelope. Measure the fallback route with representative short prompts, long prompts, maximum output requests, and streaming calls. Record the highest sustainable in-flight work before latency or error objectives become unacceptable. Subtract a reserved slice for health checks, operator traffic, and the highest-priority class. The resulting ceiling is a local safety limit, not a statement about the provider’s advertised capacity. Revisit it when the model mix, timeout budget, or request distribution changes.
Cost unit. Choose one internal unit that correlates with resource use. It can combine estimated input units, reserved output units, and a multiplier for long-lived streams. Keep the definition stable enough to compare tenants, but record both the estimate and the final observed amount. If the estimate is unavailable, place the request in a deliberately small unknown-work pool rather than granting an unlimited default.
Tenant identity. Resolve the tenant after your trusted ingress has authenticated the calling application. Store an opaque reference for rate-limit state and logs. If a forwarded tenant header is allowed for internal hops, overwrite it at the trust boundary and reject requests whose mapping is missing or contradictory. Never key a production quota only by IP address, browser value, or an arbitrary prompt field.
Policy shape. A starting policy can look like this, with every number replaced by a value from your own test:
fallback_policy:
global_concurrency: 40
global_queue_depth: 20
reserved_capacity_percent: 10
tenant_classes:
standard:
concurrency: 4
work_units_per_minute: 20000
queue_depth: 2
priority:
concurrency: 12
work_units_per_minute: 60000
queue_depth: 4
unknown_tenant:
concurrency: 1
work_units_per_minute: 1000
queue_depth: 0
local_rejection_reason: fallback_tenant_limit
The global check must remain authoritative even when a tenant has unused allowance. If borrowing is permitted, make it conditional on spare global capacity and reclaimable when a protected class needs it. Do not let the sum of nominal tenant shares become an excuse to exceed the tested ceiling during a correlated outage.
Happy path workflow. First, the primary route produces a failure classification that is eligible for fallback under your existing user-action policy. The gateway resolves the tenant reference, assigns the request class, and estimates its work. It atomically checks the global ceiling, the tenant’s rolling allowance, and the class concurrency cap. If all three pass, it reserves the work, forwards the request to the configured CometAPI route, and starts the normal timeout. On a successful completion, cancellation, or timeout, it releases the in-flight reservation and records the observed cost. A monitor can compare admitted work, queue wait, and outcome by tenant without storing content.
Error path workflow. If the tenant allowance is exhausted while global capacity remains, reject at the gateway before an upstream call. Return the status and error body your client contract defines for local admission denial, with a stable reason such as fallback_tenant_limit; a 429 is a common choice for a local rate-limit decision and is supported by the Envoy pattern, but verify your own client behavior. If the global ceiling or queue is exhausted, use a separate reason such as fallback_global_limit and shed the least important class first. Do not translate either local decision into an upstream failure, and do not immediately retry it through another fallback route. If CometAPI itself returns a rate-limit or server error, record that as upstream_rate_limit or upstream_failure and keep it distinct from a local tenant denial.
Sanitized logging fields. Log the decision, not the prompt. A useful event contains an opaque tenant reference, request class, route class, estimate, limits, queue delay, decision reason, upstream status family, and latency. For example:
{
"event": "fallback_admission",
"request_ref": "req-042",
"tenant_ref": "tenant-7",
"route": "cometapi-fallback",
"workload_class": "interactive",
"estimated_work_units": 3,
"tenant_work_units_in_use": 8,
"tenant_work_units_limit": 12,
"global_concurrency_in_use": 29,
"global_concurrency_limit": 40,
"queue_wait_ms": 7,
"decision": "admit",
"reason": "within_limits",
"upstream_status": 200,
"latency_ms": 830
}
For a denial, change only the decision and reason fields and leave the upstream status empty because no upstream request was made. Exclude prompts, generated text, raw tenant names, cookies, forwarded identity claims, and credential material. If an investigation needs a sensitive value, use [REDACTED] rather than copying it into an event. Keep counters low-cardinality and put tenant detail in access-controlled logs or a separate aggregate.
Distributed scope. Verify whether the limiter is global, per process, per connection, or per route. Envoy’s local filter documentation states that its default local limit is applied per Envoy process, with an option for downstream-connection scope. That is useful for a local guard, but a fleet-wide tenant guarantee needs a coordinated store or a carefully calculated per-instance slice. Test scale-out and scale-in, not just a single proxy.
Client and provider contract. Before enforcement, document local status codes, machine-readable reason values, queue deadlines, cancellation behavior, and retry rules. Separately verify the current CometAPI endpoint, model access, account scope, request-size limits, stream behavior, usage fields, and upstream rate-limit response. The three sources checked here are deliberately silent on those product-specific values.
Rollout contract. Use an observe-only phase that emits would-admit and would-reject decisions without blocking. Envoy’s documented separation between filter-enabled and filter-enforced settings is one example of this rollout idea. Compare predicted denials with actual latency, queue wait, and tenant impact. Then enforce for one low-risk class, run a noisy-neighbor test, and expand only when protected tenants remain within their declared service objectives.
Failure modes
A user-controlled tenant key. A caller changes a header or parameter and receives another tenant’s allowance. Resolve identity at a trusted boundary and make the derived reference immutable for the request.
Requests-per-minute fairness. A short request and a long context request consume one count each even though their resource use differs. Google SRE’s overload guidance calls out this exact weakness in query-count metrics. Use measured work units and concurrency, then recalibrate the estimator from real outcomes.
Per-process limits mistaken for fleet limits. A six-instance gateway with a local bucket on every process can admit much more aggregate traffic than the policy suggests. Either coordinate the decision or divide the local allowance by the number of active instances and account for autoscaling lag.
An unbounded queue. Queue depth hides overload until the original deadline has expired. Bound both entries and estimated work, and reject when the remaining deadline cannot cover queue plus service time.
Borrowed capacity that never returns. Allowing a busy tenant to consume every idle share leaves no room when another tenant surges. Put a hard global ceiling and a reclaimable burst cap around borrowing.
Local and upstream 429s merged together. Operators then blame CometAPI for a gateway policy decision, or they retry a local denial until the gateway itself is overloaded. Use separate reason codes and dashboards for tenant, global, and upstream outcomes.
Retry amplification. A client retries a local denial, the fallback selector treats it as a new outage, and the same action consumes several reservations. Mark local admission failures terminal for that attempt and keep the fallback decision log guide aligned with the retry policy.
Leaked streaming reservations. A disconnected client leaves a seat occupied until a long upstream timeout. Release on disconnect and timeout, and test the release path under half-open streams.
Priority starvation. A permanent priority class can crowd out standard traffic, while an overly strict reserve strands capacity during normal periods. Define floors, a maximum borrow amount, and an operator-visible reason for every shed decision.
FAQ
Are equal limits fair? Not necessarily. Equal limits are a reasonable starting point only when tenants have similar entitlements and request costs. Otherwise, use documented weights, a minimum floor, and a global ceiling that applies to everyone.
Is a rolling rate bucket enough? No. A rate bucket controls arrivals over a window, but long-running requests consume concurrency while they are in flight. Pair a work budget with a hard concurrent-seat limit and a queue deadline.
Should a tenant-limit rejection trigger another fallback? Usually no. It is a local admission result, not evidence that a different provider will succeed. Keep it separate from upstream failures and bind retry attempts to the original user action using the existing attempt-limit contract.
Can an idle tenant’s share be borrowed? Yes, if borrowing is bounded, revocable, and subordinate to the global ceiling and protected classes. Treat borrowed capacity as a burst, not a new entitlement.
Should the gateway return 429 or 503? Pick a stable mapping and document it. A tenant-specific exhausted allowance can use a local rate-limit response; shared fallback exhaustion can use a separate overload response. Clients must be able to tell which condition occurred without inspecting free-form text.
How do I size the first limits? Start with a representative load test, define the sustainable global envelope, reserve headroom, and then divide the remainder by tenant class and measured work. Begin in shadow mode and change one variable at a time. There is no universal CometAPI number in the sources checked here.
Reader next step
Start with a small policy sheet: trusted tenant identity, three workload classes, one global fallback ceiling, a minimum floor per class, a bounded queue, and reason codes for local denial. Run four tests: normal mixed traffic, one noisy tenant, simultaneous tenant bursts, and a limiter-state failure. Verify that a quiet tenant still receives its floor, that no local denial reaches CometAPI, and that every reservation is released after completion or cancellation.
Before broadening failover, review reserve capacity before fallback and pair it with bounded fallback attempts . When the admission contract and noisy-neighbor test are passing, Start with CometAPI from the guarded route, then keep the tenant and global decisions visible in your on-call dashboard.