Last reviewed: 2026-08-31

Direct answer

Adaptive concurrency limits for LLM APIs should sit immediately before an upstream route and control how many calls may be in flight at once. The controller admits work while measured latency remains near a low-load baseline, reduces the limit when latency indicates that work is queueing, and rejects or deliberately degrades excess demand before an unbounded queue forms.

This is different from a request-per-second limit. A rate limit counts arrivals during an interval; a concurrency limit measures how much work is occupying capacity now. The Netflix concurrency-limits project explains why a static request-rate threshold can become stale as systems scale and service time changes. It uses Little’s Law to connect concurrency, throughput, and latency, then treats latency growth as a congestion signal. That distinction is especially useful for an LLM gateway whose routes may have materially different request lifetimes.

The Envoy adaptive concurrency documentation describes a concrete gradient controller. It periodically measures an ideal round-trip time, summarizes completed-request latency over sampling windows, and compares the two. Its published relationship is gradient = (minRTT + buffer) / sampleRTT, followed by an updated limit based on the gradient, the previous limit, and headroom. You do not have to copy that controller exactly, but the production contract needs the same essentials: a baseline, a current sample, bounded growth, fast contraction, explicit floors and ceilings, and complete control over admissions in its scope.

Do not begin with one global limit for every LLM call. Define a controller scope narrow enough that its latency samples describe comparable work. A practical starting key is provider route, region, model class, response mode, and traffic class. Streaming and non-streaming calls may need separate scopes because their completion boundaries differ. Interactive and batch work may also need separate partitions so a batch surge cannot occupy every slot.

Operator workflow: happy path

  1. Define the route scope and the exact lifecycle being measured. For example, choose either time to first usable response or time to terminal completion; do not silently mix the two in one controller.
  2. Establish the low-load baseline under controlled concurrency. Set an initial minimum, maximum, and safe static rollback limit from staged tests rather than guessing from peak request rate alone.
  3. On arrival, check the request’s traffic class, remaining deadline, and current in-flight count. If the count is below the active limit, admit the request, increment the gauge, and start the selected latency timer.
  4. On every terminal path—success, provider rejection, timeout, or cancellation—decrement the gauge exactly once. Add a latency sample only when it matches the controller’s documented sampling contract.
  5. At the end of a sampling window, compare the representative sample with the baseline plus its variance buffer. If latency remains close to baseline and outcomes remain within policy, allow a small increase, never exceeding the ceiling.
  6. Record the admission decision and controller state without prompts, generated content, personal data, or request headers.

The healthy result is not a permanently rising limit. It is a limit that explores available capacity cautiously, holds when latency begins to bend, and leaves room for normal bursts without hiding a persistent queue.

Operator workflow: error path

  1. If in-flight work has reached the current limit, do not forward another request automatically. Admit it to a small bounded queue only when a slot exists and its remaining deadline exceeds the maximum queue wait; otherwise return the documented local-overload response or invoke an approved degraded mode.
  2. Mark the failure as a local admission decision so operators can distinguish it from a provider response. Include the active limit, in-flight count, queue state, route scope, and traffic class in the event.
  3. If sampled latency rises materially above baseline, contract the limit. Timeouts and explicit rejections can be additional backoff signals, as described by Netflix, but the controller must not count its own fast local rejections as successful upstream latency samples.
  4. Do not turn every rejection into an immediate retry or fallback. A retry consumes another admission slot and can multiply overload. Apply a per-action attempt budget and the controls in the retry-storm guardrails .
  5. If the controller oscillates, collapses to its floor, or produces implausible measurements, freeze automatic growth, apply the tested static limit, and enable the preselected degraded mode while the operator checks sampling and bypass traffic.
  6. Recover through multiple healthy windows and bounded increases. Do not restore the previous peak in one jump merely because a single window looks healthy.

This approach follows the central overload principle in Google SRE’s Handling Overload chapter : a backend should continue accepting work it can process and reject excess work gracefully. The goal is useful throughput with controlled latency, not acceptance of every arrival.

Who this is for

This guide is for platform engineers, SREs, and gateway owners responsible for chat, completion, embedding, multimodal, or streaming LLM routes. It assumes you can measure request lifecycles at the admission point, classify traffic, issue an explicit local-overload result, and change controller parameters independently of application releases.

It is most useful when fixed rate limits either waste capacity during fast periods or allow queues to grow during slow periods. It is not a substitute for provider quotas, timeouts, circuit breakers, tenant fairness, or capacity planning; it is the admission layer that keeps those mechanisms from being overwhelmed by too much simultaneous work.

Key takeaways

  • Limit concurrent occupancy, not only arrivals per second.
  • Scope each controller around workloads with comparable latency behavior.
  • Measure a low-load baseline and a representative current-latency sample.
  • Grow cautiously, contract promptly, and enforce tested floors and ceilings.
  • Prefer immediate shedding or a short, deadline-aware bounded queue over an unbounded backlog.
  • Partition capacity when interactive, batch, streaming, or other traffic classes need different protection.
  • Distinguish local admission rejection from upstream rejection in metrics and logs.
  • Keep retries and fallbacks behind their own admission checks and attempt budgets.
  • Preserve a runtime disable control and a tested static rollback limit.

Sources checked

  • Envoy: Adaptive Concurrency documents latency-window sampling, periodic minimum-round-trip-time measurement, the gradient calculation, minimum and maximum controls, recalculation jitter, runtime settings, limitations, and controller statistics.
  • Netflix: concurrency-limits explains why fixed request-rate limits can become stale, relates concurrency to rate and latency through Little’s Law, and describes delay- and loss-informed algorithms plus immediate and partitioned enforcement.
  • Google SRE: Handling Overload explains the limits of queries-per-second capacity models, graceful load shedding, degraded responses, client-side throttling, request criticality, and bounded retry behavior during overload.

These sources support the control-loop and overload principles. Route keys, response semantics, latency boundaries, partition sizes, and rollout thresholds still need validation against the gateway’s own traffic and service contract.

Contract details to verify

Controller scope and authority

Write down the exact routes governed by each controller and verify that no side channel bypasses it. Envoy explicitly notes that a latency-feedback filter must control concurrency for the whole cluster scope it is measuring. If another gateway, background worker, or direct client can send unmeasured traffic to the same capacity pool, the controller may interpret unexplained latency as a reason to contract without ever controlling the cause.

Assign one owner for route classification and parameter changes. Avoid creating so many tiny scopes that none receives enough completed requests to form useful samples, but do not combine workloads merely to increase sample volume. The scope should be the smallest operationally manageable pool whose requests have comparable occupancy behavior.

Measurement contract

Specify the timer start, timer stop, sampling percentile, sampling-window duration, baseline recalculation cadence, buffer, and excluded traffic. Envoy exposes a configurable sample percentile and warns that health-check latency can contaminate minimum-latency measurements. Apply the same discipline to probes, synthetic traffic, cache hits, locally rejected requests, and canceled calls.

For streaming routes, decide whether the constrained resource is occupied until the first response unit, the final response unit, or downstream delivery completion. If slow consumers can retain gateway resources, pair the concurrency controller with streaming backpressure controls . Whatever boundary you choose, name it in telemetry so a future implementation change cannot silently alter the signal.

Admission, queue, and response contract

Define the queue as a finite resource with both a slot limit and a maximum wait. A request whose remaining deadline is shorter than that wait should fail immediately. When a slot opens, recheck cancellation and deadline state before forwarding.

Choose a stable local-overload response and document whether callers may retry it. Netflix’s examples use immediate rejection when the concurrency gauge reaches its limit, including HTTP 429 for a servlet integration and an unavailable result for a gRPC integration. Envoy notes that its minimum-latency measurement period can increase 503 responses. Those are implementation examples, not a universal status-code rule. Your contract must let operators and clients distinguish local shedding, upstream overload, quota exhaustion, and ordinary provider failure.

Growth, contraction, and recovery

Define the minimum and maximum limit, increase rule, decrease rule, sample sufficiency threshold, and behavior when no valid sample is available. A minimum that is too high defeats protection; a maximum that is absent lets a healthy period expand into an unsafe range. If baseline measurement deliberately reduces concurrency, add jitter across instances so they do not all probe at once. Envoy recommends jitter for that reason and exposes separate controls for the measurement concurrency and the normal minimum limit.

Recovery should require stable evidence. Track the current limit next to sampled latency, baseline latency, in-flight work, queue wait, blocked decisions, timeouts, and upstream rejection classes. A rising limit without stable latency is not recovery; it is renewed pressure.

Sanitized observability

A useful admission event contains controller facts, not payloads. The following example uses coarse workload buckets and non-sensitive identifiers:

{
  "event": "llm_admission_decision",
  "request_id": "req-42",
  "route_id": "chat-standard",
  "provider_id": "provider-a",
  "region": "region-a",
  "model_class": "general-chat",
  "traffic_class": "interactive",
  "streaming": true,
  "decision": "rejected",
  "reason": "adaptive_concurrency_limit",
  "outcome": "local_overload",
  "in_flight": 48,
  "concurrency_limit": 48,
  "queue_depth": 0,
  "queue_wait_ms": 0,
  "sample_latency_ms": 920,
  "baseline_latency_ms": 310,
  "controller_window_ms": 1000,
  "attempt": 1,
  "retryable": false,
  "http_status": 429,
  "prompt_size_bucket": "small",
  "input_token_estimate_bucket": "medium",
  "controller_version": "v1",
  "timestamp": "2026-08-31T00:00:00Z"
}

Do not log prompt text, generated output, raw headers, user identifiers, or unrestricted model parameters merely to debug the controller. Keep high-cardinality correlation identifiers short-lived and access-controlled. Aggregate dashboards should emphasize distributions and route classes rather than payload-level inspection.

Failure modes

  • A static request-rate limit masks changing service time. The same arrival rate can create very different in-flight occupancy when latency changes. Netflix and Google both warn against treating request rate as a complete capacity model.
  • Some traffic bypasses the gate. The controller sees latency from capacity it does not control, contracts its own clients, and still cannot stop overload. Inventory every ingress path before enforcement.
  • The baseline is not a low-load baseline. Health checks, mixed response modes, cold starts, or already-congested traffic can produce a misleading reference. Exclude incompatible samples and make baseline recalculation visible.
  • Instances recalibrate together. If every controller lowers its limit for baseline measurement at the same moment, capacity drops sharply and rejection spikes. Apply randomized jitter and watch the measurement-active gauge.
  • The floor is unsafe. A generously chosen minimum can keep too much work in flight even while the controller is trying to protect the route. Test the floor during injected latency and reduced-capacity scenarios.
  • The queue has no hard boundary. Waiting work consumes memory, ages past caller deadlines, and causes a burst when capacity returns. Limit both queue slots and wait time, and revalidate a request before release.
  • Unlike workloads share one latency signal. Long streams can dominate occupancy while short calls make the sampled latency look healthy, or batch work can consume slots needed by interactive calls. Split scopes or reserve partitions deliberately.
  • Retries erase the shedding benefit. A locally rejected call immediately reappears as another attempt, possibly through a fallback route. Carry attempt state, enforce an action-level budget, and admit every attempt independently.
  • The loop oscillates. Large increases, aggressive decreases, tiny windows, or sparse samples can make the limit swing. Bound every update, require enough samples, and keep a static rollback mode.
  • Logs expose payloads while missing controller state. Prompt capture does not explain an admission decision if the active limit, in-flight count, route scope, and sampling window are absent. Log the control facts and sanitize the rest.

FAQ

Is adaptive concurrency the same as rate limiting?

No. Rate limiting governs how many requests may arrive over time. Adaptive concurrency governs how many may occupy the protected route simultaneously and changes that number from latency feedback. Use both when you need contractual quotas as well as real-time overload protection.

Which latency should an LLM controller measure?

There is no universal choice. Select the boundary that corresponds to the resource being protected, such as time to first usable response or time to terminal completion. Do not combine boundaries within one controller. Envoy supports a configurable percentile over a sampling window; choose the percentile and window from your route’s latency objective and validate them under representative traffic.

Should excess requests wait in a queue?

Only within a strict slot and time budget. A short queue can absorb a small scheduling mismatch, but a long queue hides overload and spends the caller’s deadline before upstream work begins. If the queue is full or the remaining deadline is insufficient, reject or degrade immediately.

Should a local concurrency rejection trigger fallback?

Not automatically. A fallback is additional work and may share the same failure domain. It should proceed only when the alternate route has its own admission capacity, the user action still has time, the request remains safe to replay, and the attempt budget permits it. Otherwise, fallback turns local protection into distributed overload.

Should the gateway return 429 or 503?

Use the status defined by the client contract and keep the reason machine-readable. The public implementations use different responses in different contexts. More important than choosing one universal code is distinguishing a local concurrency decision from provider rate limiting, provider unavailability, and application failure.

How quickly should the limit recover?

Require several valid, healthy sampling windows and increase in bounded steps. Preserve a maximum ceiling even when observed latency is excellent. If the route reaches the ceiling repeatedly without latency growth, treat that as evidence for a controlled retest—not permission to remove the ceiling in production.

Can one controller cover both streaming and non-streaming requests?

It can only if their measured lifecycle and resource occupancy are genuinely comparable. In most designs, separate scopes are easier to reason about because a stream may retain a slot far longer than a non-streaming response. If they share a capacity pool, coordinate their ceilings or partitions so their combined maximum remains safe.

Reader next step

Pick one production-shaped route and write a one-page admission contract before enabling enforcement. Record its controller scope, latency boundary, baseline method, minimum and maximum, update rules, queue limits, traffic partitions, local-overload response, retry policy, sanitized log fields, and static rollback setting.

Then exercise four cases in staging: normal traffic, an upstream latency increase, a sudden arrival burst, and a retry surge. Confirm that the happy path uses available capacity without sustained queue growth. Confirm that the error path sheds work before deadlines expire, contracts the limit, preserves interactive capacity, and recovers gradually.

Build the operator dashboard around the overload signal triage checklist , and keep the retry-storm and streaming-backpressure controls linked from the runbook. Enable the controller for a small traffic slice with an explicit rollback trigger, then expand only after both admitted-request latency and local-rejection behavior match the written contract.