Use Bulkhead Pools to Contain LLM Provider Saturation

Last reviewed: 2026-08-01

Direct answer

A bulkhead is a deliberate resource boundary. Instead of letting every LLM request draw from one global connection pool, semaphore, worker queue, and retry budget, give each important dependency or traffic class its own bounded pool. A slow provider can then fill its own waiting room without consuming the permits reserved for a healthy provider or for a high-priority customer.

The Microsoft Bulkhead Pattern guidance describes this as isolating elements into pools so a fault does not cascade. It also calls out AI and inference workloads, where deployment quotas and concurrency limits make strict isolation useful. For an LLM gateway, a practical boundary is usually provider plus traffic class: for example, ‘primary-interactive’, ‘fallback-interactive’, ‘primary-batch’, and ‘fallback-batch’. Split further by model or tenant when their limits, latency objectives, or business priority differ materially.

A pool needs more than a maximum number of requests. Set an active-request limit, a pending-queue limit, a connection limit, and a retry limit. Envoy’s circuit-breaking documentation exposes these same kinds of per-upstream and per-priority controls, including counters for overflow. When a limit is reached, make the gateway fail fast or choose an explicitly permitted fallback; never let an unbounded queue silently become the fallback policy.

A concrete operator workflow looks like this:

  1. Classify the request before it touches a provider: traffic class, tenant tier, provider, model family, and whether fallback is allowed.
  2. Check the selected pool’s active, pending, connection, and retry budgets atomically.
  3. On the happy path, admit the request, record the pool decision, dispatch once, and release the permit on completion or cancellation.
  4. On a full-pool error, return a normalized overload decision. Route to a separately budgeted fallback pool only if policy allows it and that pool has capacity.
  5. If every eligible pool is full, serve a documented degraded result or a clear error. Do not add another hidden retry loop.
  6. Compare pool-level saturation and user-visible latency after the event, then adjust one boundary at a time.

A bulkhead is not the same as a circuit breaker. The bulkhead decides how much shared resource a class may consume. A circuit breaker decides whether calls should be attempted after observed failures. A retry budget decides how many additional attempts are allowed. Combining them is safer than asking any one mechanism to solve all three problems.

The following values are sanitized examples, not provider limits or production defaults.

event: bulkhead_admission
request_id: req-42
provider: primary
traffic_class: interactive
partition: primary-interactive
priority: high
active: 23
active_limit: 24
pending: 1
pending_limit: 4
retry_slots: 0
retry_limit: 2
decision: dispatch
reason: capacity_available

The important invariant is simple: a request rejected by one partition must not consume the capacity of another partition merely because the first provider is slow.

Who this is for

This article is for platform engineers, SREs, gateway owners, and on-call teams operating multi-provider LLM routing. It is especially useful when a single service handles interactive requests, batch work, tool-call workflows, and failover traffic at the same time.

It is not a universal sizing table. The right limits depend on measured provider behavior, request cost, connection reuse, and your user-facing latency contract. Start with explicit boundaries and observable decisions, then tune them with controlled load tests.

Key takeaways

  • Isolate the resource, not just the URL. Separate connection pools, concurrency permits, pending queues, and retry slots for each provider or dependency that can fail independently.
  • Keep primary and fallback budgets separate. A fallback that shares the primary’s exhausted queue is not a real fallback.
  • Bound waiting work. A small, visible queue is easier to reason about than requests waiting until their deadlines expire.
  • Make priority a first-class dimension. A high-priority interactive pool should not be drained by batch traffic or a noisy tenant.
  • Count the resource that actually saturates. Google’s SRE overload guidance warns that simple queries-per-second assumptions can misrepresent request cost and recommends graceful rejection, throttling, or degradation when capacity is exhausted.
  • Give overflow a contract. Operators should know whether a full pool causes a fast error, a controlled fallback, a degraded response, or a local admission rejection.
  • Instrument every partition. Aggregate provider latency can look healthy while one tenant, model, or priority lane is repeatedly denied.
  • Test the error path deliberately. A successful normal-load test does not show whether a slow provider can starve its neighbors.

Sources checked

The design above is grounded in five public sources that were refetched for this article:

  • Bulkhead Pattern - Azure Architecture Center defines isolated pools, separate connection pools, priority partitions, tenant granularity, and strict bulkheads for AI and inference workloads.
  • Circuit breaking - Envoy documentation documents per-cluster and per-priority limits for connections, pending requests, active requests, retries, and connection pools, along with overflow statistics.
  • Handling Overload - Google SRE covers capacity-aware rejection, per-customer limits, client-side throttling, criticality, and retry suppression during broad overload.
  • Bulkhead - Resilience4j shows semaphore and fixed-thread-pool implementations with bounded parallelism, wait time, pool size, and queue capacity.
  • Circuit Breaking - Istio demonstrates connection and pending-request caps, outlier detection, and verification through overflow metrics.

These sources describe general resilience mechanisms, not a promise about any particular LLM provider’s limits. Treat provider documentation and your own measurements as the authority for numeric values.

Contract details to verify

Partition key. Decide which dimensions must never compete for the same permit. Provider is the minimum useful boundary when providers have independent failure domains. Add model or deployment when quotas differ. Add traffic class or tenant tier when interactive work must survive batch bursts. Do not create a partition for every label automatically; each extra pool reduces sharing and increases configuration work.

Admission limits. Define active_limit, pending_limit, connection_limit, and retry_limit for every partition. Envoy’s model is useful because it distinguishes active requests from requests waiting for a connection and from retries. Resilience4j offers a similar application-level choice between a semaphore bulkhead and a bounded thread-pool bulkhead. Pick one enforcement layer as the source of truth, or document how gateway and mesh limits compose.

Overflow behavior. Specify what happens at each boundary. A request that cannot enter the primary interactive pool might be eligible for the fallback interactive pool. A batch request might be delayed or rejected locally. A high-priority request might bypass a low-priority queue only if its own reserve exists. Return a stable internal reason such as bulkhead_overloaded; keep the external error contract consistent with the rest of the gateway.

Queue semantics. Set a maximum age as well as a maximum length. A queue that is short but older than the caller’s deadline is still a failure. Never count a request as admitted until the permit and the downstream connection are both available. If a request is canceled while waiting, remove it and release any reservation exactly once.

Retry ownership. Decide which layer owns retries. Envoy warns that retry volume can explode into cascading failure, so retries need their own cap rather than borrowing the initial-request pool invisibly. A retry should be charged to the same logical traffic class but a separate retry counter. If the provider is broadly overloaded, stop retrying and let the caller or an approved fallback decide.

Priority and fairness. Write down whether limits are per provider, per upstream cluster, per priority, or per tenant. Istio’s example uses connection and pending-request caps and then verifies overflow behavior; the same test shape can validate an LLM route. Monitor whether a high-priority pool is protected without allowing it to consume every shared worker or socket.

Observability. Emit one admission event for every decision and one completion event for every admitted request. Keep logs sanitized: request ID, provider label, partition, traffic class, priority, active count, pending count, limits, retry count, decision, reason, latency, and failure class are enough to reconstruct saturation. Do not log prompts, completions, or credential material in this control-plane record.

The following sanitized event uses arbitrary example counts, not a recommended configuration.

event: bulkhead_decision
request_id: req-42
provider: primary
partition: primary-interactive
traffic_class: interactive
priority: high
active: 24
active_limit: 24
pending: 4
pending_limit: 4
retry_count: 0
retry_limit: 2
decision: fallback
reason: pending_limit_reached
failure_class: upstream_overload
fallback_partition: secondary-interactive

Rollout contract. Start with shadow counters, then enforce one partition at a time. Inject a slow response from the primary route, fill its pending queue, cancel waiting requests, and confirm that the secondary route still admits work. Record the exact limit version with each decision so a later incident review can distinguish a code change from a provider event.

Failure modes

One global semaphore. Every provider shares one permit pool, so a stalled upstream consumes permits needed by the healthy route. The gateway reports “all providers unavailable” even though only one is unhealthy. Split the permit pool at the dependency boundary.

An unbounded pending queue. Requests wait until their caller deadlines expire, then all fail together and may trigger synchronized retries. Bound queue length and age, and make the overflow reason visible.

Retries bypass the partition. Initial calls are capped, but a retry client creates a second queue or uses a separate worker pool with no limit. This hides the real load and can create a retry cascade. Charge retries to an explicit budget and expose their overflow counter.

A noisy tenant consumes the reserve. One tenant or batch job fills the shared pool. Per-tenant or per-class partitions, quotas, or a lower-priority queue keep the impact local. Google SRE’s per-customer limit model is a useful mental model even when the implementation is local to one gateway.

QPS looks safe while work gets heavier. LLM requests can have different processing and connection costs. A fixed request-per-second limit may be too high for long or expensive requests and too low for short ones. Measure active work, queue age, and resource use rather than trusting one aggregate rate.

Fallback loops on full pools. The primary is full, fallback is full, and middleware alternates between them until the caller times out. Mark the request with its attempt and pool decisions, forbid a route that has already rejected it unless policy explicitly allows it, and stop when the request budget is exhausted.

Two layers disagree. The gateway admits a request that the mesh immediately rejects, or the mesh admits retries that the gateway did not count. Compare limits and overflow metrics at both layers, then choose one owner for each contract.

Metrics hide the partition. A healthy aggregate success rate can conceal a failing interactive lane or a single provider’s pending overflow. Alert on active, pending, rejected, retry, and completion counts per partition and priority.

Configuration changes drain the wrong pool. A rollout creates new workers with a different limit while old workers keep the previous one. Include the limit version in sanitized events and stage changes so operators can correlate the transition.

FAQ

Is a bulkhead just another circuit breaker?
No. A bulkhead limits resource ownership before a dependency failure spreads. A circuit breaker changes whether calls are attempted after failures or latency spikes. They complement each other: the bulkhead protects capacity, while the breaker protects the decision to call.

Should every model have its own pool?
Only when model deployments have different quotas, latency objectives, or failure behavior. Start with provider plus traffic class, then split a model or tenant when measurements show that sharing causes interference. Too many tiny pools waste capacity and complicate operations.

What should happen when a pool is full?
Choose one documented action per class: fast local rejection, a bounded wait, an approved fallback, or a degraded result. Do not silently enqueue forever and do not assume another retry will help. A full pool is a signal to execute policy, not an invitation to create more work.

How big should the pending queue be?
There is no safe universal number. Set it from the caller’s deadline, observed service time, and the amount of work you can complete before the response becomes useless. Then test queue age under a deliberately slow provider.

Does a fallback need its own bulkhead?
Yes. If fallback shares the saturated primary’s connection pool or worker queue, it cannot provide isolation. Give it an independent budget, while retaining a global guard that prevents all providers together from exhausting the gateway.

Reader next step

Create a small matrix today with rows for each provider and columns for interactive, batch, and retry traffic. For every cell, write the active, pending, connection, retry, queue-age, and overflow limits, plus the allowed fallback action. Instrument the sanitized admission fields above, then run a controlled test that makes one provider slow while sending normal traffic to the other.

After the test, compare the result with capacity planning before an LLM API fallback and overload signal triage for LLM API on-call . Keep the partition that preserves a healthy route’s response budget, and revise only one limit at a time so the next incident leaves a clear record.