Last reviewed: 2026-09-15

Direct answer

Treat the caller’s deadline as a single, non-renewable budget that follows the request through every retry and fallback provider. At ingress, convert the caller’s timeout into an absolute deadline using the gateway’s monotonic clock. Before each attempt, calculate the remaining budget and set the provider timeout to the smaller of that remainder and the attempt cap. If the remaining budget is zero or below a safety floor, stop and return a timeout; do not start another provider call.

This is the practical meaning of deadline propagation. The gRPC deadlines guide distinguishes a deadline (a point in time) from a timeout (a duration), and explains that downstream calls should honor the original deadline. It also notes that elapsed time must be deducted when a deadline is converted to a timeout, which avoids clock-skew surprises. The Go context package documentation applies the same idea to API boundaries: contexts carry deadlines and cancellation, and all derived contexts are canceled when the parent is canceled or expires.

Keep three controls separate:

  1. End-to-end deadline: the user-visible limit, owned by the ingress request.
  2. Per-attempt timeout: the maximum time one provider call may consume, including connect and response phases as appropriate for your client.
  3. Idle timeout: the maximum silence interval on a stream or connection. It is not a substitute for the end-to-end deadline.

The Envoy route component reference documents this distinction and exposes per_try_timeout for each upstream attempt and idle-timeout controls for routes. Configure those limits so a proxy cannot outlive the application budget. Finally, classify the outcome before deciding whether to retry or fail over. Google AIP-194 treats UNAVAILABLE as generally retryable, while CANCELLED and DEADLINE_EXCEEDED must be honored rather than automatically retried. The AWS SDK retry behavior reference is a useful implementation comparison: exponential backoff with jitter, a maximum-attempt setting, and a retry-quota budget all prevent retries from becoming an outage amplifier.

Happy-path operator workflow

Suppose a client gives the gateway a 2.5-second deadline. The gateway records deadline_at and starts the primary provider with a 1.2-second attempt cap. The provider returns a transient 503 after 700 milliseconds. The gateway subtracts elapsed time, waits for a jittered delay only if enough budget remains, and starts the next attempt with timeout_ms = min(remaining_ms, 1200). If the fallback responds in 900 milliseconds, the gateway returns that response before the original deadline. The response metadata should identify that a fallback was used, but the user does not need to know the internal retry count unless your product contract exposes it.

Error-path operator workflow

If the first provider consumes the entire 2.5 seconds, the gateway marks the request deadline_exhausted, cancels the provider call, and returns a bounded timeout response. It does not launch CometAPI or any other fallback after the deadline. If the client disconnects at 1.1 seconds, cancellation wins over retryability: cancel the active provider request, release its connection, and suppress all subsequent attempts. If a provider returns INVALID_ARGUMENT, PERMISSION_DENIED, or another non-transient status, record the classification and fail fast unless an explicit application policy says otherwise. This keeps a bad request from being replayed across every provider.

Who this is for

This design is for SREs, platform engineers, and service owners running an LLM gateway that can retry a provider, switch to CometAPI, or route between regions. It is especially relevant when requests stream tokens, invoke tools, or pass through multiple proxies. Product teams benefit too: a clear deadline contract lets them choose user-facing latency targets without guessing how many hidden retries a gateway might add.

You do not need gRPC or Go to use the pattern. The sources provide concrete terminology, while the same arithmetic applies to HTTP clients, queue workers, and sidecar proxies. What matters is one authoritative deadline, explicit cancellation, and a retry policy that cannot extend the caller’s promise.

Key takeaways

  • Convert the incoming timeout to an absolute deadline once; derive every attempt timeout from remaining time.
  • Apply min(remaining_budget, per_attempt_cap) before every provider call, including fallback calls.
  • Reserve time for serialization, network overhead, backoff, and response delivery instead of spending the entire budget upstream.
  • Treat cancellation and deadline expiry as terminal for the user action. Do not retry them automatically.
  • Keep route idle timeouts, per-try timeouts, and total request timeouts distinct in proxy configuration.
  • Log decisions with sanitized, low-cardinality fields so an on-call engineer can reconstruct the timeline without prompts or credentials.

Sources checked

The following public sources were refetched for this article:

  • gRPC Deadlines defines deadlines, client and server cancellation, and downstream deadline propagation. It recommends explicit realistic deadlines and explains converting an absolute deadline to a remaining timeout.
  • Go context package documents propagation of deadlines and cancellation across API boundaries, derived contexts, and the need to call cancellation functions to release timers and resources.
  • Envoy HTTP route components describes route and per-attempt timeout fields, including the difference between idle timeout and an overall request timeout.
  • Google AIP-194 gives a status-code policy: retry safe, non-transactional unary work on UNAVAILABLE, but honor CANCELLED and DEADLINE_EXCEEDED and avoid automatic retries for invalid or unauthorized requests.
  • AWS SDK retry behavior details exponential backoff with full jitter, max attempts, retry quotas, and the difference between standard and adaptive retry modes.

These sources are independent: a protocol framework, a language standard library, a proxy, a cross-API design proposal, and a cloud SDK guide. Together they support the deadline arithmetic and retry guardrails, while provider-specific limits still need verification in your own client libraries.

Contract details to verify

Ingress contract. Decide whether callers send a relative timeout, an absolute timestamp, or both. Normalize to an internal absolute deadline and reject missing or implausibly long values. Use a monotonic clock for subtraction so wall-clock adjustments cannot add budget.

Propagation format. If a downstream service accepts an absolute deadline, pass a remaining timeout instead when clock domains differ. gRPC explicitly converts deadlines to timeouts with elapsed time deducted. For HTTP, choose one documented header or metadata field and strip conflicting values at trust boundaries. Treat that field as routing metadata, not as a credential or authorization mechanism.

Attempt budget. Set a cap for connect, first-byte, and read phases, then enforce attempt_timeout = min(per_attempt_cap, remaining - reserve). The reserve covers response encoding and delivery. A cap of zero should mean “no attempt,” not “wait forever.”

Cancellation. Wire client disconnects and deadline expiry to the transport cancellation primitive. In Go, pass the context as the first argument, derive child contexts for each attempt, and call the returned cancel function on every control-flow path. For other runtimes, verify that canceling the task actually interrupts socket reads and model streaming.

Retry and fallback policy. Maintain an allow-list of transient statuses and transport errors. Require the request to be safe to repeat before replaying it; AIP-194 warns against automatic retries for transactional or side-effecting operations. Stop when the attempt count, retry quota, or deadline is exhausted. Jitter backoff and cap it by the remaining budget.

Proxy alignment. Compare Envoy per_try_timeout, route timeout, and stream idle timeout with the application deadline. Ensure the proxy does not reset a stream for idleness while tokens are legitimately expected, and ensure it cannot keep an upstream attempt alive after the gateway has canceled it. Test both HTTP/1.1 and HTTP/2 paths used by your clients.

Observability contract. Emit one event at ingress, one at each attempt start and end, and one final decision event. Use a stable request ID and trace ID, but never log prompt text, message content, authorization material, or full request bodies. Store durations and classifications, not sensitive payloads.

A minimal sanitized record can look like this:

{
  "request_id": "req_7f3c",
  "trace_id": "trace_91ab",
  "provider": "primary",
  "model_alias": "model-a",
  "attempt_index": 1,
  "deadline_at": "2026-09-15T12:00:02.500Z",
  "remaining_ms": 1800,
  "timeout_ms": 1200,
  "status_class": "5xx_transient",
  "outcome": "retry_eligible",
  "retry_decision": "continue_if_budget_remains",
  "cancellation_reason": null
}

Use short synthetic IDs like the example above; do not substitute real secrets or user identifiers. Aggregate remaining_ms into coarse buckets for metrics to avoid high-cardinality labels.

Failure modes

No deadline at ingress. A client that waits forever can pin workers and connections. Require an explicit deadline or apply a conservative service default, then expose the applied value in telemetry.

Resetting the timeout on fallback. Starting CometAPI with a fresh full timeout violates the user contract. Derive the fallback timeout from the same absolute deadline and leave a delivery reserve.

Retrying after cancellation. A disconnected browser or canceled job should stop work. If cancellation is converted into a generic 5xx, a retry layer may replay it; preserve a distinct cancellation reason through the stack.

Fixed backoff that overruns the budget. A two-second sleep is unsafe when only 400 milliseconds remain. Compute jittered delay, compare it with remaining time, and skip the retry when the delay plus minimum attempt time cannot fit.

Proxy and client disagree. An Envoy idle timeout can terminate a quiet stream even though the application still has total budget, while an oversized route timeout can let the proxy outlive the caller. Test effective values from configuration, not just intended values.

Misclassified statuses. Retrying INVALID_ARGUMENT, authorization failures, or quota exhaustion can multiply load and cost. Conversely, treating a short network UNAVAILABLE as permanent can create avoidable user errors. Keep the classification table versioned and review it with provider owners.

Cancellation leaks. If child timers or goroutines survive a completed attempt, they can consume resources and emit late callbacks. Ensure every derived operation is canceled and that late provider responses are discarded after the decision is committed.

Streaming ambiguity. Once tokens have been delivered, a retry may duplicate visible output. Mark a stream as committed after the first user-visible chunk and require an explicit resume or replay contract before switching providers. A deadline policy cannot by itself make partial output safe to duplicate.

FAQ

Should I send an absolute deadline or a timeout to providers? Keep an absolute deadline inside the gateway, but send a remaining timeout when crossing services with independent clocks. Recompute it immediately before each call.

How many fallback attempts should fit? There is no universal number. Choose the maximum from load tests, then gate each attempt on remaining time, retry quota, and operation idempotency. One well-timed fallback is safer than several attempts that cannot finish.

Do idle and total timeouts need the same value? No. An idle timeout protects a connection from silence; a total deadline protects the user action. Set the idle value to match expected token gaps and keep the total deadline tied to product latency goals.

What should happen on DEADLINE_EXCEEDED? Stop automatic retries and return a bounded timeout result. Preserve the original cause in telemetry so operators can distinguish provider slowness from a caller that supplied an unrealistically short budget.

Can adaptive retry limiters replace deadline propagation? No. A limiter can reduce request rate, but it does not know the caller’s remaining deadline or cancel work already in flight. Use both only when their delays are charged against the same budget.

Reader next step

Pick one production route and trace a single request from ingress through primary, retry, and fallback. Record the deadline, remaining budget, per-attempt timeout, backoff delay, and final decision at each hop. Add a test that disconnects the client mid-stream and another that returns a transient 503 with less than one attempt window remaining. Then compare your route settings with the retry and backoff evidence checklist and tune concurrency only after the budget behavior is correct. The adaptive concurrency limits guide can help you prevent a queue of already-expired work from consuming fallback capacity.

Document the chosen deadline and cancellation contract in your service runbook, alert on requests that finish with less than the delivery reserve, and review the sanitized timeline during the next reliability exercise.