Last reviewed: 2026-08-30

Direct answer

Do not treat an HTTP/2 GOAWAY frame as proof that every in-flight LLM request failed. Stop assigning new requests to that connection, capture the GOAWAY last-stream ID, and compare it with the stream ID of each affected request. The RFC 9113 HTTP/2 specification defines the last-stream ID as the highest peer-initiated stream on which the sender might have acted. A request on a higher-numbered stream can be treated as unprocessed and retried on a new connection. A request at or below the boundary might already have reached application logic, so a missing response is not proof that it is safe to replay.

Use four recovery outcomes:

  1. For a 421 Misdirected Request, quarantine the current pooled connection and retry through a fresh connection to the intended authority. MDN’s 421 reference explicitly says clients may retry over a different connection.
  2. For GOAWAY where the request stream ID is greater than the last-stream ID and no response has started, reconnect and retry under the existing user-action attempt budget.
  3. For a stream ID at or below the GOAWAY boundary, classify the outcome as potentially processed. Reconcile it with application-level evidence before considering another execution.
  4. For a connection close without a usable GOAWAY boundary, a lost stream mapping, or a response that already started, fail closed against automatic replay and mark the outcome ambiguous or committed.

Changing providers does not remove ambiguity. Sending the same request through an alternate route, including CometAPI, is another execution attempt. Do that only after transport evidence proves the earlier request was not processed or application evidence reconciles the earlier outcome.

Who this is for

This guide is for engineers responsible for LLM gateways, provider SDKs, service meshes, reverse proxies, and multi-provider fallback policy. It assumes the system can associate an application request with its HTTP/2 connection and stream.

The focus is recovery after a client receives GOAWAY or 421. For the complementary server-side procedure used when your own gateway is shutting down, see the graceful stream-drain guide . If a response may already contain useful output, use the partial-success classification guide before authorizing another call.

Key takeaways

  • GOAWAY is a connection-drain signal, not a blanket per-request failure result.
  • The request stream ID and GOAWAY last-stream ID form the critical transport boundary.
  • A request above that boundary is eligible for transport-level retry on a new connection; a request at or below it is not automatically replay-safe.
  • A 421 response calls for a different connection to the intended authority, not an immediate assumption that the provider is unavailable.
  • The gRPC retry guide treats receipt of response headers as its commit point and limits transparent retry to cases where the client knows server application logic did not process the call. An HTTP LLM gateway can adopt an equally conservative local commit boundary.
  • Connection-pool behavior matters. The Envoy connection-pooling documentation describes draining an HTTP/2 pool connection after GOAWAY and opening connections for pending requests within configured circuit-breaker limits.
  • Exactly one layer should own semantic retry. SDK, proxy, gateway, and application retries must share one attempt budget or be explicitly disabled at the other layers.

Sources checked

  • The IETF HTTP/2 standard establishes GOAWAY behavior, stream boundaries, connection shutdown, and request-reliability semantics.
  • The MDN 421 Misdirected Request reference establishes that the response concerns an unsuitable scheme-and-authority destination and may be retried over a different connection.
  • The Envoy connection-pooling documentation provides a concrete implementation example for draining HTTP/2 and HTTP/3 connections after GOAWAY and creating replacement capacity.
  • The gRPC retry guide documents replay-based retry, transparent-retry constraints, response-header commitment, backoff, throttling, and retry observability.

Together, these references support the transport and retry boundaries in this runbook. They do not establish provider-specific deduplication or model-equivalence guarantees.

Contract details to verify

Define an explicit recovery state machine

Every request record needs state that survives teardown of its connection. At minimum, retain the user-action reference, attempt number, route, intended authority, HTTP version, connection reference, stream ID, whether response headers or content were observed, and any GOAWAY last-stream ID.

A conservative decision function looks like this:

if status_code == 421:
  quarantine the current connection
  retry once on a fresh connection to the original authority
else if goaway_seen and stream_id > goaway_last_stream_id and response_not_started:
  classify as safe_transport_retry
  retry on a fresh connection
else if request_never_left_the_local_queue:
  dispatch within the existing user-action attempt budget
else:
  classify as ambiguous_or_committed
  do not automatically replay

The 421 branch is an explicit protocol-directed exception to a general response-start commit rule. Its response is routing evidence, not model output. Do not expose a diagnostic 421 body as generated content.

Happy-path operator workflow

Suppose the gateway receives GOAWAY with last-stream ID 41 while an LLM request is assigned stream 43. No response headers or content arrived.

  1. Mark the connection as draining before scheduling more work.
  2. Snapshot the stream mapping and GOAWAY fields before the client library disposes of them.
  3. Confirm that stream 43 is greater than 41 and that the application has emitted no response content.
  4. Classify the request as safe for a transport retry.
  5. Open or select a non-draining connection. Do not place the retry back onto the connection that sent GOAWAY.
  6. Replay the immutable request envelope with the same user-action reference and a new attempt number.
  7. Count the replay against the same bounded retry policy. The per-action attempt-limit guide explains why reconnects and provider changes must not create independent retry loops.
  8. Record the final outcome and close the recovery record when a response reaches the local commit boundary.

This is a happy recovery path because the GOAWAY boundary supplies negative evidence that the higher-numbered stream was not processed.

Error and ambiguous-path operator workflow

Now suppose the request used stream 39 while the GOAWAY last-stream ID is 41, or the TCP connection disappeared before the gateway captured any GOAWAY frame.

  1. Mark the request potentially processed; do not infer failure from the absent response.
  2. Stop automatic retries and alternate-route fallback for that user action.
  3. Look for application-level completion evidence that your integration already exposes, such as a durable result reference, completion callback, or reconciled usage record.
  4. If a completed result is found, return or reconcile that result instead of issuing another generation.
  5. If non-execution can be proven, move the request into the normal retry path.
  6. If neither completion nor non-execution can be proven, surface an indeterminate outcome through the application’s documented error contract. Require an explicit policy or user action before creating another semantic attempt.
  7. If any streamed content reached the caller, preserve that partial outcome and classify it before deciding whether a new request is appropriate.

An alternate provider is not a shortcut around this branch. A second model response can create duplicate cost, conflicting answers, or repeated tool actions while the first provider continues processing.

Handle 421 on a fresh authority-specific connection

For a 421 response:

  1. Confirm that the status is actually 421 rather than a generic 4xx response.
  2. Record the intended scheme and authority plus the pooled connection reference.
  3. Remove or quarantine that connection for the affected authority.
  4. Establish a fresh connection whose routing and TLS server-name configuration match the intended authority.
  5. Retry once, subject to the user-action attempt limit.
  6. If 421 repeats, stop. Investigate connection coalescing, authority selection, and proxy routing rather than looping or automatically widening fallback.

This path normally repairs a connection-selection problem. It should not immediately penalize provider health unless separate evidence shows a provider outage.

Keep transport logs useful and sanitized

A recovery log should let an operator reconstruct the decision without storing prompts, generated text, tool arguments, tool results, cookies, or authentication material. A sanitized event can look like this:

event=llm_transport_recovery
action_ref=act-42
attempt=1
route_class=primary
authority=provider.example
http_version=2
connection_ref=conn-7
stream_id=43
goaway_seen=true
goaway_last_stream_id=41
goaway_error_code=NO_ERROR
http_status=none
response_headers_seen=false
response_content_started=false
request_bytes_sent=512
response_bytes_received=0
classification=safe_transport_retry
decision=new_connection_original_route
content_logging=disabled

Also retain event time, client-library version, retry owner, remaining deadline, attempt budget remaining, and whether a circuit breaker admitted the replacement connection. Normalize transport error codes; do not copy arbitrary GOAWAY debug data into general logs. The retry logging field guide provides a broader evidence model for reviewing gateway retries.

Verify ownership, capacity, and fallback compatibility

Before enabling automatic recovery, verify these contracts:

  • The HTTP client exposes both request stream IDs and GOAWAY last-stream IDs to the decision layer.
  • The connection pool stops new admissions after GOAWAY and can prove a retry used a different connection.
  • Only one component owns the semantic retry decision. Lower-level transparent retries must be visible to the shared attempt counter.
  • Replacement connections have circuit-breaker headroom. Envoy’s documented behavior creates new connections only within configured limits, so a drained connection can otherwise turn into queue growth rather than recovery.
  • The request body is immutable and replayable. Review the request replayability guide if the body is streamed or generated on demand.
  • An alternate route preserves required model capabilities, structured-output rules, safety policy, tool behavior, and data-handling constraints.
  • The application has an explicit error response for ambiguous execution and does not silently transform it into a generic retryable failure.

Failure modes

  • Retrying every stream after GOAWAY. Streams at or below the last-stream boundary might have been processed. Replaying them can duplicate generation, spend, or side effects.
  • Losing the stream map during teardown. A log that records GOAWAY but cannot connect it to application requests is not actionable. Treat those requests as ambiguous rather than guessing.
  • Reusing the draining connection. A pool that accepts new streams after GOAWAY can generate more failures and obscure which requests crossed the boundary.
  • Treating a bare connection close as equivalent to a clean GOAWAY. Without a trustworthy boundary, the client cannot use stream ordering to prove non-execution.
  • Retrying 421 through the same pooled connection. This can repeat the same authority or connection-coalescing mistake indefinitely.
  • Restarting after streamed output. A hidden retry can append a second answer to the first, repeat tokens, or produce a conflicting response. Once content is visible, treat the request as committed or partially successful.
  • Stacking retries across layers. One retry in the SDK, another in the proxy, and another in the application can multiply calls even though every component believes its local limit is small.
  • Exhausting connection circuit breakers. Draining is ineffective when no replacement connection can be admitted. Queue depth and deadline expiry then become the visible symptoms.
  • Routing an ambiguous request to a fallback provider. Provider diversity improves availability only when the earlier attempt is known not to be executing or has been reconciled.
  • Logging raw content for diagnosis. Prompt and response capture can create a separate security and privacy incident. Store decision metadata and short internal references instead.

FAQ

Does GOAWAY mean the provider is down?

No. GOAWAY initiates connection shutdown and can be used for graceful maintenance or an error condition. Evaluate its error code, last-stream boundary, and surrounding health signals before changing provider health state.

Is every request above the last-stream ID safe to retry?

At the HTTP/2 transport level, RFC 9113 allows higher-numbered streams to be treated as though they were never created. Your gateway should still require a trustworthy connection-to-stream mapping, no emitted response, a replayable request body, an unexpired deadline, and remaining attempt budget.

What should happen to a stream at or below the boundary?

Do not automatically replay it. The server might have passed some request data to higher-layer logic. Reconcile application evidence, return an existing result when possible, or expose an indeterminate outcome.

What if the connection closes without GOAWAY?

Treat the request as ambiguous unless another trusted signal proves it never left the client or never reached provider application logic. A socket error alone does not establish non-execution.

Should a 421 response trigger provider failover?

Usually the first repair is a fresh connection to the same intended authority because 421 specifically indicates that the selected server cannot answer for the request’s scheme-and-authority combination. Repeated 421 responses should stop the loop and trigger routing investigation. An alternate route remains a policy decision, not the default interpretation of 421.

Does an idempotency key make every ambiguous retry safe?

Only if the complete downstream path documents and enforces duplicate suppression for the operation, scope, and retention period you rely on. A locally generated identifier is not evidence that every proxy and provider honors it.

What changes for streaming responses?

Use a stricter commit boundary. Once response headers or any model content have been exposed, do not invisibly restart the generation. Preserve the partial response, classify its usability, and let the application decide whether a distinct new attempt is warranted.

When can the gateway route the retry through CometAPI?

Only when CometAPI is an approved alternate in the application’s routing policy and the request has been classified as safe to replay, or the earlier outcome has been reconciled. Preserve the original user-action reference, increment the attempt count, and revalidate the request and response contract for the selected route.

Reader next step

Implement the decision function behind a feature flag and test at least five cases: a request above the GOAWAY boundary, a request below it, a connection loss without GOAWAY, a 421 followed by a successful fresh connection, and a stream where response content has already started. Use the controlled fault-injection guide to exercise those paths without waiting for a production incident.

Before enabling automatic replay, alert on missing stream mappings, retries sent to draining connections, repeated 421 responses, and attempts classified as ambiguous but nevertheless routed again. Confirm that all retrying layers contribute to one bounded attempt count.

If your approved fallback design needs a separate model-access path after a request is proven safe to replay, Start with CometAPI .