Last reviewed: August 11, 2026

Direct answer

Drain a CometAPI streaming gateway in three ordered phases: stop admitting new work to the retiring instance, preserve streams it has already accepted, and terminate its workers only after every active stream reaches a recorded outcome or the shutdown deadline expires. Starting the termination clock before removing the instance from normal routing creates a race in which new streams arrive while old ones are still draining.

The CometAPI Chat Completions documentation describes streaming output as Server-Sent Events containing incremental chunks, followed by a data: [DONE] marker. A gateway therefore needs more state than an open-connection count. For each stream, record whether the upstream terminal marker was received, whether it was forwarded to the client, and whether the downstream write completed. Receiving [DONE] upstream is not a successful drain if shutdown interrupts the final client write.

Separate requests into three states. A request rejected before an upstream stream starts can be routed to another healthy gateway. A request that has produced no downstream bytes may be eligible for normal fallback rules. A request that has already delivered one or more chunks is a partial result and should not trigger blind replay. In that case, classify the result before retrying , because a second generation can duplicate text, diverge from the first response, or incur additional work.

Set a finite drain deadline, but derive it from the longest stream the product intends to support rather than borrowing a generic request timeout. The normal streaming timeout, the orchestrator grace period, and the proxy drain period must agree. The practical goal is to let expected streams complete while ensuring that a stuck connection cannot block a deployment indefinitely.

Who this is for

This guidance is for backend and platform engineers who operate an application gateway, reverse proxy, service mesh, or container workload that relays CometAPI streaming responses. It is also useful to on-call engineers investigating client reports of responses ending during a deployment.

The design applies whether the gateway owns the SSE connection directly or relies on a proxy sidecar. The implementation details vary, but the contract does not: new work must leave the retiring instance, admitted streams need explicit lifecycle tracking, and forced termination must be distinguishable from upstream failure and client cancellation.

Key takeaways

  • Remove the instance from new-request admission before starting its application drain.
  • Track logical streams, not only TCP connections or worker processes.
  • Treat [DONE] as complete only after it has been forwarded successfully.
  • Keep partial-output requests out of automatic fallback unless the application has an explicit continuation or replay contract.
  • Make the proxy drain period, application deadline, and orchestrator grace period consistent.
  • Use graceful proxy controls; a fast shutdown signal defeats connection draining.
  • Log outcomes and timing metadata without prompts, generated content, or request headers.

Sources checked

  • CometAPI Chat Completions documentation establishes the SSE chunk format, the terminal [DONE] marker, and the warning that request and response details can differ between model providers.
  • Kubernetes Pod Lifecycle documents Pod termination, preStop execution, termination signals, and the termination grace period. It also notes a default graceful-termination period of 30 seconds, which should not be assumed to fit every streaming workload.
  • NGINX controlling nginx distinguishes fast shutdown from QUIT graceful shutdown and explains that old workers close listening sockets while continuing to serve existing clients.
  • Envoy draining documentation explains graceful listener draining, HTTP/1 connection discouragement, HTTP/2 GOAWAY, and waiting for existing streams when supported connection pools are removed.

These sources describe separate parts of the path. None replaces an end-to-end deployment test through the exact load balancer, proxy, gateway, client library, and CometAPI route used in production.

Contract details to verify

Define the drain contract before changing deployment signals. At minimum, make these decisions explicit:

  • Admission boundary: Identify the component that stops new requests from reaching the retiring gateway. Changing readiness and stopping the application listener are different events; record both.
  • Stream registry: Register a stream before opening the upstream request, then remove it only after a terminal outcome is persisted. Include streams waiting for their first upstream chunk.
  • Completion boundary: Distinguish done_received from done_forwarded. The latter should mean the terminal event passed through the downstream writer without an error.
  • Deadline ownership: Select one application drain deadline and ensure the proxy and orchestrator allow enough time for it. A preStop hook consumes part of the Pod termination window, so account for its work rather than adding it mentally after the grace period.
  • Fallback boundary: State whether zero-byte failures can use another route and how partial output is surfaced. Do not let process termination masquerade as a provider error.
  • Protocol behavior: Test HTTP/1 and HTTP/2 separately. Envoy can use Connection: close for HTTP/1 and GOAWAY for HTTP/2 during draining, but those signals do not remove the need to track each application stream.

A concrete happy-path workflow is:

  1. Before deployment, confirm that the configured termination window exceeds the supported stream duration plus route-withdrawal and process-exit time.
  2. Start the drain state and make the retiring gateway ineligible for new normal traffic.
  3. Stop accepting new application work on that instance. Requests that arrive during propagation should receive the gateway’s documented not-ready outcome before any upstream stream is opened.
  4. Snapshot the active-stream count and the oldest stream age. Keep serving already admitted requests.
  5. Forward each SSE chunk without buffering it until process exit. Continue recording whether downstream writes succeed.
  6. When the gateway receives [DONE], forward it, confirm the downstream write outcome, and mark that stream complete.
  7. Exit when the active-stream count reaches zero. The proxy and worker process should then perform their graceful shutdown steps.
  8. Verify that the replacement instances are healthy and that the deployment produced no forced stream terminations.

The error-path workflow needs equally explicit branches:

  1. If the upstream fails before any response bytes reach the client, apply the normal bounded fallback policy, provided the target route remains compatible.
  2. If any chunks have reached the client, mark the result partial. Do not synthesize [DONE], and do not silently restart generation from the beginning.
  3. If the client disconnects, classify the event as downstream cancellation and stop upstream work where the stack supports cancellation. Do not count it automatically as an upstream availability failure.
  4. If the drain deadline arrives with active streams, record forced_shutdown for each unfinished stream, close them through the documented application path, and let the process exit. The client-visible behavior should match the product’s incomplete-stream contract.
  5. Pause the rollout when forced terminations exceed the deployment’s accepted threshold. Investigate timing, admission, and proxy sequencing before advancing to more instances.

Use structured, sanitized events so operators can reconstruct either path without storing content. A completion event can look like this:

{
  "event": "stream_drain_complete",
  "request_id": "req-7f3a",
  "deployment_id": "deploy-42",
  "route": "chat-completions",
  "model": "configured-model",
  "stream_state": "complete",
  "done_received": true,
  "done_forwarded": true,
  "chunks_forwarded": 37,
  "downstream_bytes": 4821,
  "duration_ms": 18420,
  "drain_elapsed_ms": 5210,
  "deadline_remaining_ms": 11580,
  "client_disconnected": false,
  "fallback_started": false,
  "error_class": null,
  "content_capture": "[REDACTED]"
}

Keep field values bounded and enumerable where possible. Useful fields include route, configured model, deployment identifier, worker identifier, request identifier, admission timestamp, first-chunk timestamp, terminal-marker state, downstream-write state, chunk count, byte count, disconnect origin, and forced-shutdown reason. Exclude prompts, generated text, request headers, and raw tool arguments. If a separate content-capture system is required for another purpose, it should not be part of routine drain telemetry.

Also verify reload behavior at every layer. NGINX documents QUIT as graceful shutdown and TERM as fast shutdown; using the wrong signal can cut connections even when the application waits correctly. Envoy requires its graceful listener-drain sequence before shutdown if that behavior is desired. A container grace period cannot compensate for a proxy that closes listeners and connections immediately.

Finally, keep streaming policy separate from non-streaming behavior. A short non-streaming deadline may be appropriate for ordinary requests while being destructive for SSE. Review the streaming and non-streaming timeout checks before treating one timeout value as universal.

Failure modes

  • New streams enter after drain starts. Readiness changed, but a load balancer or local proxy still admitted work. The active count never settles, and the oldest stream can outlive the deployment window. Stop admission at the application boundary as well as the routing layer.
  • The grace period is shorter than supported generation time. Kubernetes documents a 30-second default, but that is a platform default rather than evidence that 30 seconds fits the product. Measure real stream durations and set an explicit ceiling.
  • The proxy receives a fast shutdown signal. The application waits, but NGINX, Envoy, or another intermediary closes the client connection first. Validate the actual process signal and proxy drain command used by the deployment controller.
  • Upstream completion is confused with downstream completion. [DONE] reached the gateway, but the final event was still buffered or its write failed. Record receipt and forwarding separately.
  • A partial stream triggers fallback. The user sees the beginning of one answer followed by a second generation, or the application hides the first output and repeats work. Partial output requires an explicit product decision, not a generic retry rule.
  • A client disconnect is treated as provider failure. The gateway starts replacement work after the user has left. Follow the same cancellation boundary described in stopping canceled requests from triggering fallback .
  • HTTP/2 draining is tested only with connection counts. One connection can carry multiple streams. Track logical request streams and verify how GOAWAY affects the actual client and proxy versions in use.
  • Provider-specific behavior is assumed uniform. CometAPI warns that parameters and response fields can vary across providers. Run the drain test for each production route and model family instead of extrapolating from one successful stream.
  • The rollout advances despite forced terminations. Capacity looks healthy, but every retiring instance clips its longest requests. Make incomplete-stream events a rollout signal, not merely a post-incident log field.

FAQ

Does removing a Pod from readiness automatically preserve its SSE streams?

No. Routing eligibility is only one part of the sequence. The application and any proxy still need to keep admitted streams alive, respond correctly to the termination signal, and exit within the configured grace period.

How long should the drain period be?

Use the longest stream duration the product supports, add measured time for routing withdrawal and orderly process exit, and keep a finite upper bound. Compare that result with the proxy drain setting and orchestrator termination grace period. Do not rely on an unmeasured default.

Is seeing [DONE] enough to mark a request successful?

Not by itself. It proves that the terminal marker reached one point in the gateway. Mark the client-facing stream complete only after the downstream writer reports that the marker was forwarded successfully.

Should shutdown trigger CometAPI fallback?

Shutdown should route new work to healthy instances. For an admitted request, fallback depends on what the client has already received. A zero-byte failure can follow the normal fallback contract; a partially delivered stream should not be replayed automatically.

Can NGINX or Envoy handle everything without application changes?

They can stop or discourage new connections and preserve supported existing traffic during a graceful drain. The application still owns request admission, logical stream state, completion classification, cancellation, and sanitized outcome logging.

What should happen when the deadline expires?

The process must be allowed to terminate. Record each unfinished stream as forced shutdown, preserve the reason and timing fields, and expose the documented incomplete-stream behavior to the client. Never manufacture a successful terminal marker merely to make telemetry look clean.

Reader next step

Implement an active-stream registry with separate flags for upstream terminal receipt and downstream terminal forwarding. Then run a controlled deployment test with a stream that remains active across the drain transition. Verify that new requests move to healthy instances, the existing stream reaches [DONE], the retiring process exits before its deadline, and the logs contain no content.

Repeat the test for a client disconnect, an upstream error before the first chunk, an upstream error after partial output, and a forced deadline. Treat unexplained fallback or any clipped happy-path stream as a deployment blocker until the admission and shutdown sequence is corrected.

When that gateway contract is ready to test against a unified model endpoint, Start with CometAPI .