Close CometAPI HTTPX Streams Before Connection Pools Stall

Last reviewed: October 29, 2026 (Asia/Shanghai).

Direct answer

Give every streamed CometAPI response one owner that closes it when consumption ends, including error and cancellation paths. In HTTPX, prefer async with client.stream(...) and keep consumption inside that block. If you use manual streaming through client.send(..., stream=True), explicitly arrange an eventual await response.aclose() on every path that receives a response.

The HTTPX async guide documents automatic response closure on leaving a stream block and warns that unclosed manual responses can leave connections open and cause resource leaks. An abandoned stream can therefore consume capacity needed by later calls. Changing the model or increasing retries through the same constrained client does not release that response.

Distinguish this from provider slowness. The HTTPX timeout guide defines PoolTimeout as failure to acquire a connection within the pool-wait limit. It is evidence of local acquisition pressure, not proof that the waiting call reached CometAPI or that the provider rejected it. Long legitimate streams and undersized pools can produce pressure too; prove a leak by inspecting response ownership and recovery after closure.

Who this is for

This guide is for Python gateway owners and SREs forwarding CometAPI streams through HTTPX, especially when requests succeed after a worker restart but later stall again. It focuses on the lifetime of an individual upstream response. Keep your existing queue limits, deadlines, SSE parser, and downstream backpressure controls alongside this cleanup contract.

Key takeaways

• Reuse a client at the application or worker scope; close each response at the request scope.
• Keep the response owner alive until its consumer finishes or fails.
• Treat manual streaming as an explicit ownership transfer.
• Classify pool wait separately from connect, read, and provider HTTP errors.
• Verify cleanup with forced consumer failures and cancellation, then verify pool recovery.
• Record local closure separately from protocol completion and remote execution.

Sources checked

These primary references were fetched successfully on October 28 UTC, October 29 in Asia/Shanghai:

• CometAPI: Create a chat completion documents POST /v1/chat/completions, bearer authentication, stream: true, SSE chunks, and the terminal data: [DONE] marker.
• HTTPX: Async Support documents shared client lifetimes, context-managed streams, manual streaming, and the requirement to close manually managed responses.
• HTTPX: Timeouts distinguishes pool, connect, read, and write timeouts and explains their associated exceptions.

These are maintained technical references for an evergreen lifecycle problem. They establish the API and client behavior used below; the ownership rules and rollout checks are engineering recommendations. This article does not claim a new provider release or incident.

Contract details to verify

Separate the client lifetime from the response lifetime

Create a reusable AsyncClient in your application’s startup or worker scope and close it during shutdown. Pass that client to request handlers. HTTPX warns against repeatedly creating clients in a hot loop because that defeats connection-pooling benefits.

Each request still needs its own response scope. Closing a response releases its transport resources according to the protocol and client implementation. It does not necessarily close the physical socket: a completed connection may remain reusable, while an abandoned HTTP/1.1 body may require discarding its connection. The useful invariant is that a completed local attempt no longer owns an unclosed response, rather than that socket counts fall to zero.

Assign ownership before dispatch. Name the component responsible for reading the body, closing it after an exception, and recording cleanup completion. If a streaming web framework takes over consumption, verify its finalization behavior on normal completion, consumer exceptions, and disconnects. A callback that runs only after a successful send leaves an error-path gap.

Keep consumption inside the stream scope

Save this helper as stream_cleanup.py. It illustrates response ownership for an injected, reusable HTTPX client:

async def consume_chat_stream(client, payload, api_key, consume):
    async with client.stream(
        "POST",
        "https://api.cometapi.com/v1/chat/completions",
        headers={"Authorization": f"Bearer {api_key}"},
        json={**payload, "stream": True},
        follow_redirects=False,
    ) as response:
        response.raise_for_status()
        if response.status_code != 200:
            raise ValueError("unexpected_stream_status")
        async for chunk in response.aiter_bytes():
            await consume(chunk)

The caller supplies a valid payload, a server-side credential, and an asynchronous consumer. The consumer runs before the block exits, so its exception unwinds the response scope. HTTPX’s documented context-manager behavior closes the response on exit. A status failure also occurs inside the scope. Redirect following is disabled; raise_for_status() prevents a redirect response from being consumed as a successful stream.

This helper forwards byte chunks to your consumer. HTTP chunks are not complete SSE events. Use a bounded decoder and the CometAPI stream parsing guide to handle event boundaries and [DONE]. The enclosing handler still owns the total deadline, downstream cancellation, event-size limits, and the decision about whether output is complete.

Do not return response.aiter_bytes() from inside this block for another component to consume later. By then the response scope has exited. Either keep iteration within the scope, as above, or use manual streaming with an explicit finalizer owned by the later consumer. HTTPX’s async guide shows a manual forwarding example with BackgroundTask(r.aclose); validate equivalent behavior in the exact framework and version you deploy.

Preserve three separate outcomes

Track protocol completion, consumer completion, and resource cleanup independently. Receiving [DONE] is protocol evidence. Successfully sending it downstream is delivery evidence. Closing the upstream response is cleanup evidence. One field cannot express all three.

For manual mode, wrap all processing after the response is obtained in try/finally, with await response.aclose() in the finalizer. Include branches that reject the status, reject headers, fail decoding, stop early, or encounter a disconnected consumer. Arrange framework cancellation so cleanup can run; test interruption during cleanup itself before relying on stronger guarantees.

Mark cleanup complete only after the close operation returns. If cleanup raises or is interrupted, retain a distinct cleanup-failed outcome and contain the affected worker or client according to your runbook. Do not report a released response merely because the user-facing handler ended.

Local closure does not prove remote generation stopped or determine the billed amount. Follow the request cancellation contract for propagating cancellation and suppressing further attempts when the user action ends.

Diagnose the pool before routing again

When pool waits rise, reduce new admission while inspecting the oldest owned responses, legitimate active streams, configured connection limit, and requests waiting for acquisition. Compare response-open and response-close events within a quiet test interval. A growing count of responses without a close event is a lead to investigate; verify whether they are still legitimately active.

Use the exact exception class. A ReadTimeout concerns waiting for response data, while PoolTimeout concerns acquiring a connection. Increasing the read timeout cannot repair a missing close. Increasing the pool size can postpone the symptom while allowing more retained responses.

Do not automatically mark CometAPI unhealthy or launch another attempt through the same exhausted pool. First determine whether the original attempt was dispatched and whether the logical action is still active. A separately budgeted fallback may help an otherwise eligible request, but it does not repair the local leak. The bulkhead pool guide explains how to keep that separate capacity bounded.

Reproduce cleanup and recovery locally

With HTTPX installed, run this check beside stream_cleanup.py. Its synthetic model, messages, and credential are consumed only by MockTransport; this is a cleanup test rather than a valid provider request.

import asyncio
import httpx
from stream_cleanup import consume_chat_stream

class Body(httpx.AsyncByteStream):
    def __init__(self):
        self.close_calls = 0

    async def __aiter__(self):
        yield b'data: {"choices": []}\n\n'

    async def aclose(self):
        self.close_calls += 1

async def check():
    body = Body()

    async def upstream(request):
        return httpx.Response(200, stream=body)

    async def reject_chunk(chunk):
        raise ValueError("controlled_consumer_failure")

    async with httpx.AsyncClient(
        transport=httpx.MockTransport(upstream)
    ) as client:
        try:
            await consume_chat_stream(
                client, {"model": "fixture", "messages": []},
                "fixture-placeholder", reject_chunk,
            )
        except ValueError as exc:
            assert str(exc) == "controlled_consumer_failure"
        else:
            raise AssertionError("consumer failure was lost")
        assert body.close_calls == 1

asyncio.run(check())

Local checks on October 28, 2026 UTC with HTTPX 0.28.1 verified closure once for normal consumption, an HTTP 503 response, a consumer exception, and task cancellation during consumption. The consumer-failure case above is the compact reproducible check. A separate loopback HTTP/1.1 test used max_connections=1: an unconsumed manual response held the only slot, the next request raised PoolTimeout, and a request succeeded after await response.aclose().

The loopback pool timeout was 50 milliseconds to keep that controlled test short; it is not a production recommendation. MockTransport checks finalization but does not exercise a real connection pool, which is why recovery was checked separately. Neither test contacted CometAPI. Repeat through your deployed transport, framework, and proxies, including cancellation during response acquisition and cleanup.

Record a small event containing route label, attempt number, client-library version, response-owned flag, protocol-done flag, consumer outcome, close outcome, cleanup duration, and exception class. Keep prompts, chunks, credentials, and raw exception objects out of routine logs. Use application lifecycle events unless your client explicitly exposes supported pool metrics; do not build operational checks on undocumented HTTPX internals.

Failure modes

• A manual response loses its owner. A handler returns early or a consumer raises before the close callback is installed. Establish the finalizer immediately after acquiring the response.
• An iterator outlives its response scope. A handler returns an iterator from inside client.stream(). Move consumption into the scope or transfer ownership explicitly.
• Only successful streams close. Status rejection, parser errors, and cancellation gradually retain capacity. Exercise every terminal branch.
• Pool wait becomes a provider error. A generic timeout handler starts repeated calls through the same constrained client. Preserve the acquisition failure class and contain local demand.
• A larger pool hides the defect. The service lasts longer before stalling. Require cleanup and recovery evidence before treating a limit increase as a fix.
• Idle sockets are labeled leaks. Reusable connections remain open after completed requests. Inspect response ownership and acquisition progress alongside socket counts.

FAQ

Should I read the entire rejected body to release capacity?

Do not drain an unbounded or stalled body just to preserve reuse. Close it through the client’s supported response API. Accept that the transport may discard a connection. A bounded resource lifetime matters more than keeping every socket reusable.

Is every PoolTimeout a leaked response?

No. Legitimate concurrency, long streams, and a small pool can exhaust acquisition capacity. A leak diagnosis requires an ownership gap or a response that remains open after its consumer ended. Verify recovery after cleanup under controlled load.

Should I close the shared client after each failed request?

Close the individual response. Repeatedly destroying the shared client removes pooling benefits and can disrupt unrelated in-flight work. Reserve whole-client replacement for a controlled drain or a specific recovery procedure.

Does this prevent duplicate output during fallback?

Cleanup releases local resources; it does not authorize replay. Preserve whether any output reached the consumer and apply the existing partial-response and cancellation policy before attempting another generation.

Reader next step

Audit one streaming handler from response acquisition to finalization. Add the failing-consumer test, then verify that a single-slot local pool recovers after closing an abandoned response. Expand to cancellation and framework disconnects before rollout. Require both balanced response ownership and successful acquisition by subsequent requests.

When those local checks pass, Start with CometAPI and run a small authorized staging stream using your configured model. Verify protocol completion, downstream delivery, and response closure as separate outcomes.