Cap LLM Response Bodies Before JSON Decoding

Last reviewed: November 3, 2026 (Asia/Shanghai).

Direct answer

Enforce an LLM response body size limit during HTTP body consumption, before calling a full-body reader or JSON decoder. Open the response in transport streaming mode, count each incoming byte chunk, and reject a chunk that would exceed the route’s allowance before appending it. Close the response on rejection. A check on len(response.content) after a normal buffered request runs after the body has already been retained.

Choose a content-encoding policy at the same boundary. The conservative HTTPX example below requests an uncoded response, rejects any Content-Encoding field, and reads with aiter_raw() so it never invokes HTTPX’s content decompressor. Routes that need compressed responses require separately bounded decompression. Merely counting compressed bytes or selecting a small decoded iterator chunk size does not establish that bound.

The Python JSON reference warns that untrusted JSON can consume considerable CPU and memory and recommends limiting the input size. Its implementation-limits section also makes clear that the standard decoder does not supply application-specific size, nesting, or string-length limits. A byte admission check is one layer of the acceptance contract.

Who this is for

This guide is for Python gateway maintainers and SREs accepting complete JSON responses from LLM endpoints. Use it when an SDK or HTTP convenience method buffers a response before validation, especially where a provider, reverse proxy, or error handler could return a much larger body than the feature expects.

“Non-streaming” describes the model response format here: one complete JSON document. The HTTP client can still read that document incrementally. The helper does not parse an SSE event stream and does not prescribe a universal byte allowance.

Key takeaways

• Apply the limit before extending the application buffer or invoking a decoder.
• Treat Content-Length as metadata, never the sole enforcement mechanism.
• Specify whether a counter measures content-coded or decoded bytes.
• Requesting identity encoding still requires checking the received headers.
• Reserve memory for concurrent bodies, transient copies, and parsed objects.
• Separate local rejection and response closure from remote cancellation and replay permission.

Contract details to verify

Put the check before the first full-body read

Trace the adapter from the transport to the first JSON object. A sequence such as await client.post(...), then response.json(), then a size check is too late to control initial buffering. Likewise, inserting a check after an SDK has materialized a completion cannot recover the memory already consumed.

The HTTPX QuickStart describes streaming responses as a way to avoid loading the entire body into memory at once. Its binary-content section documents automatic content decoding; its streaming section distinguishes raw iteration, which skips that decoding. The HTTPX async guide supplies the corresponding aiter_raw() interface and context-managed closure used below.

Inventory event hooks, middleware, tracing integrations, and SDK wrappers as well. If an earlier layer calls aread(), accesses a fully buffered body, or decompresses it, the proposed admission point is no longer the first one. Test the real adapter path in addition to the isolated helper.

Distinguish the byte domains

RFC 9110 , sections 8.4 and 8.6, defines content coding and content length. Representation metadata generally describes the coded form. A small length on a gzip response therefore does not promise a small decompressed document. Content length can also be absent; continue counting the actual body delivered by the transport.

An optional length precheck can reject a clearly oversized declared body early, but it must not authorize an unlimited read when the declaration is small. Leave HTTP message framing and inconsistent length handling to the HTTP stack. Do not invent framing rules or accept malformed lengths to make a fixture pass.

For this guide’s narrow route, an uncoded body makes the raw body bytes the JSON input bytes. RFC 9110 section 12.5.3 defines Accept-Encoding: identity as a preference for no content coding. The reader still verifies the response. If compressed bodies are required, use a reviewed incremental decompressor with input and output allowances, bounded output production, supported-coding checks, and explicit handling of truncation and trailing data. Check that library’s allocation behavior before claiming a hard memory bound.

Read a bounded body with an explicit policy

Save this as response_limit.py. Supply an existing reusable httpx.AsyncClient, an approved endpoint, the endpoint’s payload and request-specific headers, and a positive max_bytes selected for that route.

class ResponseBodyRejected(ValueError):
    pass


async def read_json_body(client, url, payload, *, headers, max_bytes):
    if type(max_bytes) is not int or max_bytes <= 0:
        raise ValueError("invalid_response_byte_limit")
    request_headers = dict(headers)
    for name in list(request_headers):
        if name.lower() == "accept-encoding":
            del request_headers[name]
    request_headers["Accept-Encoding"] = "identity"
    request_headers["Accept"] = "application/json"

    async with client.stream(
        "POST", url, json=payload, headers=request_headers,
        follow_redirects=False,
    ) as response:
        response.raise_for_status()
        if response.status_code != 200:
            raise ResponseBodyRejected("unexpected_response_status")
        # This route accepts an uncoded body, with no Content-Encoding field.
        if "content-encoding" in response.headers:
            raise ResponseBodyRejected("content_encoding_not_allowed")
        media = response.headers.get("content-type", "").split(";", 1)[0]
        if media.strip().lower() != "application/json":
            raise ResponseBodyRejected("unexpected_response_media_type")

        body = bytearray()
        async for chunk in response.aiter_raw():
            if len(chunk) > max_bytes - len(body):
                raise ResponseBodyRejected("response_body_too_large")
            body.extend(chunk)
        return bytes(body)

The helper deliberately accepts only status 200, application/json, and an absent Content-Encoding field. This is a route policy, not a claim that other successful statuses, JSON media types, or encoded representations are universally invalid. It even rejects a response labeled Content-Encoding: identity; RFC 9110 reserves that token for its negotiation role and says it should not appear in the response field. Review any compatibility exception explicitly.

HTTPX documents that leaving the stream context closes the response. Status, header, byte-limit, and read failures therefore unwind the same ownership scope. Keep that scope around the read rather than returning its iterator for later use. The site’s HTTPX response cleanup guide covers the surrounding lifecycle and pool checks.

Define what the limit guarantees

The bytearray contains at most max_bytes of body data because the size check precedes every append. The next transport chunk already exists when the application examines it; its allocation is outside this check. Returning bytes(body) creates another representation while the buffer still exists, and text decoding and JSON parsing allocate further objects. This is a retained-body allowance, not an exact process-memory ceiling.

Budget the number of concurrent readers alongside their per-body limits. In an illustrative configuration, 100 readers each allowed one mebibyte represent 100 mebibytes of body data alone. Transient copies, parsed objects, transport state, and the rest of the worker need additional headroom. Use admission control and measured worker limits rather than treating that arithmetic as a memory forecast.

Keep network operation timeouts and an end-to-end deadline around the request. A body that trickles indefinitely can remain below its byte allowance. JSON decoding also needs its own resource plan: synchronous parsing cannot be preempted reliably by simply placing it inside an asynchronous timeout. Use an appropriately limited parser or an isolated, bounded worker when the threat model requires enforceable parsing limits.

After the bounded read, apply the route’s text-decoding policy, JSON decoder policy, endpoint shape checks, and application validation. The site’s duplicate-name decoding guide covers a separate ambiguity check. Passing a byte limit does not make the response semantically acceptable.

Happy-path operator workflow

Choose one complete-JSON route and record its allowed status, media type, content coding, byte limit, total deadline, and concurrent-reader allowance. Select the body limit from representative response sizes and the feature’s accepted envelope, including metadata and structured arguments. A request-side token allowance can help shape normal output, but it is not a byte-level transport guarantee.

Read through the helper, then pass only the returned bounded bytes to the decoder and validators. Record a route revision, configured byte allowance, accepted body size, terminal classification, and response-close outcome. Keep prompts, generated text, credentials, and raw exceptions out of routine logs.

Verify an ordinary response, a response exactly at the limit, and the same document split across several chunks. Accept the change when downstream decoding receives the complete bounded body and each request releases its response ownership.

Error-path operator workflow

Reject an unexpected encoding or media type before reading the body. On overflow, stop consuming immediately and leave the response context; do not drain the rest merely to preserve a connection. Rejecting an error status also avoids loading an arbitrary error page. If another part of the adapter needs a diagnostic body, give that read its own small limit and redaction policy.

Treat overflow as a local acceptance failure with a stable reason code. Distinguish it from a transport read error, HTTP status failure, deadline expiry, and decoder rejection. Do not feed a truncated prefix to the JSON parser or keep a partial result as an accepted completion.

Response closure does not establish that remote generation stopped, that no usage accrued, or that a replacement request is safe. Preserve the original operation’s attempt and deadline policy. Investigate a sudden rejection increase by comparing route revisions, response size distributions, and intermediary behavior before raising the allowance or enabling fallback.

Verify the boundary offline

With HTTPX installed, save this as response_limit_probe.py beside the helper and run python3 response_limit_probe.py. MockTransport intercepts every request; the reserved .invalid hostname is a fixture.

import asyncio
import httpx

from response_limit import read_json_body, ResponseBodyRejected


class Chunks(httpx.AsyncByteStream):
    def __init__(self, parts):
        self.parts = parts
        self.reads = 0
        self.closes = 0

    async def __aiter__(self):
        for part in self.parts:
            self.reads += 1
            yield part

    async def aclose(self):
        self.closes += 1


async def check(parts, limit, *, encoding=None, reason=None, reads=None):
    stream = Chunks(parts)

    async def reply(request):
        assert request.url.host == "fixture.invalid"
        assert request.headers["accept-encoding"] == "identity"
        headers = {"Content-Type": "application/json"}
        if encoding is not None:
            headers["Content-Encoding"] = encoding
        return httpx.Response(200, headers=headers, stream=stream)

    async with httpx.AsyncClient(
        transport=httpx.MockTransport(reply)
    ) as client:
        try:
            body = await read_json_body(
                client, "https://fixture.invalid/completion", {},
                headers={}, max_bytes=limit,
            )
        except ResponseBodyRejected as exc:
            assert reason is not None and str(exc) == reason
        else:
            assert reason is None and body == b"".join(parts)
        assert stream.closes == 1
        if reads is not None:
            assert stream.reads == reads


async def main():
    await check([b'{"ok":', b'true}'], 11)
    await check([b"1234", b"5678", b"unread"], 6,
                reason="response_body_too_large", reads=2)
    await check([b"1234567"], 6, reason="response_body_too_large")
    await check([b"unread"], 100, encoding="gzip",
                reason="content_encoding_not_allowed", reads=0)
    print("body acceptance, overflow, encoding, and closure: passed")


asyncio.run(main())

The overflow bytes are synthetic transport fixtures, not purported valid JSON completions. The first case supplies a valid JSON document exactly at its eleven-byte allowance. The next cases verify that overflow prevents any returned body and stops iteration before a later chunk. The final case proves that header rejection occurs without requesting a body chunk.

Local checks on November 3, 2026 in Asia/Shanghai used Python 3.12.3 and HTTPX 0.28.1. The compact probe above passed. Extended fixtures also covered a misleading small length in an in-memory response, wrong media type, HTTP 503, an unexpected 202, a controlled read failure, and cancellation during iteration. Response closure occurred once in those checks, and cancellation propagated. The mock length fixture tests the application’s counter; it does not simulate HTTP framing or override a network stack’s framing validation.

A separate synthetic HTTPX check decoded 132 gzip bytes into 100,000 repeated bytes. That observation illustrates the different byte domains; it does not measure production decompressor memory. No provider requests or real connection-pool tests were performed. Before rollout, repeat through the deployed transport and intermediaries, including omitted length, oversized error bodies, stalled reads, cancellation, and a subsequent healthy request.

Failure modes

The byte check follows a buffered request. The application rejects the result after retaining the body. Move the check to the first incremental read.

Only Content-Length is checked. Missing length or compressed content defeats the intended size guarantee. Count the bytes in the domain the policy actually limits.

Decoded chunks are small, so decompression is assumed bounded. An iterator can split output after the decompressor has produced it. Require a decompressor-level output contract or reject content coding.

An oversized prefix reaches downstream code. A helper returns partial bytes after catching overflow. Propagate rejection and verify zero accepted-result writes and zero tool executions.

Each request is bounded but concurrency is not. Many permitted bodies compete for the same worker. Enforce admission and reserve parsing headroom.

A larger allowance is the only response to new failures. A proxy or response-format change goes unexplained. Compare the exact path and preserve the rejection evidence before changing policy.

FAQ

Is a model’s output token limit enough?

It is not a transport-byte guarantee. The complete HTTP envelope, structured fields, and error responses still need a local body policy. Keep both controls for their respective purposes.

Must compressed responses always be rejected?

No. This helper chooses that narrow policy to avoid an unbounded decompression step. A route can support compression with independently reviewed encoded and decoded byte limits and bounded decompressor output production.

Does aiter_raw() expose HTTP chunk framing?

It skips HTTP content decoding. It does not turn the application into an HTTP framing parser. Leave message framing to the transport and count the body bytes it delivers.

Can I parse an oversized response’s first few bytes for diagnostics?

Do not treat a truncated prefix as a JSON document or an accepted result. Prefer status, sizes, and bounded reason codes in routine diagnostics. Any separately retained content needs the approved incident and redaction process.

Reader next step

Find one adapter’s first full-body read, move byte enforcement ahead of it, and write down the encoding policy. Run the fixtures through that adapter, then verify rejection, closure, and subsequent request progress through the deployed path. Approve the change only when oversized bodies cannot reach the decoder, storage, or tool execution and the worker’s concurrency budget includes parsing overhead.

Sources checked

• HTTPX: QuickStart — fetched November 2 UTC, November 3 in Asia/Shanghai. The streaming and binary-content sections support incremental reads, automatic content decoding, and raw iteration without content decoding.
• HTTPX: Async Support — fetched November 2 UTC, November 3 in Asia/Shanghai. Supports AsyncClient.stream(), aiter_raw(), and automatic response closure when the stream context exits.
• Python Software Foundation: Python 3.12 JSON reference — fetched November 2 UTC, November 3 in Asia/Shanghai. Supports input-size cautions and the distinction between application limits and interpreter or datatype limits.
• RFC Editor / IETF: RFC 9110, HTTP Semantics — fetched November 2 UTC, November 3 in Asia/Shanghai. June 2022 Internet Standard; sections 8.4, 8.6, and 12.5.3 support content coding, content length, and identity negotiation.

These maintained references and the standing HTTP standard support an evergreen implementation task. The route policy, sizing process, incident decisions, and rollout criteria are engineering recommendations. The dated observations apply only to the stated local fixtures and versions; they do not announce a provider incident or release.