A fallback is only as safe as the request artifact it receives. If the primary provider has already consumed a streaming body, handing that same stream to a CometAPI attempt can produce an empty or truncated request. If the primary timed out after accepting the request, sending a second copy can also repeat downstream work. Reliable failover starts before the first network call: capture one bounded, immutable representation of the logical operation, then create a fresh reader for each provider attempt.

Last reviewed: 2026-08-09

Direct answer

Make replayability an ingress contract. At the gateway boundary, validate the media type and size, read the body once into a bounded buffer or controlled spool, and freeze the bytes that define the operation. Keep a stable operation identifier and a fingerprint of the canonical payload beside that artifact. A provider adapter may construct a new request from those bytes, but it must not mutate the logical payload between attempts.

In Fetch-based code, a request body is a one-use stream. The MDN Request.clone() documentation says that clone() creates an exact copy for another consumer and throws a TypeError when the original body has already been used. Clone before consumption, or retain immutable bytes and construct a fresh request for every attempt. MDN also warns that cloning can enqueue unread data on the slower consumer without a limit, so a size cap and explicit buffering policy are reliability controls, not optional tuning.

Replayability does not grant permission to retry. Classify the first outcome. A connection failure before provider-visible work differs from a timeout after the request may have been accepted, and a partial stream differs again. The Azure Retry pattern guidance covers transient faults, delayed retries, logging, and idempotency because a service can finish an operation while the response is lost. When the outcome is unknown, reconcile or use a deduplication record; do not infer that no response means no work.

Use one operation ID for the user action and a separate attempt ID for each provider call. Store the canonical body fingerprint with the operation. If fallback is allowed, send the same semantic request through a new reader and carry the operation identity in the provider-supported correlation mechanism. Atomically record the first acceptable result. A late primary response can then be recorded as a loser instead of becoming a second user-visible result or triggering a second tool execution. This follows the concern in the AWS Builders’ Library article on idempotent APIs , which discusses client request IDs, semantic equivalence, and late-arriving requests.

The happy path is prepare once, call the primary, validate, commit once, and release the replay artifact. The error path is prepare once, classify the fault, check the remaining budget and unknown-outcome policy, call CometAPI with a fresh reader, and commit whichever acceptable attempt wins. If preparation fails, return an explicit non-replayable error or a deliberate degraded response; do not improvise a second read from an exhausted stream.

type ReplayPlan = {
  operationId: string
  bodyBytes: Uint8Array
  digest: string
  replayable: boolean
}

async function run(plan: ReplayPlan) {
  if (!plan.replayable) return surface('body_not_replayable')
  const first = await callProvider('primary', freshReader(plan.bodyBytes))
  if (first.kind === 'accepted') return commitOnce(plan.operationId, first)
  if (!isSafeToFailOver(first)) return surface(first.kind)
  const second = await callProvider('cometapi', freshReader(plan.bodyBytes))
  return commitOnce(plan.operationId, second)
}

The pseudocode leaves provider-specific headers and endpoint details to the adapter contract. Its invariant is what matters: no attempt reads from the same exhausted stream, and no attempt bypasses the decision about whether repeating the logical operation is safe.

Who this is for

This is for platform engineers, gateway maintainers, and on-call responders routing chat, responses, or other LLM operations across a primary provider and a CometAPI fallback. It is particularly useful when the service accepts streamed HTTP bodies, forwards tool definitions, has retries in multiple library layers, or can cause a database write, notification, or tool call after model output.

It is less relevant to a one-shot client that never retries or changes providers. Even there, the ingress and logging checks help if a proxy, SDK, queue, or browser runtime might replay a request on the client’s behalf.

Key takeaways

  • Treat the body as an immutable operation artifact. A mutable object serialized twice is not a replay contract.
  • Give every provider attempt a fresh reader. Request.clone() helps only before consumption; it is not a substitute for bounded storage.
  • Keep operation identity stable across primary and fallback attempts, while giving each attempt its own diagnostic ID.
  • Separate retryable, terminal, and unknown outcomes. Unknown means the remote service may have acted even though the client saw no response.
  • Commit one winner atomically. Late responses should enrich telemetry, not overwrite the result or execute a side effect again.
  • Bound memory, disk, and retention. Reject or deliberately degrade oversized or non-replayable bodies instead of silently truncating them.
  • Log enough to reconstruct the decision without logging prompts, body bytes, cookies, or sensitive headers.

Sources checked

The MDN Request clone() reference was checked for Fetch mechanics: a clone is an exact copy, cloning a consumed body throws, and stream cloning has a backpressure caveat. Those details support the preparation and fresh-reader rules here; they do not define a provider’s retry policy.

The Azure Architecture Center Retry pattern was checked for the reliability boundary around transient failures. It recommends choosing a retry strategy based on the fault, tuning delays and attempt limits, logging failures, and examining idempotency and transaction consistency. This article applies those principles to a multi-provider LLM gateway without assuming every LLM operation is safe to repeat.

The AWS Builders’ Library guidance on making retries safe was checked for the distinction between a repeated request and a repeated intent. Its discussion of client request IDs, semantic equivalence, and late-arriving requests motivates the operation record and single-winner commit. Provider-specific behavior still must be verified in your own contract.

Contract details to verify

Ingress and body source. Decide whether the gateway buffers bytes in memory, writes a bounded temporary spool, or receives a replayable application object. Set a maximum size before reading. Record whether the body was complete, decoding succeeded, and the source can produce a new reader. A partially read body must be marked non-replayable even if its prefix looks valid.

Canonical request. Define the fields that make the logical request different: model selection, ordered messages, tools, response-format requirements, sampling controls, stream mode, and tenant policy. Serialize deterministically or retain the original bytes. If an adapter translates fields, test semantic preservation and record its version. A digest detects byte drift; it cannot prove that providers interpret every field identically.

Identity and deduplication. Generate an operation ID at the user-action boundary and keep it stable for the permitted failover window. Generate a new attempt ID for each call. Store the intent fingerprint, preparation state, accepted-result state, and winner. Scope the record so a later, genuinely different action cannot reuse the old operation ID.

Outcome policy. Write down which errors permit another attempt. A connect failure before transmission may be retryable; a timeout after transmission is often unknown; a validation or policy rejection is terminal. For unknown outcomes, use a provider lookup, an application-level deduplication record, or a human-visible retry choice. Do not let a generic HTTP client retry beneath the gateway policy without counting that attempt.

Winner commit. Make publication conditional on an unclaimed operation record. The first response passing schema, safety, and business checks claims the winner; later responses are non-winning telemetry. Put downstream work behind the same operation guard or make it independently idempotent. Canceling a losing HTTP call helps load, but does not prove remote processing stopped.

Streaming. Specify whether fallback is allowed after headers, after the first token, or only before bytes reach the client. Track chunk count and completion state. If a stream breaks after visible output, do not concatenate a fresh generation unless a tested continuation protocol exists. For adjacent routing evidence, review partial-success classification before retrying .

Resource and security limits. Put separate limits on body bytes, spool lifetime, concurrent replay artifacts, and total fallback time. Do not store raw prompts by default. Redact sensitive authentication material, cookies, uploaded content, and tool arguments. If a digest is useful for correlation, expose only a shortened, non-sensitive representation in routine logs.

Operator evidence. A sanitized event can look like this:

operation_id: op_demo
attempt_id: attempt_1
provider: primary
body_state: prepared
replayable: true
body_bytes: 18420
digest: sha256:…
outcome: timeout_before_headers
emitted_chunks: 0
fallback_reason: transient_network
winner: cometapi
prompt_logged: false
sensitive_headers_logged: false

Keep fields stable across providers so an operator can follow one operation without reading content. Pair this with retry and backoff evidence for CometAPI gateway calls and response-shape checks before fallback promotion . Those links cover adjacent decisions; this article’s distinct focus is preserving the request body that makes those decisions meaningful.

Failure modes

The body is consumed before failover. In a Fetch runtime, cloning after consumption throws. The symptom is a clone error or a fallback attempt that never starts. Capture before the first read and test replayable; do not catch the clone error and continue with a fabricated empty body.

Cloning creates uncontrolled buffering. Two consumers with different speeds can cause unread data to accumulate. Large uploads or slow providers can turn a latency event into memory pressure. Enforce a size cap, use a bounded spool, and stop when the cap is exceeded. A hard, observable refusal is safer than silent truncation.

The retry payload drifts. Middleware may append a message, reorder tools, change a default, or toggle streaming. Compare the canonical fingerprint and adapter version before sending. If they differ, classify the operation as contract-invalid and require a new user action.

A timeout hides successful work. The provider may have generated output or started a tool call before the connection failed. Blindly sending the same command to CometAPI can duplicate work. Mark the outcome unknown, reconcile where possible, or return a controlled retry choice. Azure’s guidance warns that non-idempotent operations can execute more than once in this situation.

A partial stream is treated as clean failure. Once tokens are visible, a fallback generation cannot automatically restore the same sequence. Preserve the partial state, show a truthful degradation message, or use a tested continuation design. Do not label a partial response as no_work_done.

Retries multiply across layers. A gateway, SDK, queue consumer, and provider client can each retry the same operation. Assign one owner for the overall budget and pass an attempt number downward. Log the policy source so operators can identify hidden amplification.

A late response wins accidentally. A race can publish two results if the commit is not conditional. Use an atomic compare-and-set on the operation record, and make late attempts observable as losers rather than silently overwriting state.

Telemetry leaks the payload. Debug logging often captures request bodies during retry investigations. Keep prompt and tool content out of routine events, redact headers, restrict any temporary spool, and test redaction with known sentinel text.

FAQ

Is Request.clone() enough to make a fallback safe?

No. It addresses one-use body mechanics, not whether repeating the operation is semantically safe. Clone before consumption or create fresh readers from immutable bytes, then apply an outcome and idempotency policy before starting fallback.

Should every timeout trigger CometAPI?

No. A timeout before transmission may be retryable, while a timeout after transmission is often unknown. Use request lifecycle, emitted-byte state, provider evidence, and remaining budget to decide. If you cannot distinguish them, choose reconciliation or an explicit user retry rather than pretending certainty.

Can I replay a streaming request body?

Only if you captured a complete, bounded representation before it was consumed, or your runtime supplies a safe replayable source. A live socket stream that has advanced cannot be reconstructed from clone() afterward. Test slow readers and aborts, not only small in-memory examples.

Does a matching digest prove providers will behave the same?

No. It proves the gateway sent the same bytes or canonical representation. Adapters can differ in defaults, supported fields, tokenization, or tool semantics. Treat the digest as an audit aid and verify the response contract separately.

What belongs in the retry log?

Record operation and attempt IDs, provider, body state, bounded byte count, replayability, coarse outcome, elapsed time, fallback reason, emitted chunk count, adapter version, and winner. Do not record raw prompts, body bytes, sensitive authentication material, cookies, or tool arguments. The log should explain the decision without becoming another data store.

How long should the replay artifact live?

Keep it only for the maximum period in which a permitted retry or reconciliation can occur, subject to privacy and retention policy. Delete it after a committed result or terminal failure. If an unknown outcome needs longer reconciliation, store the smallest protected record that can establish identity instead of retaining the full prompt indefinitely.

Reader next step

Put this pattern through a controlled staging exercise before changing production routing. Capture a representative request and assert that its prepared bytes and fingerprint remain unchanged after adapter translation. Force a primary timeout before headers and confirm that CometAPI receives a fresh reader, the operation ID stays stable, and exactly one result is committed. Then force a timeout after transmission, inject a partial stream, and exceed the body-size cap. Each case should end in a documented state: fallback and commit, reconcile, surface a partial result, or reject as non-replayable.

Review sanitized events with the owner of downstream tool execution. If the team cannot tell whether work may have happened, the fallback policy is not ready. Add alerts for rising body_not_replayable, unknown_outcome, and late_response_loser counts, and sample the decision path without collecting payload content.

When the contract is explicit and tested, evaluate a managed fallback route with a clear expectation of what is preserved and what remains provider-specific. Start with CometAPI and carry the same operation, replayability, and winner-commit checks into your integration review.