Prompt caches can make repeated LLM requests cheaper or faster, but they are not portable conversation state. A reliable CometAPI fallback treats cache metadata as an optional optimization and treats the canonical request, policy, and budget as the source of truth. If the primary provider fails, the fallback adapter rebuilds the request for the target provider, assumes the cache may be cold, and records the uncertainty instead of promising a cache hit that cannot be verified.

Last reviewed: 2026-08-28

Direct answer

Keep three boundaries separate: the canonical prompt, provider cache controls, and provider usage accounting. Store the canonical prompt and its version in your own request envelope. Let each provider adapter add its own cache hints only for that attempt. On a provider switch, discard the primary provider’s cache key, breakpoint, TTL interpretation, and state pointer unless the target adapter has explicitly documented that the value is valid there. Re-send the canonical input and budget for the uncached case.

The OpenAI Responses create reference exposes prompt_cache_breakpoint, prompt_cache_options.ttl, prompt_cache_key, and cache-related usage details. The Anthropic prompt caching guide uses a different cache_control model, with automatic or explicit breakpoints, a default five-minute lifetime, and an optional one-hour lifetime. The Gemini context caching guide describes implicit caching and reports cache hits through usage.total_cached_tokens. Those controls are useful within their respective adapters, but their names and lifecycle rules do not form a portable contract.

Do not confuse an OpenAI previous_response_id with a cache handle that can be handed to another provider. The OpenAI conversation state guide describes that pointer as part of a response chain, notes that an uncached ID can require full context, and says prior input tokens remain billed. A cross-provider fallback therefore needs a complete, provider-neutral transcript or request representation of its own.

Use this operator workflow:

  1. Create an immutable request envelope containing a short-lived request fingerprint, tenant policy, prompt version, model intent, tool or output requirements, and a fallback budget. Keep the prompt itself out of routine logs.
  2. Pass that envelope to the primary adapter. The adapter may add a cache breakpoint or other provider hint, but it must record which hint it requested and which provider owns it.
  3. On a complete response, commit the business result and usage record together. Mark cache observations as confirmed only when the response reports them.
  4. If the primary returns a retryable error, times out, or closes before a terminal response, mark cache state as unknown and set usage_state to unavailable. Leave every unobserved usage counter unknown; do not count an expected hit as savings or write missing counters as zero.
  5. Before calling CometAPI, render the same canonical envelope through the fallback adapter. Omit primary-only cache controls and state IDs, and reserve enough budget for a full uncached input plus the fallback output.
  6. Commit the fallback result with an attempt number and a link to the same request fingerprint. If the uncached estimate exceeds policy, queue or degrade deliberately instead of silently overspending.

A small normalized ledger makes the decision auditable:

request_fingerprint: h7
prompt_version: p3
provider: primary
model_alias: model-a
route_attempt: 1
cache_requested: explicit
cache_observed: unknown
usage_state: unavailable
cache_read_tokens: unknown
cache_write_tokens: unknown
input_tokens: unknown
output_tokens: unknown
total_tokens: unknown
fallback_reason: upstream_timeout
previous_state_present: false

Who this is for

This is for platform engineers, SREs, and gateway owners who route one logical LLM request through more than one provider or endpoint. It is especially relevant when a gateway exposes a CometAPI fallback, supports long prompts or multi-turn conversations, or enforces per-request and per-tenant spend limits. It is not a provider pricing comparison. The goal is a stable adapter contract that remains correct when a cache is absent, expired, unavailable, or represented differently.

Key takeaways

  • A cache hit is a performance observation, not proof that the request is complete or safe to retry.
  • The canonical prompt, policy, and prompt version belong to the gateway; cache hints belong to provider adapters.
  • Treat a timeout after upstream acceptance as cache state unknown and usage state unavailable, not as a confirmed miss or hit.
  • A confirmed zero is different from unknown: record zero only when the provider response or adapter contract proves it.
  • A failed or partial attempt has unavailable usage; unobserved counters stay unknown rather than being recorded as zero.
  • Normalize read, write, and total usage into separate local fields. Never add provider-specific counters together without a documented mapping.
  • Budget every failover as though the target cache is cold unless the target response confirms otherwise.
  • Keep provider state pointers, such as previous_response_id, scoped to the provider and lineage that issued them.
  • Log fingerprints and counters, not prompt text, raw cache keys, or full conversation contents.

Sources checked

The design above is based on four public references checked for this article:

  • The OpenAI Responses create reference documents input cache breakpoints, cache options, cache keys, and usage fields including cached and cache-write counts.
  • The Anthropic prompt caching guide documents automatic and explicit cache_control, five-minute and one-hour lifetimes, full-prefix caching, and separate cache-write and cache-read pricing categories.
  • The Gemini context caching guide documents implicit caching, the distinction between Interactions API and generateContent caching, minimum input thresholds, and usage.total_cached_tokens.
  • The OpenAI conversation state guide documents manual history, previous_response_id, connection-local continuation behavior, and billing of prior input tokens.

These references support the provider-specific facts. The failover state machine, normalized fields, and conservative budgeting rules are implementation guidance derived from those differences.

Contract details to verify

Start with a canonical request contract. It should identify the logical operation, prompt version, model intent, maximum input and output budget, safety policy, tool schema version, region policy, and fallback eligibility. A stable fingerprint can be computed from those fields without storing the prompt in a log. The fingerprint must remain the same across attempts so an operator can join primary and fallback records.

Next, define one adapter contract per provider. The adapter should declare which cache controls it can send, the accepted TTL values, the minimum input or prefix conditions, and the response fields that prove a hit, write, or miss. It should also declare whether conversation state is durable, connection-local, or unavailable to a new route. For Anthropic, verify whether the selected platform supports automatic caching; its documentation notes a legacy Amazon Bedrock integration where a top-level automatic cache control can return a client error, requiring explicit breakpoints instead.

Then define a usage normalizer. Keep provider counters in a raw subrecord and map them into fields such as input_tokens, output_tokens, cache_read_tokens, cache_write_tokens, and total_tokens. A missing field is not zero until the adapter documents that interpretation. On a failed or partial attempt, set usage_state to unavailable and leave the corresponding counters unknown; use zero only for a confirmed zero. For Gemini, map usage.total_cached_tokens to the read side of the local ledger, while retaining the raw provider field for audit. For Anthropic, preserve the provider usage object because cache reads and writes have distinct pricing treatment. For OpenAI, retain both cached and cache-write details when present.

Finally, verify the budget contract. A cache hit can lower processing cost, but the fallback decision must remain valid when the hit is absent. Set a reserve for the uncached input and define whether a cache write is allowed during brownout or high-load conditions. Record the estimate used at routing time and the final usage observed later; this distinguishes a sound decision from an unexpectedly expensive result.

A provider-neutral adapter record can look like this:

logical_request: h7
prompt_version: p3
provider_attempt: fallback
cache_hint_sent: none
cache_state: cold_assumed
input_budget: 12000
output_budget: 2000
usage_source: response
raw_usage_retained: true
state_pointer_forwarded: false

Do not forward a primary cache key merely because its text fits the fallback request. A key can have meaning only inside the provider, account, model, region, or connection that created it. The gateway should carry its own fingerprint and let the target issue a new provider-local hint.

Failure modes

  1. Blindly copying a cache key. The fallback accepts the request but ignores the key, or rejects an otherwise valid request because the control is unknown. The fix is an adapter allowlist and a default of no cache hint on provider change.

  2. Treating an expired TTL as a correctness failure. A cold cache increases work but should not change whether the canonical request is admissible. Separate cache state from request validity and alert on cost or latency impact instead.

  3. Counting unknown usage as a confirmed hit. A stream or connection can fail before final usage arrives. Set usage_state: unavailable, leave unobserved counters unknown rather than zero, reserve uncached budget, and reconcile later if the provider exposes a retrievable response.

  4. Adding incompatible counters. One provider may report cumulative cache reads, another may report a single total cached count, and another may omit cache details. Preserve raw fields and map only documented equivalents.

  5. Forwarding a conversation state pointer across providers. A previous_response_id or similar identifier may refer to provider-owned lineage. Reconstruct the canonical history for the target instead of sending an opaque pointer it cannot resolve.

  6. Ignoring cache-write pricing. A first fallback request may write a large prefix and cost more than a later hit. Include write behavior in the admission estimate and make the policy visible to on-call staff.

  7. Logging sensitive context while debugging misses. Prompt text and raw cache identifiers make incident logs harder to protect. Use a fingerprint, prompt version, breakpoint position, counters, route, and error class; retrieve content only through an approved, access-controlled workflow.

FAQ

Can a cache hit on the primary be reused by CometAPI?

Assume no. Reuse is safe only when the target adapter and provider documentation explicitly define the same cache namespace and request semantics. In the normal cross-provider case, send the canonical request and let CometAPI establish its own optional hint.

Should a fallback always disable caching?

No. A fallback may request its own cache optimization when the target supports it and the policy permits the write. The important rule is that the request must remain correct and within budget when that optimization is absent. Record the hint as an attempt-level choice rather than a property of the logical request.

What does an unknown cache state mean after a timeout?

It means the gateway cannot prove whether the provider read, wrote, or discarded a cache entry. It does not prove that the model did no work. Set usage state to unavailable, route using the uncached budget, avoid duplicate state assumptions, and reconcile any later usage evidence with the same request fingerprint.

How should a multi-turn request be represented?

Keep a provider-neutral transcript or item list with explicit prompt and tool schema versions. An adapter can then render that representation into OpenAI, Anthropic, Gemini, or CometAPI formats. Do not rely on a provider response ID as the only copy of conversation state.

Is a cache miss an incident?

Usually it is a cost or latency signal, not a correctness incident. Escalate when miss rates violate a stated service objective, when cache writes exhaust a budget, or when an adapter silently changes the canonical prompt while trying to recover.

What should operators inspect first?

Check the logical request fingerprint, attempt number, provider and model alias, cache state, normalized usage, fallback reason, and budget estimate. Those fields tell you whether the route made a conservative decision without exposing the prompt.

Reader next step

Implement the ledger and adapter contract for one non-production route. Inject a primary timeout after the request is accepted, then verify that the fallback sends the full canonical input, does not forward a primary state pointer, assumes a cold target cache, and records cache_state: unknown plus usage_state: unavailable with unknown counters for the failed attempt. Repeat with a confirmed successful response and compare the raw provider usage to the normalized record.

For the surrounding cache-boundary design, review the CometAPI fallback response cache-isolation guide and the LLM API response contract evidence guide . Then make the cold-cache budget and unknown-state behavior explicit in your gateway runbook.