Last reviewed: 2026-08-01

Direct answer

Build a CometAPI fallback response cache identity only after the gateway has selected the actual route and produced the exact outbound request. Partition entries by tenant scope, API operation, primary or fallback role, resolved model ID, request schema, prompt revision, policy revision, tool schema, output mode, streaming mode, locale, canonical request body, and an invalidation epoch. A response written to a fallback namespace must never satisfy a primary-route lookup.

The cache should fail closed. If a compatibility dimension is missing or uncertain, bypass it. Store only complete, validated responses that your application has explicitly classified as reusable. Do not cache transient failures, interrupted streams, or responses that could repeat a side effect. A short TTL limits the duration of a mistake, but it cannot repair a key that merges incompatible requests.

This design belongs in the application or gateway layer in front of CometAPI. The documented CometAPI surface provides an OpenAI-compatible chat endpoint and many model choices, but that does not make two routes or model contracts interchangeable.

Who this is for

This guide is for platform engineers, SREs, and application owners who operate a shared response cache in front of primary and CometAPI fallback routes. It is especially relevant when several tenants, prompt versions, model aliases, structured-output modes, or tool definitions pass through one gateway.

It addresses completed-response caching, where an earlier model response can satisfy a later request. Provider-side prompt caching is a different mechanism: it may reduce repeated input processing while still producing a new response. Keep those two mechanisms separate in diagrams, configuration, metrics, and incident reviews.

Key takeaways

  • Make the routing decision before looking up a response. A primary decision may read only the primary namespace; a fallback decision may read only the selected fallback namespace.
  • Include the resolved model and every response-affecting contract revision, not just a logical alias or normalized prompt.
  • Scope personalized entries to the correct tenant or user boundary. Permit global reuse only for content deliberately classified as public and shareable.
  • Compute the key from structured, canonical data. Do not normalize away message order, tool definitions, stop conditions, output requirements, or meaningful whitespace.
  • Cache only complete, successful, schema-valid responses. Treat errors, partial streams, and tool calls as bypasses unless a narrower contract proves reuse is safe.
  • Version the key format and include an invalidation epoch. This lets operators stop reading old entries without relying on a slow or incomplete bulk deletion.
  • Log safe contract metadata and short digest prefixes, never raw prompts, response text, tenant identifiers, or transport credentials.

Sources checked

The CometAPI documentation describes an OpenAI-compatible chat-completions interface, a unified endpoint, and a catalog spanning many models. It establishes why a gateway may serve multiple model contracts through one integration. It does not establish that CometAPI implements the application response cache described here.

The Cloudflare AI Gateway caching documentation provides a concrete gateway example: its default identity includes provider, endpoint, model, and the full request body, while body differences such as messages, tools, or model parameters create separate entries. It also documents per-request bypass, TTL, custom-key behavior, cache-status signals, and the possibility that simultaneous misses both reach the origin. Those details are specific to that product, but they demonstrate the dimensions an LLM cache must consider.

The LiteLLM caching documentation documents completed-response reuse, cache namespaces, configurable TTLs, and multiple exact and semantic cache backends. Its configuration examples reinforce that cache behavior depends on the deployed gateway and backend rather than on the model API alone.

RFC 9111 on HTTP caching defines a cache key as the information used to choose a stored response. It explains freshness, validation, invalidation, shared versus private caches, and adding identity dimensions to reduce inappropriate reuse and privacy risk. LLM POST-response caches need application-specific logic beyond ordinary HTTP caching, but the governing principle remains useful: reuse must preserve request semantics.

Contract details to verify

Start with a written compatibility rule: a cache hit is permitted only when the new request could safely receive the stored response without knowing that a previous request produced it. Convert that rule into structured key fields and store-eligibility checks.

Build a versioned cache contract

The following is a sanitized operator record, not the full cache key. Production lookup uses complete internal values and full digests; logs expose only short prefixes.

cache_key_version: 3
namespace: fallback-v3
tenant_scope_prefix: t_7a91
api_operation: chat-completions
route_role: fallback
gateway: cometapi
resolved_model_id: model-a
request_schema_revision: chat-v4
prompt_revision: support-v12
policy_revision: safety-v6
tool_schema_digest_prefix: 4c19d8a2
response_mode: json
stream: false
locale: en-US
request_body_digest_prefix: 8f31c2ab
invalidation_epoch: 7

The actual identity should include a pseudonymous tenant scope, unless an explicit policy allows cross-tenant reuse. Include the API operation because chat, image, audio, and other operations do not share a response contract. Include both route role and resolved model ID so a fallback result cannot masquerade as a primary result, even when both routes began with the same logical alias.

Hash the exact canonical outbound body after routing and transformation. Canonicalization may stabilize object-key ordering and encoding, but it must preserve message order, tool descriptions, sampling parameters, output schemas, and any other response-affecting value. Include configuration revisions that are not present in the body, such as the prompt template, safety policy, alias map, or post-processing schema.

Review alias behavior separately with the model alias drift checks . If an alias can resolve to a different model without changing the request body, the alias alone is not a sufficient boundary.

Decide lookup and write timing

The gateway should first classify cache eligibility and select the current route. It can then calculate a key for that route and perform a lookup. If a primary miss leads to a live primary call that fails, the gateway must make a new fallback decision, transform the request for the selected fallback, and calculate a new fallback key. It must not reuse the failed primary key.

On write, record the route that actually produced the response. Validate the terminal status, response shape, output mode, and completion state before storage. The response-shape promotion checklist provides a related boundary: a syntactically successful response is not reusable if it violates the consumer contract.

Limit what can be stored

Use a deny-by-default store policy. A response becomes eligible only when all required conditions are true:

  1. The response is complete and belongs to a successful outcome class approved by the application.
  2. The consumer-facing response shape passes validation.
  3. The request contains no unscoped personal or tenant-specific material.
  4. Reuse cannot repeat a tool call, purchase, message send, write, or other side effect.
  5. The response is not an interrupted stream or an assembled stream with missing terminal evidence.
  6. The content has a defined maximum acceptable staleness and an enforced TTL.
  7. The cache record carries its key version, route role, model identity, policy revision, creation time, expiry time, and invalidation epoch.

Happy-path operator workflow

Consider a cacheable, tenant-scoped support request. The current routing policy selects the primary route.

  1. Classify the request as eligible and derive its pseudonymous tenant scope.
  2. Resolve the primary model, build the exact outbound body, and calculate the primary key.
  3. On a primary cache miss, make the live primary call. Suppose it returns a retryable overload response and is not stored.
  4. Bind the fallback decision to the current user action , select the CometAPI fallback route, and apply its request transformation.
  5. Calculate a new key in the fallback namespace using the resolved fallback model and transformed body.
  6. On a fallback miss, call CometAPI. Validate the completed response against the expected shape and policy.
  7. Store the response only in the tenant-scoped fallback namespace with the approved TTL and current invalidation epoch.
  8. A later equivalent request can hit that entry only if routing again selects the same fallback contract. When routing returns to primary, primary lookups cannot see the fallback entry.

This sequence preserves the distinction between a routing decision and a cache hit. The existence of a fallback entry must never cause the gateway to choose fallback by itself.

Error-path operator workflow

Suppose monitoring reports a cache hit tagged as fallback while the request trace says the selected route was primary. Treat that as a contract violation, not a harmless stale hit.

  1. Bypass reads for the affected route or advance its invalidation epoch immediately. Prefer a targeted boundary, but disable the shared response cache if the affected scope cannot be proven.
  2. Send the current request to the selected live route. If no safe live route is available, use the product’s defined degradation response rather than serving an unverified cached answer.
  3. Preserve sanitized lookup, routing, and write metadata. Do not preserve raw prompts or outputs merely to investigate the cache.
  4. Compare the key version, namespace, tenant scope, resolved model, prompt revision, policy revision, body digest, store time, and expiry time across the write and hit events.
  5. Invalidate the affected namespace, advance the key version or epoch, and correct the missing dimension or ordering bug.
  6. Replay a synthetic test matrix covering different tenants, route roles, models, policies, tools, output modes, and interrupted responses.
  7. Re-enable reads gradually only after every incompatible pair produces a miss and every eligible equivalent pair produces the intended hit.

Log enough metadata to reconstruct that sequence without exposing content:

event: llm_cache_lookup
request_id: req_42f
cache_outcome: bypass
cache_key_version: 3
cache_digest_prefix: 8f31c2ab
namespace: fallback-v3
tenant_scope_prefix: t_7a91
route_role: fallback
gateway: cometapi
resolved_model_id: model-a
api_operation: chat-completions
prompt_revision: support-v12
policy_revision: safety-v6
tool_schema_digest_prefix: 4c19d8a2
response_mode: json
stream: false
ttl_seconds: 300
bypass_reason: contract_mismatch
invalidation_epoch: 7

Also record the store decision, store-denial reason, status class, response validation result, cache age, and routing-policy revision. Keep raw request bodies, response bodies, user identifiers, tenant identifiers, complete digests, and transport credentials out of routine logs.

Failure modes

  1. The key is computed before routing. The fallback receives a transformed request, but the response is stored under the original primary identity. A later primary lookup can then return fallback output.
  2. A logical alias replaces the resolved model. The alias remains stable while its target changes, allowing responses from different model contracts to collide.
  3. Prompt, policy, or tool revisions are omitted. The visible user message is identical, but hidden instructions or callable tools have changed. The old response no longer satisfies the current contract.
  4. Tenant scope collapses to global scope. An exact prompt match crosses an isolation boundary and can disclose or infer another tenant’s content. A body digest does not make cross-tenant reuse acceptable.
  5. Errors or partial streams are stored. A timeout page, rate-limit response, truncated answer, or incomplete structured object becomes a fast and repeatable failure.
  6. Fallback entries remain visible after recovery. The primary route is healthy, but a shared namespace continues returning fallback output and hides whether primary service has actually recovered.
  7. A custom key is too coarse. A memorable label replaces the full contract fingerprint. Different messages, tools, or parameters then map to one stored answer.
  8. Semantic matching ignores hard constraints. Similar prompts with different tenants, dates, output schemas, or safety requirements are treated as equivalent. Keep semantic caches isolated from fallback response reuse until task-specific evaluation establishes safe boundaries.
  9. Simultaneous misses cause duplicate fills. Two workers call the fallback before either write becomes visible. This usually creates load and spend rather than a wrong response, but it can amplify an incident. Use bounded request coalescing only after preserving cancellation and timeout rules.
  10. Invalidation deletes records but not readers. Old workers continue constructing the previous key or reading an old namespace. Versioned keys and an epoch checked at lookup provide a clearer stop condition.

FAQ

Does CometAPI provide the response cache described here?

This article describes a cache implemented by your application, proxy, or gateway. The refetched CometAPI documentation establishes the unified API and model surface, not a guarantee about response-cache behavior. Verify every deployed cache layer independently.

Can primary and fallback routes share a cache when they use the same model name?

Do not share by default. The routes can still differ in resolved model, endpoint behavior, request transformation, policy, capacity tier, or post-processing. A shared namespace is reasonable only when those contracts are explicitly proven identical and cross-route reuse is an intentional product decision.

Is hashing the full request body enough?

No. The body may omit tenant scope, resolved routing information, prompt-template revision, alias-map revision, safety policy, or response post-processing. The body digest is one key component, not the complete compatibility decision.

Should retryable errors be cached?

Not by default. Although general HTTP caches can store some negative results under defined rules, an LLM gateway should require an explicit, short-lived negative-cache contract. Never let a cached error trigger fallback recursively or replace a valid completed response.

How should operators choose the TTL?

Start with the maximum staleness the feature can safely tolerate, then shorten it for volatile prompts, policies, aliases, and data. Cost savings are not the safety boundary. A correct TTL cannot compensate for an incomplete key, and an invalidation epoch must still be available for urgent changes.

Can semantic caching be used on fallback routes?

Only after task-specific evaluation proves that the similarity rule preserves tenant, policy, tool, and output constraints. Exact matching is easier to audit. Keep semantic entries in a separate namespace with separate metrics, rollout controls, and invalidation.

Must fallback entries be deleted when primary service recovers?

They may remain until expiry if the primary namespace cannot read them and the fallback contract is still valid. Invalidate them when the model, prompt, policy, privacy classification, or response contract changes. Recovery is a routing event; it should not silently merge cache namespaces.

Reader next step

Trace one representative request through every cache layer and write down where routing, transformation, lookup, validation, storage, and invalidation occur. Then run a synthetic matrix: the same tenant and contract should hit; a different tenant, route role, resolved model, prompt revision, policy revision, tool schema, or output mode should miss. Interrupted streams and failed responses should not create entries.

Pair that test with the site’s fallback attempt limits so a cache miss cannot become an unbounded retry chain. Record the resulting key contract and error-path procedure in the runbook before enabling shared caching.

When those boundaries are testable, Start with CometAPI and exercise both the primary and fallback namespaces with synthetic, non-sensitive requests before allowing production cache writes.