Timeout-budget fallback checks for chat completions

Source pack

This refresh uses the following evidence set:

1. Existing in-place refresh target for this CometAPI fallback runbook — used to preserve the existing URL and avoid creating a new page.
2. AWS retry with backoff cloud design pattern — used for retry/backoff framing and cautions around transient failures.
3. Google SRE guidance on handling overload — used for overload, load-shedding, and retry-amplification considerations.
4. CometAPI API documentation — used as the contract source to verify endpoint paths, authentication, request fields, response fields, errors, limits, and billing-related behavior before implementation.

Related internal references:

• LLM API reliability home
• LLM API reliability posts

Intent brief

Operators need a way to test fallback behavior for CometAPI chat completion calls without turning retries into hidden latency, duplicate work, or overload. The job is not merely “retry on failure.” The job is to preserve an end-user deadline, decide when the primary attempt has consumed enough of the budget, and prove that the fallback path only runs when it is still useful.

This draft is for production-minded engineers who already have a chat-completion integration or are preparing one. It avoids undocumented endpoint paths, model IDs, auth schemes, prices, and rate-limit values. Those must be verified directly from the CometAPI API documentation before rollout.

Timeout-budget fallback checks for chat completions

Last reviewed: 2026-06-05

Who this is for: engineers operating CometAPI-backed chat completion workflows who need to validate primary-attempt timeouts, retry limits, and fallback behavior before exposing the path to users.

Key takeaways

• Treat fallback as a deadline-management problem, not a generic retry loop.
• Keep one end-to-end request deadline and allocate portions of it to connect time, primary response time, optional retry backoff, fallback attempt time, and response cleanup.
• Use retries only for failure classes that are safe and likely to be transient; the AWS retry with backoff pattern frames retry/backoff as appropriate for transient failures, not every error.
• Protect the upstream and your own service from retry storms. Google’s SRE material on handling overload is a useful reminder that client behavior can worsen overload when retries are uncontrolled.
• Verify all CometAPI contract details from the CometAPI API documentation before encoding paths, headers, model IDs, response fields, rate-limit assumptions, or billing expectations.
• Log the fallback decision, not just the final success or failure. You need to know whether the request succeeded because the primary was healthy, because the fallback was used, or because the fallback masked a slow primary.

Concise definition

A timeout-budget fallback check is a validation procedure that confirms a chat completion request can move from a primary attempt to a fallback path only when three conditions are true:

1. the primary attempt has failed, timed out, or become too slow for the remaining deadline;
2. the remaining deadline is large enough for a useful fallback attempt;
3. the fallback attempt will not violate retry, billing, quota, or overload controls that you verified from the API contract and your own policy.

In practice, this means your application should not run separate, uncoordinated timeouts for primary and fallback calls. It should run one request budget and spend it deliberately.

The operator problem

A chat completion call can fail in several ways: connection timeout, read timeout, upstream 5xx, quota or rate-limit response, malformed request, invalid auth, client cancellation, or a response that arrives too late to be useful. The wrong fallback design treats these cases equally:

“If the primary fails, call another model or provider.”

That can cause avoidable damage:

• a fallback starts after the user deadline has already expired;
• retries multiply traffic during an upstream incident;
• duplicate attempts create unexpected cost or quota use;
• the fallback hides a broken primary path until latency or spend trends become obvious;
• the application retries non-transient errors such as invalid request shape or authentication failures.

A safer design starts with the deadline and then asks: how much time is left, what failure class did we observe, and is fallback still allowed?

Runbook: validate the timeout budget before enabling fallback

1. Define one end-to-end deadline

Start from the user-facing timeout for the whole operation. Do not begin with the provider timeout.

Example structure to tune for your application:

The exact values depend on your traffic, latency SLOs, model behavior, and user experience. Treat any numeric threshold as a local control to tune, not a universal recommendation.

2. Verify the CometAPI contract before coding

Before a fallback test reaches production, verify the exact base URL, chat completion path, authentication header, request fields, response fields, error format, rate-limit indicators, and billing semantics from the CometAPI API documentation.

Do not copy these values from a runbook, issue comment, or old implementation. This article intentionally uses placeholders in examples because the provided source pack does not quote exact endpoint paths, auth schemes, model IDs, prices, or rate-limit values.

3. Classify failures before deciding fallback

A practical fallback gate can use this decision table:

This table is an operating policy template. The exact HTTP status codes, error fields, and retry headers must be verified from the CometAPI contract.

4. Cap retry and fallback amplification

AWS describes retry with backoff as a pattern for transient failures, commonly using progressive delay between attempts. The operational trap is forgetting that each retry is extra load. Google SRE’s overload guidance emphasizes that systems must shed or control work under overload rather than blindly adding more work.

For chat completions, that means:

• set a maximum number of attempts per user request;
• keep retries inside the original deadline;
• apply jitter or randomized delay when your policy uses backoff;
• do not retry invalid request or auth failures;
• stop retrying when the caller has canceled;
• record each attempt so duplicate upstream calls are visible;
• ensure fallback traffic is part of capacity planning, not an invisible side effect.

5. Make the fallback decision observable

At minimum, emit structured fields similar to these:

Avoid logging raw prompts, secrets, API keys, or full response payloads unless your security and privacy review explicitly approves it.

Contract details to verify

The following table separates implementation assumptions from values that must be verified against the primary source.

Sanitized timeout-budget probe

The following example is intentionally contract-neutral. Replace placeholders only after checking the CometAPI documentation and your internal configuration source.

#!/usr/bin/env bash
set -euo pipefail

: "${COMETAPI_BASE_URL:?set from CometAPI docs}"
: "${COMETAPI_CHAT_PATH:?set from CometAPI docs}"
: "${COMETAPI_API_SECRET:?set from your secret manager}"
: "${VALIDATED_MODEL_ID:?set from approved config}"

REQUEST_ID="fallback-probe-$(date +%s)"
TOTAL_DEADLINE_SECONDS="<TOTAL_DEADLINE_SECONDS_TO_TUNE>"
PRIMARY_MAX_TIME_SECONDS="<PRIMARY_TIMEOUT_SECONDS_TO_TUNE>"
FALLBACK_MAX_TIME_SECONDS="<FALLBACK_TIMEOUT_SECONDS_TO_TUNE>"

payload_primary='{
  "model": "<VALIDATED_MODEL_ID>",
  "messages": [
    {
      "role": "user",
      "content": "Return a short health-check sentence."
    }
  ]
}'

echo "request_id=${REQUEST_ID} attempt_role=primary max_time=${PRIMARY_MAX_TIME_SECONDS}"

primary_status="$(
  curl \
    --silent \
    --show-error \
    --output /tmp/cometapi-primary-response.json \
    --write-out "%{http_code}" \
    --max-time "${PRIMARY_MAX_TIME_SECONDS}" \
    --connect-timeout "<CONNECT_TIMEOUT_SECONDS_TO_TUNE>" \
    --request POST \
    --url "${COMETAPI_BASE_URL}${COMETAPI_CHAT_PATH}" \
    --header "<AUTH_HEADER_FROM_DOCS>: ${COMETAPI_API_SECRET}" \
    --header "Content-Type: application/json" \
    --data "${payload_primary}" \
  || echo "curl_error"
)"

echo "request_id=${REQUEST_ID} attempt_role=primary status=${primary_status}"

case "${primary_status}" in
  200)
    echo "request_id=${REQUEST_ID} final_outcome=primary_success"
    ;;
  408|409|425|429|500|502|503|504|curl_error)
    echo "request_id=${REQUEST_ID} fallback_reason=primary_transient_or_timeout"

    payload_fallback='{
      "model": "<VALIDATED_FALLBACK_MODEL_ID>",
      "messages": [
        {
          "role": "user",
          "content": "Return a short health-check sentence."
        }
      ]
    }'

    fallback_status="$(
      curl \
        --silent \
        --show-error \
        --output /tmp/cometapi-fallback-response.json \
        --write-out "%{http_code}" \
        --max-time "${FALLBACK_MAX_TIME_SECONDS}" \
        --connect-timeout "<CONNECT_TIMEOUT_SECONDS_TO_TUNE>" \
        --request POST \
        --url "${COMETAPI_BASE_URL}${COMETAPI_CHAT_PATH}" \
        --header "<AUTH_HEADER_FROM_DOCS>: ${COMETAPI_API_SECRET}" \
        --header "Content-Type: application/json" \
        --data "${payload_fallback}" \
      || echo "curl_error"
    )"

    echo "request_id=${REQUEST_ID} attempt_role=fallback status=${fallback_status}"
    ;;
  *)
    echo "request_id=${REQUEST_ID} final_outcome=fail_fast status=${primary_status}"
    ;;
esac

Before using this in a real test, adjust it in three ways:

1. Replace placeholder endpoint, path, header, and model values with values verified from the CometAPI docs.
2. Add your application’s actual remaining-deadline calculation instead of using static shell variables.
3. Replace the broad status-code case statement with the error policy you verified from the API contract.

Practical validation steps

Step 1: Run a contract-only smoke check

Use a non-production prompt and a validated model ID. Confirm that:

• the configured base URL and chat path match the documentation;
• the auth header is accepted;
• the request schema is valid;
• the parser can extract the fields your application needs;
• the logs contain no secrets or raw sensitive prompt content.

This is not yet a fallback test. It only proves the primary contract is wired correctly.

Step 2: Force the primary path to exceed its local budget

In a staging environment, force the primary attempt to time out locally. Prefer a local client timeout or test double over causing real upstream harm.

Expected result:

• primary attempt records a timeout or controlled client-side cancellation;
• fallback runs only if the remaining deadline is above your configured minimum;
• final outcome says fallback_success, fallback_failed, or deadline_exhausted;
• there is no unbounded retry loop.

Step 3: Verify fail-fast classes

Send controlled requests that represent non-retryable classes, such as malformed request shape in a safe test environment or intentionally missing non-secret configuration.

Expected result:

• invalid request classes do not trigger fallback;
• auth or permission failures do not trigger fallback;
• caller cancellation stops all downstream work;
• logs identify the fail-fast reason.

Step 4: Validate overload behavior

Run a small, controlled concurrency test in staging. The goal is not to benchmark CometAPI. The goal is to prove your own client does not amplify load.

Check that:

• maximum attempts per user request remains bounded;
• fallback volume is visible in metrics;
• backoff is capped and jittered if you use it;
• circuit breakers or concurrency limits prevent a slow primary from consuming all workers;
• your service returns a controlled failure when both primary and fallback budgets are exhausted.

This aligns with the general overload concern described in Google’s SRE guidance: when a system is stressed, uncontrolled additional work can make recovery harder.

Step 5: Reconcile attempts with usage reporting

If your organization tracks token usage, spend, or quotas, reconcile your observed attempt count with the usage data available through your approved reporting path. Do not assume a failed or timed-out attempt is non-billable unless the contract or your account data supports that conclusion.

Release guardrails

Use these guardrails before enabling fallback for production traffic:

• Feature flag: enable fallback by route, tenant, or traffic percentage.
• Attempt cap: enforce a hard maximum number of upstream attempts per user request.
• Deadline cap: cancel all downstream attempts when the original caller deadline expires.
• Fallback floor: skip fallback when the remaining time is below your minimum useful threshold.
• Error allowlist: retry or fallback only for error classes you have explicitly approved.
• Circuit breaker: stop sending traffic to a path that is failing consistently.
• Usage review: check request counts, token usage, quota, and billing signals after the test window.
• Rollback path: keep a one-step way to disable fallback without redeploying code.

If you are evaluating whether CometAPI fits your application’s routing and reliability requirements, you can start with CometAPI and verify the current API contract before building production fallback logic.

FAQ

Should every timeout trigger fallback?

No. Fallback should run only when the remaining deadline is large enough for a useful attempt and the failure class is approved for fallback. If the user deadline is already gone, fallback only creates extra work.

Should a 429 response be retried?

Only if your documented contract and local policy support it. Verify the rate-limit behavior, headers, and any retry-after semantics from the CometAPI documentation. Without that verification, treat 429 handling as a policy decision that needs review.

Is exponential backoff always required?

No. Backoff is useful for transient failures, and the AWS retry/backoff pattern explains that retry delays can reduce repeated immediate failures. But backoff must fit inside your user deadline. For short interactive chat requests, a long backoff may be worse than a controlled failure.

Can fallback hide primary-path incidents?

Yes. That is why attempt-level metrics matter. If you only measure final success, fallback can make the product appear healthy while primary latency, error rate, or quota problems grow.

Should fallback use the same model?

That depends on your product requirements and what CometAPI currently supports for your account. Verify model identifiers and compatibility from the CometAPI docs or your approved configuration source. Do not hard-code model IDs from examples.

What is the safest first production rollout?

Start with a feature flag, a small traffic percentage, strict attempt caps, and dashboards that split primary success, fallback success, fallback failure, and deadline exhaustion. Review usage and error data before expanding.

Sources checked

Access date: 2026-06-04