Last reviewed: 2026-07-02
Direct answer
A reliable LLM API fallback runbook needs a quality gate that checks three things before the runbook is trusted: the documented request path still matches the operator’s integration, the response evidence is enough to compare primary and fallback behavior, and retry handling avoids turning one failure into repeated pressure on the same upstream path.
Use the gate as a smoke test, not as proof of model quality, account capacity, price, latency, or provider availability. For a related reliability checklist, see Build a CometAPI Fallback Evidence Checklist .
Smoke-test workflow:
- Setup assumptions: the operator has a test environment, a scoped CometAPI credential stored outside the runbook, the current CometAPI documentation open, and a harmless test prompt that does not contain customer data.
- Happy-path request plan: send one minimal request to the documented Chat Completions path or Responses path used by the service, then record only status, selected request category, response object category, and whether the expected top-level response fields were present.
- Error-path check: repeat with an intentionally invalid placeholder credential such as
<API_KEY_PLACEHOLDER>and confirm the runbook records the failure class without retrying indefinitely. - Minimum assertions: request reaches the expected documented API family, authentication failure is distinguishable from a successful response, retry handling is bounded, and the incident note includes enough fields for escalation.
- Pass/fail logging fields:
run_id,api_family_checked,status_class,response_object_category,retry_attempts_used,fallback_decision,evidence_links_attached, andoperator_notes. - Do not assert: exact prices, exact quota, exact latency, model availability, billing impact, or long-term uptime from this smoke test.
Sanitized log-record template:
{
"run_id": "smoke-test-placeholder",
"api_family_checked": "chat-completions-or-responses",
"status_class": "2xx-or-4xx-placeholder",
"response_object_category": "documented-category-placeholder",
"retry_attempts_used": "bounded-placeholder",
"fallback_decision": "promote-hold-or-investigate",
"evidence_links_attached": ["documentation-url-placeholder"],
"operator_notes": "sanitized note without prompts, credentials, prices, limits, or full responses"
}
Who this is for
This guide is for engineers who own LLM gateway behavior, on-call runbooks, fallback routing, incident notes, or release checks for applications that can call CometAPI through documented text API families.
It is especially useful when a team already has a fallback path but needs a repeatable review before letting operators use that path during an incident.
Key takeaways
- Check the active API family first: Chat Completions and Responses have separate documentation pages and should not be treated as interchangeable without verification.
- Keep fallback smoke tests narrow: they should prove the runbook can collect evidence, not that a provider will meet future commercial or performance expectations.
- Retry logic needs limits and backoff so a transient failure does not create repeated load on the same dependency.
- Support escalation notes should include sanitized request context, observed status class, response category, retry count, and links to the exact documentation used during the check.
Failure modes
- Evidence gap: the agent cannot inspect the failing log, source page, pull request, or local command output. The safe action is to stop and record the missing evidence instead of guessing.
- Scope drift: the agent edits files that are not connected to the observed failure. Keep the repair tied to the failing signal and leave unrelated cleanup for a separate task.
- Environment mismatch: the local check uses different versions, credentials, feature flags, or runtime settings than the hosted path. Record the mismatch before treating the result as proof.
- Unreviewed fallback: the agent changes models, endpoints, permissions, or retry behavior to make a run pass without preserving the review boundary. Treat access and provider failures as operational blockers, not topic failures.
- Weak handoff: the final note says the issue is fixed but omits the command, result, changed files, and remaining uncertainty. That makes the next operator repeat the investigation.
Sources checked
- CometAPI chat completions reference - accessed 2026-07-02; purpose: verify chat completion contract areas.
- CometAPI help center - accessed 2026-07-02; purpose: verify support and escalation documentation areas.
- CometAPI documentation - accessed 2026-07-02; purpose: verify current CometAPI documentation navigation.
- CometAPI responses reference - accessed 2026-07-02; purpose: verify responses endpoint contract areas.
- AWS retry with backoff pattern - accessed 2026-07-02; purpose: verify retry and backoff guidance.
Contract details to verify
| Area | What to verify | Source URL | Accessed | Safe candidate wording |
|---|---|---|---|---|
| Chat request path | Confirm whether the service still calls the documented Chat Completions API family and records the request category used by the smoke test. | https://apidoc.cometapi.com/api/text/chat | 2026-07-02 | “The runbook should verify the documented Chat Completions path before using chat fallback evidence.” |
| Responses request path | Confirm whether the service uses the Responses API family for the tested workflow and records response evidence separately from chat evidence. | https://apidoc.cometapi.com/api/text/responses | 2026-07-02 | “Responses evidence should be checked against the Responses documentation, not inferred from chat behavior.” |
| Authentication failure handling | Confirm the runbook can distinguish a credential or authorization failure from a successful response without storing a real credential. | https://apidoc.cometapi.com/api/text/chat | 2026-07-02 | “The smoke test should use placeholder-safe logging and keep credentials outside the article or runbook.” |
| Escalation packet | Confirm that support notes include sanitized request context, observed status class, and documentation links. | https://apidoc.cometapi.com/support/help-center | 2026-07-02 | “Escalation notes should be specific enough to reproduce the issue without exposing secrets or full prompts.” |
| Retry behavior | Confirm retries are bounded and use backoff for transient failures. | https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/retry-backoff.html | 2026-07-02 | “A fallback runbook should avoid open-ended retries and should record how many retry attempts were used.” |
Reader next step
Compare the workflow against Start with CometAPI .
FAQ
Is this gate enough to approve a fallback for production traffic?
No. It is a runbook quality check. It verifies that operators can collect clean evidence and follow documented API families, but it does not prove production capacity, pricing, latency, or long-term availability.
Should Chat Completions and Responses share the same fallback checklist?
They can share the same review structure, but the evidence should point to the API family actually used by the service. Keep Chat Completions and Responses request and response checks separate in the log.
What should operators avoid putting in the smoke-test log?
Do not record real credentials, customer prompts, full generated responses, exact account limits, prices, billing assumptions, or uptime claims. Use sanitized fields that preserve the failure category and reproduction path.
When should the team update the runbook?
Update it when the integration changes API family, request shape, retry behavior, escalation requirements, or the documentation source used by the team changes.