Timeouts
Timeout budgets, retries, and safe fallback boundaries for LLM API calls.
Reliability Runbook
LLM API Reliability Notes
Practical guides for LLM API reliability and fallback engineering.
Route health
monitored
- DetectTimeout or 5xx spike
- RouteFallback policy check
- VerifyResponse contract smoke test
- EscalateHuman review when evidence is thin
116
Published and archived Reliability notes
SLO
Primary lens
Daily
Review cadence
Failure Map
Reliability notes organized by failure mode
Use these notes to decide what to retry, route, stop, or escalate before production traffic is exposed.
Rate limits
How to identify quota behavior without hiding customer-impacting failures.
Fallback routes
Decision rules for switching providers, models, or degraded modes.
Incident checks
Smoke tests and evidence requirements before calling a route production-ready.
Latest Runbook Notes
Recent reliability guides
Historical archive entries remain available to readers while staying out of RSS, sitemap, and llms.txt.
Parse CometAPI Streams Without Corrupting Partial Events
Build a CometAPI SSE stream parser that preserves event boundaries, validates JSON fragments, and rejects truncated output safely.
Classify provider errors into retry, fallback, and fail-fast actions
A practical error taxonomy for deciding when an LLM gateway should retry, fail over, or stop immediately.
Prevent Split-Brain Routing During CometAPI Fallback Policy Rollouts
Roll out versioned CometAPI fallback policies across gateway replicas without mixed routing decisions, stale configuration, or unsafe promotion.
Stop LLM API Queues From Becoming Outages With Adaptive Concurrency Limits
Use latency feedback, bounded queues, and early load shedding to keep changing LLM workloads from exhausting gateway and upstream capacity.
Recover in-flight LLM requests after HTTP/2 GOAWAY or 421 responses without replaying calls that may already have been processed.
A protocol-aware runbook for recovering LLM requests after HTTP/2 GOAWAY or 421 responses without causing duplicate execution.
Catch LLM API Reliability Trouble With Error-Budget Burn Alerts
Design LLM API burn-rate alerts that separate urgent incidents from slow reliability erosion and guide safer operator action.