Last reviewed: 2026-08-10

Direct answer

Reliable LLM batch job recovery starts by treating each input record, not the provider job, as the unit of truth. Give every record a stable application-owned ID, preserve an immutable input manifest, and maintain a durable ledger of accepted successes, permanent failures, retryable failures, and unresolved records. After a job fails, is canceled, expires, or must move to another provider, build the recovery batch only from records that remain eligible for another attempt.

This distinction matters because a batch is not necessarily atomic. OpenAI’s Batch API guide requires a unique custom_id, separates successful and failed results, and warns that output order may differ from input order. Anthropic’s batch processing guide likewise uses custom_id and exposes per-request succeeded, errored, canceled, and expired outcomes. Amazon Bedrock’s input guide uses recordId and also says output order is not guaranteed. These contracts make positional reconciliation unsafe.

Google Cloud’s resume guide demonstrates the desired recovery behavior directly: a new job processes incomplete or failed requests and merges those results with work that already completed. Even when a provider offers native resumption, keep your own ledger so the same rule works across providers.

Use this recovery sequence:

  1. Freeze the original manifest and calculate a digest for each normalized input.
  2. Assign each record a stable ID that remains unchanged across every provider and attempt.
  3. Ingest result and error artifacts by ID, never by line number.
  4. Accept a success only when its record ID, input digest, and contract version match the ledger.
  5. Classify all remaining records as permanent failure, retryable failure, canceled, expired, or unresolved.
  6. Create a recovery manifest containing only retryable and unresolved records.
  7. Submit that manifest to a compatible provider and store the new provider job-to-record mapping.
  8. Merge results conditionally so a late response cannot overwrite an already accepted terminal result.

Happy-path operator workflow

Before submission, the operator writes the canonical manifest and verifies that IDs are unique. The gateway creates the provider batch, records the provider job ID, and marks each included record submitted without changing its application ID. As results arrive, the reconciler validates the record ID and contract, stores the result artifact, and atomically changes the ledger state to succeeded or a classified error state. When the provider job ends, the operator confirms that every manifest record has one terminal ledger state and that accepted-result counts match the reconciled artifact counts. The completed result set can then move downstream.

Error-path operator workflow

When a job fails, expires, or is canceled, first stop automatic fan-out for that workflow. Snapshot the ledger, then retrieve every available result and error artifact from the original provider. Do not infer record outcomes from the top-level job status. Reconcile all returned records by stable ID, preserve accepted successes, and classify errors according to a documented retry policy. Next, verify the fallback provider’s request and response contract, build a manifest containing only eligible records, and submit the recovery job. During recovery, use a conditional write that accepts a result only if the record is still unresolved. Quarantine conflicting terminal results for review, and stop further submissions when the workflow’s attempt or spend budget is exhausted.

Who this is for

This pattern is for platform engineers, AI infrastructure teams, data-pipeline owners, and SREs running asynchronous LLM workloads. It is especially useful when batches are large enough to produce partial results, when provider retention windows make delayed recovery risky, or when an incident may require moving unfinished work to another API.

It assumes the application can persist a small amount of control-plane state independently of the provider. If a pipeline stores only the provider job ID and the original input file, it lacks enough information to prove which records completed safely.

Key takeaways

  • A failed batch job does not prove that every record failed.
  • Stable record IDs must survive retries, restarts, and provider changes.
  • Output position is not a reliable join key.
  • Accepted successes should be terminal unless a deliberate reprocessing policy says otherwise.
  • Recovery manifests should contain unresolved work, not a copy of the original batch.
  • Provider-native resume features are useful, but an application-owned ledger makes recovery portable.
  • A provider switch is safe only after request, response, safety, and data-handling contracts have been checked.
  • Logging should capture state transitions and identifiers without storing raw prompts, outputs, or sensitive headers.

Sources checked

Contract details to verify

The recovery ledger should be stricter than any single provider contract. At minimum, define these fields and rules:

  • Workflow identity: A stable ID for the business batch or dataset version.
  • Record identity: A unique, application-owned ID reused across every attempt.
  • Immutable input: A digest of the normalized request plus a versioned pointer to the retained input artifact.
  • Contract version: The prompt template, expected response shape, safety policy, and model capability profile used for acceptance.
  • Attempt ownership: The provider, provider job ID, attempt number, submission time, and recovery reason.
  • Terminal-state rule: Which success and failure states are final, and which errors can be retried.
  • Merge rule: A compare-and-set condition that prevents concurrent jobs from accepting two different results for one record.
  • Retention rule: How long manifests, mappings, results, and evidence remain available independently of provider-side retention.

A minimal canonical record can look like this:

{
  "workflow_id": "wf-042",
  "record_id": "item-0042",
  "input_digest": "[HASH]",
  "contract_version": "summary-v3",
  "state": "unresolved",
  "accepted_attempt": null
}

Map the canonical ID into each provider’s correlation field. For example, OpenAI and Anthropic use custom_id, while Amazon Bedrock uses recordId. If another provider does not echo a client-defined ID, persist an external mapping from its request identifier to the canonical record ID. Never fall back to array position.

Before cross-provider recovery, make the request body replayable and review the response-shape contract . Verify model capabilities, input limits, output schema, tool behavior, safety controls, regional handling, and error semantics. A syntactically accepted fallback request is not enough if its output cannot satisfy the original acceptance contract.

Emit a sanitized event for each meaningful state transition. Useful fields include workflow ID, record ID, attempt number, provider name, redacted provider identifiers, input digest, contract version, old and new states, normalized error class, HTTP status when available, timestamps, and usage counters. A representative event is:

{
  "workflow_id": "wf-042",
  "record_id": "item-0042",
  "attempt": 2,
  "provider": "fallback",
  "provider_job_id": "[REDACTED]",
  "provider_request_id": "[REDACTED]",
  "input_digest": "[HASH]",
  "contract_version": "summary-v3",
  "state_before": "unresolved",
  "state_after": "succeeded",
  "error_class": null,
  "http_status": 200,
  "started_at": "2026-08-10T00:00:00Z",
  "finished_at": "2026-08-10T00:00:08Z"
}

Do not place raw prompts, full model outputs, personal data, sensitive headers, or credentials in routine reliability logs. Keep larger artifacts in access-controlled storage and reference them through non-sensitive internal identifiers.

Failure modes

  • Whole-batch replay: Resubmitting the original input after a partial failure repeats records that already succeeded, increasing cost and creating conflicting results.
  • Position-based joins: A reconciler pairs the third output with the third input even though the provider returned records out of order. The wrong response is silently attached to the wrong record.
  • Top-level status inference: An operator treats failed, canceled, or expired as proof that nothing completed and discards usable partial results.
  • Mutable manifests: Inputs or prompt templates change between attempts while the record ID stays the same. The ledger then merges results produced from different contracts.
  • Late-result races: The original job returns after fallback has started. Without conditional acceptance, the later write overwrites a valid result or creates two downstream actions.
  • Provider contract drift: The fallback accepts the request but omits a required field, uses different stop semantics, or cannot honor the original capability requirement.
  • Error-class collapse: Validation errors, capacity errors, timeouts, and missing results are all labeled failed and retried identically. Permanent faults loop while transient faults may be abandoned.
  • Retry amplification: Multiple workers observe the same unresolved state and each creates a recovery batch. Use workflow-level ownership, attempt caps, and idempotent submission records.
  • Artifact expiry: Recovery begins after provider-side results or error files are no longer available. Copy required evidence into controlled storage promptly.
  • Incomplete audit evidence: Logs show a provider job transition but cannot answer which records were submitted, accepted, retried, or suppressed.

For ambiguous outcomes, apply the same discipline described in classifying partial successes before retrying : gather record-level evidence before authorizing another attempt.

FAQ

Can I resubmit the entire original file if the batch failed?

Only when you can prove that no record produced an accepted result. In the usual partial-failure case, reconcile available outputs first and build a smaller recovery manifest from unresolved or explicitly retryable records.

Does a canceled or expired job mean no work completed?

No. The refetched provider documentation describes per-request outcomes and partial results. Treat the provider job status as a signal to begin reconciliation, not as the final state of every record.

What if the fallback provider uses a different identifier field?

Keep the canonical record ID in your ledger and map it to the provider field or provider-assigned request ID. The field name can change; the application identity must not.

What if two providers return different successful answers?

Identity alone does not decide which answer is acceptable. Apply the versioned response contract and conditional merge rule. If one result is already accepted, quarantine a conflicting late result rather than silently replacing it.

Should every provider error be retried elsewhere?

No. Retry only errors that policy marks as eligible. An invalid request may remain invalid with another provider, while a capacity or availability failure may justify failover after compatibility checks.

How should operators limit repeated recovery attempts?

Set a workflow-level attempt and spend budget, record each submission durably, and require ownership before creating another job. The principles in retry budget evidence for LLM API calls help make that limit reviewable.

Reader next step

Run a controlled recovery drill before relying on this design in production. Create a small synthetic manifest with stable record IDs and known expected outcomes. Start a batch, interrupt or cancel it after some records complete, retrieve all available artifacts, and build the unresolved set from the ledger. Submit only that set to the same provider or a contract-compatible fallback.

The drill passes when every original record has exactly one accepted terminal outcome, every result maps back to the same record ID and input digest, no accepted record appears in the recovery manifest, and the event log explains each transition without exposing request content. Turn those assertions into automated checks, then attach the recovery procedure and evidence locations to the operator runbook.