Dispatch CometAPI Jobs After Django Transactions Commit

Last reviewed: November 5, 2026 (Asia/Shanghai).

Direct answer

Create the durable job inside the Django transaction, then publish its identifier to the worker queue through transaction.on_commit(). Start the CometAPI request in the worker after it reads and claims the committed job. A database rollback cannot undo an HTTP request or recall a queue message already sent.

The Django transaction reference says commit callbacks run after a successful transaction and are discarded on rollback. A callback registered inside a nested atomic() block waits for the outer transaction; a rollback to its savepoint discards it. This closes the race in which a worker receives an identifier before the corresponding database row commits.

There is a second boundary: committing the job and delivering the queue message are separate operations. An on_commit() callback can fail or disappear with the process after the database commits. Use a durable outbox or another durable recovery mechanism when the product promises eventual processing. The callback establishes ordering; recovery establishes responsibility for an accepted job.

Who this is for

This guide is for Django application owners sending document summaries, extraction jobs, or other asynchronous text work through CometAPI. It assumes a transactional database, a worker queue, and an existing provider adapter. Django 5.2 is the documented baseline; Celery’s convenience method requires its DjangoTask support.

The focus is submission before inference starts. The worker still needs deadlines, bounded requests, result validation, and duplicate handling. No authenticated CometAPI request or broker integration was run for this article. The reported local probe exercises Django with SQLite and a synthetic publisher.

Key takeaways

Save the job and its input identity before authorizing dispatch.
Register queue publication as a callable; do not call the publisher while constructing the callback.
Bind callbacks to the same database alias as the job transaction.
Return an application job ID even when no broker task ID exists yet.
Preserve accepted jobs after callback failure, and make duplicate queue deliveries harmless.
Test outer rollback, nested rollback, and failure after commit separately.

Contract details to verify

Separate database acceptance from model execution

The CometAPI chat-completion reference documents POST /v1/chat/completions on api.cometapi.com, bearer authentication, and a request containing model and messages. That outbound request is an external operation. A surrounding Django atomic() block controls database work on its connection; it does not make generation part of that transaction.

Keep the acceptance transaction short. Validate authorization, create an application job ID, bind it to the input revision, and save a pending job. The worker can later load that committed record, check its eligibility, claim an attempt, and call the provider outside a long-lived database transaction. Persist the validated result in a separate guarded transaction.

For a document that can change, store an immutable input snapshot or a revision that can be retrieved later. A mutable document ID alone can make a delayed worker summarize different text from the submission the user approved. Keep credentials in the worker’s secret configuration, and put only the necessary job reference on the queue.

Register the queue handoff on the correct connection

This service fragment assumes an existing GenerationJob model, configured Celery application, and run_generation task. It shows the transaction boundary rather than a complete web endpoint:

from functools import partial

from django.db import transaction

from .models import GenerationJob
from .tasks import run_generation


def accept_generation(input_ref, *, using="default"):
    with transaction.atomic(using=using):
        job = GenerationJob.objects.using(using).create(
            input_ref=input_ref,
            state="pending",
        )
        transaction.on_commit(
            partial(run_generation.delay, str(job.pk), using),
            using=using,
        )
    return str(job.pk)

Here input_ref must already identify the approved immutable input. The task signature accepts the job ID and database alias, and the worker reads the corresponding authoritative database. Map tenant routing through trusted application configuration rather than accepting arbitrary aliases from a request.

Pass the callable produced by partial(). Writing transaction.on_commit(run_generation.delay(...)) sends immediately because Python evaluates the argument first. When registering callbacks in a loop, binding each identifier also avoids late-bound lambdas all referring to the final loop value.

Using the same alias matters: a callback on default cannot wait for a transaction opened on another connection. Multiple database commits are not coordinated by registering one callback. Also, commit does not guarantee that an asynchronous read replica has replayed the row. Route the initial worker lookup to the authoritative database or provide an explicit replica-consistency mechanism.

If a caller already has an outer atomic() block, this function’s inner block creates a savepoint under normal Django behavior. Returning from the function does not mean the outer transaction committed. Let the outer owner decide when the job becomes accepted. Django also documents that on_commit() runs immediately when no transaction is open; registering a callback is not a way to create a transaction.

Use Celery’s shortcut within its limits

The Celery Django guide describes the worker race directly: a task sent immediately can run before the view’s database changes are committed. Celery 5.4 introduced delay_on_commit() through DjangoTask to wrap publication in Django’s commit hook.

For a supported task on the default connection, call run_generation.delay_on_commit(job_id, "default") inside the acceptance transaction. Custom task bases must inherit the appropriate DjangoTask behavior. For explicit database selection, the transaction.on_commit(..., using=using) form above makes the connection binding visible.

Celery documents that delay_on_commit() does not return the task ID, because the message has not yet been sent. Expose your own job ID for status checks and retain it across broker publication attempts. Do not switch to immediate delay() merely to obtain a transport identifier before commit.

Happy-path operator workflow

Validate the submission and its authorization. Resolve a stable input revision and application operation key.
In one short transaction, create the pending generation job and any durable dispatch record required by the delivery contract.
Commit. Only afterward may a callback or outbox dispatcher publish the job ID.
The worker reads the authoritative committed record and atomically claims an eligible attempt. A duplicate message finds an existing claim or terminal outcome and does not independently authorize generation.
The worker calls CometAPI outside the claim transaction, validates the completion, and saves its result under the claim’s acceptance rules.
The application serves status and results through the same job ID. Queue delivery and generation completion remain separate observations.

Record acceptance time, dispatch state, claim outcome, and generation outcome separately. A successful database commit proves neither broker acceptance nor a useful model result. Preserve the original deadline and attempt allowance when dispatch is delayed.

The site’s queue redelivery design provides a concrete ownership and result-fencing pattern. Its SQS-specific acknowledgement rules do not define Celery behavior, but the distinction between a delivery and an authorized generation attempt applies here.

Error-path operator workflow

Preserve the committed job when publication fails

Django says callbacks run after commit and are not part of the database transaction. A callback failure cannot roll the job back. With the default robust=False, an uncaught exception also prevents later callbacks for that transaction from running. robust=True lets later callbacks continue after ordinary exceptions; it does not make the failed callback retry or become durable.

Define how the request path reports that failure without implying that the job vanished. For retryable submissions, enforce a database uniqueness rule on a trusted operation key scoped to the caller. A repeated key should resolve to the original job after verifying that its input identity matches. Otherwise an HTTP retry following a callback error can create a second job while the first remains committed.

Use one of these explicit delivery contracts:

Product requirement	Suitable handoff
Recoverable convenience task	Commit callback plus a durable pending-job sweep with bounded recovery
Accepted job must survive a process crash before publication	Job and outbox entry committed together, with an independent durable dispatcher
Database and external effect must succeed as one indivisible action	An ordinary callback cannot supply that guarantee; redesign the operation’s acceptance contract

For the outbox design, insert one uniquely keyed dispatch intent beside the job in the same transaction. A dispatcher reads committed intents, publishes the stable job ID, and records its progress. A callback may wake that dispatcher for low latency, but a periodic durable scan must find work even when the wakeup is lost.

Publication can remain ambiguous if the broker accepts a message and the dispatcher crashes before recording acknowledgement. Retrying may deliver the same job again. Keep the application job identity stable and require the worker claim before model submission. A broker task ID or outbox status alone does not establish exactly one CometAPI execution.

The outbox and recovery choices here are application recommendations derived from the documented transaction boundary. They are not built-in Django or CometAPI delivery guarantees. When an upstream request may already have started, preserve an uncertain attempt and reconcile it under your replay policy before generating again.

Keep rollback and shutdown behavior observable

On outer rollback, require the job, its outbox intent if used, and its commit callback to disappear together. When a nested savepoint rolls back but the outer transaction succeeds, require only work belonging to the successful portion to dispatch. A callback attached to the wrong connection can bypass both expectations.

Stopping the web process after commit must leave a discoverable pending job or outbox entry. Stopping a worker after dispatch is a different recovery problem. The site’s worker shutdown guide explains why orderly draining cannot replace durable recovery after abrupt termination.

Use small diagnostic events with an opaque job reference, database label, transaction outcome, dispatch state, attempt reference, and bounded failure reason. Keep prompts, output, credentials, and raw broker arguments out of routine logs.

Verify the commit boundary offline

Save this as probe_commit_boundary.py in an isolated environment with Django 5.2 installed, then run python probe_commit_boundary.py. It uses an in-memory SQLite table and a synthetic publisher; it needs no application models, broker, credentials, or network calls.

from functools import partial

from django.conf import settings

settings.configure(
    DATABASES={
        "default": {
            "ENGINE": "django.db.backends.sqlite3",
            "NAME": ":memory:",
        }
    },
    INSTALLED_APPS=[],
)

import django

django.setup()

from django.db import connection, transaction

with connection.cursor() as cursor:
    cursor.execute("CREATE TABLE jobs (id TEXT PRIMARY KEY)")

sent = []


def publish(job_id):
    assert not connection.in_atomic_block
    with connection.cursor() as cursor:
        cursor.execute("SELECT id FROM jobs WHERE id = %s", [job_id])
        assert cursor.fetchone() == (job_id,)
    sent.append(job_id)


def accept(job_id):
    with transaction.atomic():
        with connection.cursor() as cursor:
            cursor.execute("INSERT INTO jobs (id) VALUES (%s)", [job_id])
        transaction.on_commit(partial(publish, job_id))


with transaction.atomic():
    accept("committed")
    assert sent == []
assert sent == ["committed"]

try:
    with transaction.atomic():
        accept("rolled-back")
        raise RuntimeError("force outer rollback")
except RuntimeError:
    pass
assert sent == ["committed"]

with transaction.atomic():
    try:
        with transaction.atomic():
            accept("savepoint-rollback")
            raise RuntimeError("force savepoint rollback")
    except RuntimeError:
        pass
    accept("kept")
assert sent == ["committed", "kept"]


def unavailable():
    raise ConnectionError("synthetic broker failure")


try:
    with transaction.atomic():
        with connection.cursor() as cursor:
            cursor.execute("INSERT INTO jobs (id) VALUES (%s)", ["broker-down"])
        transaction.on_commit(unavailable)
except ConnectionError:
    pass
else:
    raise AssertionError("callback failure did not propagate")

with connection.cursor() as cursor:
    cursor.execute("SELECT id FROM jobs ORDER BY id")
    assert cursor.fetchall() == [("broker-down",), ("committed",), ("kept",)]

print(f"Django {django.get_version()}: commit boundary checks passed")

A local run on November 5, 2026 in Asia/Shanghai with Python 3.12.3 and Django 5.2.19 printed Django 5.2.19: commit boundary checks passed. The assertions verified delayed publication until the outer commit, discarded callbacks and rows after outer or savepoint rollback, and a retained committed row after synthetic callback failure.

This verifies ordering on one local database connection. It does not test Celery, a real broker, replica visibility, outbox concurrency, process termination, or provider execution. Before rollout, exercise those boundaries in staging with a stubbed provider adapter. Include a crash after database commit and a crash after broker acceptance, then require recovery to preserve the same job ID without independently authorizing duplicate inference.

Django’s ordinary TestCase wraps tests in a transaction that is rolled back, so callbacks do not naturally execute. Its documented captureOnCommitCallbacks() helper can inspect or emulate them; use TransactionTestCase when you need actual commits. Emulating callback execution alone does not prove a separate worker can read committed data.

Failure modes

The provider call runs inside atomic(). A later rollback removes local evidence while external work may continue. Commit the accepted job before generation.

The callback is invoked during registration. Passing a function’s return value sends too early. Pass a bound callable instead.

The wrong database connection owns the hook. Dispatch proceeds without waiting for the job’s commit. Bind the alias explicitly and test it.

A callback is treated as a durable queue. A process crash loses the handoff. Keep a recoverable pending record or transactional outbox.

Publication ambiguity creates a new job. A retry changes identity and permits another generation. Retry the same dispatch intent, then let the worker’s durable claim decide eligibility.

FAQ

Does `on_commit()` run the callback in the background?

It schedules execution after commit, not a separate worker. Keep the callback limited to the handoff or wakeup. The actual generation belongs in an owned worker.

Does `delay_on_commit()` guarantee delivery?

It wraps the commit ordering described by Celery. It does not atomically join the database commit and broker publication or preserve an in-memory callback after process loss. Add durable recovery for that requirement.

Can callback failure undo the committed job?

No. Django explicitly places callbacks after commit. Report or recover the dispatch failure while retaining the job’s accepted identity.

Should a worker retry a missing row immediately?

First inspect connection binding and database routing. A missing row can expose early dispatch or replica lag. A bounded retry may belong to an explicit consistency policy, but it should not conceal an incorrect commit boundary or start generation without the job.

Reader next step

Audit one job-submission path from database insert to provider dispatch. Move its handoff behind the correct commit hook, define the durable recovery owner, and run the rollback probe. Accept the change only after staging shows that committed-but-undelivered jobs remain discoverable and duplicate messages preserve one application identity.

When that boundary is established, Start with CometAPI and use your existing authorized staging model to check the worker’s request and completion validation.

Sources checked

Three primary technical references were retrieved on November 5, 2026, in Asia/Shanghai:

Django 5.2: Database transactions — commit callbacks, savepoint rollback, connection selection, callback failure, and testing behavior.
Celery: First steps with Django — early task dispatch, DjangoTask, delay_on_commit(), and the absence of an immediate task ID.
CometAPI: Create a chat completion — the generation endpoint, bearer authentication, request structure, and completion fields.

These currently served official references support an evergreen implementation task. The Celery page identified itself as version 5.6.3 when fetched; its commit shortcut is documented as introduced in 5.4. Django’s versioned 5.2 reference matches the local probe’s major and minor version. This article makes no claim of a new release or measured production outcome.