Own Parallel LLM Calls with Python TaskGroup

Last reviewed: November 1, 2026 (Asia/Shanghai).

Direct answer

Use asyncio.TaskGroup when several parallel LLM calls are all required for one result. Create those calls through the group, put their shared deadline outside it, and read their results after the group exits successfully. When one child raises an ordinary exception, the group cancels the remaining children and waits for them to finish before raising the collected failures. This gives the parent a clear point at which its owned local tasks have ended.

The Python Coroutines and Tasks reference documents an important difference from default asyncio.gather(): gather propagates the first raised exception while other submitted awaitables continue running. In a long-lived gateway, returning an error from the parent therefore does not necessarily end its other model calls. Calling gather.cancel() after it has already propagated the exception does not fix that lifecycle.

For example, a report may require two independent document summaries. If either summary fails validation, the product cannot produce the report. The other summary should receive cancellation, finish its cleanup, and stop contributing local work before the report workflow returns. Choosing the fastest of two equivalent answers is a different policy; this article implements the all-required case.

Who this is for

This guide is for Python gateway maintainers running concurrent model subtasks inside one request. It applies to small workflows such as two required summaries or several required extraction results. The examples require Python 3.11 or later; the references checked here are the versioned Python 3.14 documentation.

Use this pattern for generation whose external effects are deferred until the complete result is accepted. A task group cannot roll back a tool execution, database write, or already delivered stream. Independently durable jobs also need a separate owner and recovery contract.

Key takeaways

• Register every request-owned branch with the group; detached tasks have a different lifetime.
• Treat a normal child failure, parent cancellation, and a child canceling itself as different events.
• Preserve one deadline and keep cleanup cooperative.
• Validate each branch inside its task so an unusable result can stop siblings.
• Bound branches and upstream resources separately; a task group supplies ownership rather than admission control.

Contract details to verify

Define which results are required before selecting a concurrency primitive. If an optional branch may fail while the rest remain useful, convert only its expected failure into an explicit optional-result value. An all-required task group should still propagate unexpected failures. Catching every exception and returning an empty string hides the condition that should stop the workflow.

The Python task reference recommends try/finally for cleanup and generally propagating CancelledError after cleanup. It warns that swallowing cancellation can disrupt TaskGroup and timeout behavior. Avoid except BaseException handlers that turn cancellation into a retry or a successful default.

Ownership must extend into the adapters. A branch that starts an untracked asyncio.create_task() and returns has detached work from this guarantee. Await nested work in an owned group. Likewise, shielding a separate task can allow it to continue after its awaiter is canceled; that needs an explicit owner rather than being an automatic remedy for cleanup problems.

A bounded coordinator

Save the following as required_calls.py. Each configured callable accepts the same absolute deadline and returns validated text. Provider authentication, endpoint selection, request limits, response-byte limits, and response-envelope validation belong to those adapters. This coordinator makes no provider calls itself.

import asyncio
import math
from collections.abc import Awaitable, Callable

Call = Callable[[float], Awaitable[str]]
MAX_BRANCHES = 4
MAX_RESULT_CHARS = 8_000


class InvalidBranchResult(ValueError):
    pass


async def collect_required(
    calls: dict[str, Call], *, deadline: float
) -> dict[str, str]:
    loop = asyncio.get_running_loop()
    if not math.isfinite(deadline):
        raise ValueError("invalid_deadline")
    if not 1 <= len(calls) <= MAX_BRANCHES:
        raise ValueError("invalid_branch_count")
    if deadline <= loop.time():
        raise TimeoutError("workflow_deadline")

    async def run_one(call: Call) -> str:
        value = await call(deadline)
        if (
            not isinstance(value, str)
            or not value.strip()
            or len(value) > MAX_RESULT_CHARS
        ):
            raise InvalidBranchResult("invalid_branch_result")
        return value

    tasks = {}
    async with asyncio.timeout_at(deadline):
        async with asyncio.TaskGroup() as group:
            for name, call in calls.items():
                tasks[name] = group.create_task(run_one(call))
        if loop.time() >= deadline:
            raise TimeoutError("workflow_deadline")
        return {name: task.result() for name, task in tasks.items()}

The branch and character limits are illustrative application choices. The character check runs after the adapter returns; it does not bound downloaded bytes or make JSON parsing safe. Enforce those limits before buffering and parsing upstream content. Use fixed internal branch names rather than prompt text or user identifiers.

Create the absolute deadline from asyncio.get_running_loop().time() at the owning request boundary and pass it through. Do not pass a Unix timestamp into timeout_at() or restart the full budget for each branch. Follow the site’s deadline propagation guidance when converting budgets between services. Reserve time for response delivery outside the upstream-work budget.

timeout_at() requests cancellation when its deadline is reached. The timeout scope normally translates its own cancellation into TimeoutError outside that scope. This is cooperative enforcement: blocking code or slow cleanup can delay the return, and cleanup failures can affect the exception observed. The final clock check also rejects a result that reaches the return path after the deadline. Neither check supplies hard preemption.

Keep provider pools and workload admission bounded outside this function. Four children per request can still mean thousands of children across concurrent requests. The bulkhead pool guide explains the separate capacity boundary.

Keep transport cleanup inside the branch

The HTTPX Async Support guide documents async with client.stream(...) and automatic response closure on block exit. An adapter should finish consuming or unwinding that scope before its branch ends. With manual streaming through client.send(..., stream=True), the developer must eventually call response.aclose(); HTTPX warns that leaving it open can leak resources.

Reuse an async client within its owning application lifecycle, and close that client after active workflows have drained at shutdown. Do not close a shared client simply because one workflow fails. Avoid creating a new client in every branch or hot loop. Blocking transformations still require separate handling: a coroutine that monopolizes the event-loop thread delays siblings and cancellation alike.

Happy-path operator workflow

Admit one small workflow, establish its deadline, and configure only the required adapters. Each adapter performs one policy-approved branch operation, validates the provider response, and returns text. The coordinator validates that text inside the group. After every branch completes, the caller receives one complete mapping and may commit the report.

Verify that the configured branches started once, each finished, every response scope closed, and no detached work remains. Record workflow outcome, fixed branch labels, branch outcomes, elapsed durations, and cleanup completion. Keep generated text, prompts, credentials, and raw exception messages out of operational records.

Error-path operator workflow

If a required branch raises an ordinary exception, let it escape the branch. The group requests cancellation of siblings, awaits their termination, and then raises an exception group. Only after that unwind should the request boundary map the failure to the product’s stable error response. Do not retry the entire report simply because the outer exception is grouped: an already completed branch may have consumed usage or performed other work.

The Python Errors and Exceptions tutorial explains that ExceptionGroup can preserve several failures and that except* handles matching portions while unmatched exceptions propagate. Classify the leaf failures into bounded categories. Preserve unexpected failures for investigation instead of suppressing every group with except* Exception: pass. A cleanup error and an upstream error can both matter.

For parent cancellation, propagate the signal after local cleanup. For a shared deadline, stop the workflow and report timeout according to the observed exception and scope state. Apply the existing cancellation and fallback rules : an ended parent action must not launch another provider attempt.

One subtle case needs its own policy. TaskGroup does not treat a child raising CancelledError as an ordinary failure that cancels its siblings. It waits for the remaining children. In this coordinator, accessing the canceled task’s result then propagates CancelledError, preventing a complete-success mapping. Adapters should report an ordinary branch failure with an appropriate exception, rather than using self-cancellation as a generic failure code.

Verify ownership with a local probe

Save this as probe_required_calls.py beside the coordinator and run python3 probe_required_calls.py. It uses events to establish task ordering and never contacts an API.

import asyncio
from required_calls import collect_required


async def main():
    started = asyncio.Event()
    cleaned = asyncio.Event()
    never = asyncio.Event()

    async def slow(deadline):
        started.set()
        try:
            await never.wait()
            return "unused"
        finally:
            await asyncio.sleep(0)
            cleaned.set()

    async def fail(deadline):
        await started.wait()
        raise ValueError("synthetic_failure")

    saw_failure = False
    try:
        await collect_required(
            {"slow": slow, "failure": fail},
            deadline=asyncio.get_running_loop().time() + 2,
        )
    except* ValueError:
        saw_failure = True
    assert saw_failure and cleaned.is_set()

    started.clear()
    cleaned.clear()
    parent = asyncio.create_task(collect_required(
        {"slow": slow},
        deadline=asyncio.get_running_loop().time() + 2,
    ))
    await started.wait()
    parent.cancel()
    saw_cancel = False
    try:
        await parent
    except asyncio.CancelledError:
        saw_cancel = True
    assert saw_cancel and cleaned.is_set()
    print("sibling failure and parent cancellation: passed")


asyncio.run(main())

A local run on November 1, 2026 using Python 3.12.3 passed both assertions. Additional local checks passed for complete results, invalid and oversized text, invalid branch counts, non-finite and expired deadlines, deadline cleanup, child self-cancellation, and two grouped failures. These checks establish coordinator behavior with synthetic coroutines. They do not verify an HTTP transport, provider cancellation, remote execution, or billing.

For staging acceptance, repeat failure and cancellation through the actual adapters. Observe response closure, permit release, grouped errors, and work remaining after parent exit. Exercise cancellation during connection acquisition and response reading, plus simultaneous cancellation and branch failure on the Python version you deploy. The reference notes improvements to simultaneous internal and external cancellation handling in Python 3.13; the local 3.12.3 run is not proof of identical race behavior across versions.

Failure modes

Default gather returns an error while siblings continue. Select an owning group for all-required work, or explicitly implement cancellation and joining when a different primitive is necessary.

A branch returns before its own tasks finish. The group owns only the tasks registered with it. Keep nested work awaited and avoid detached provider calls.

Cleanup suppresses cancellation. A retry wrapper catches too broadly or converts a cancellation into default text. Preserve cancellation and verify cleanup in the failing path.

Optional enrichment aborts a required report. Specify which expected optional failures become data. Keep optional-result policy explicit rather than weakening every branch’s error handling.

Local cancellation is reported as remote termination. The local branch ended, but the provider may already have done work. Keep usage and remote outcome unresolved until separate evidence establishes them.

FAQ

Does TaskGroup undo successful sibling calls?

No. It requests cancellation of remaining tasks after a normal child failure and waits for its owned tasks. A completed model call, usage charge, or external write remains a separate fact. Put irreversible effects behind the final acceptance boundary.

Is gather with return_exceptions=True equivalent?

No. It collects exceptions as results and waits for the submitted awaitables. It does not provide the same ordinary-failure cancellation policy. It can suit a deliberately independent collection when every outcome is inspected and the deadline and ownership rules are explicit.

Can a deadline force blocking code to stop?

No. asyncio cancellation needs cooperative execution. Bound input and parsing work, keep blocking operations off the loop, and measure cleanup duration. A task group cannot supply an operating-system kill boundary.

Does leaving the group guarantee that the response succeeded?

The group establishes completion of its registered tasks. Application success still requires valid results. This coordinator also rejects blank or oversized text and propagates a canceled child’s result access; adapters remain responsible for the provider’s complete response contract.

Reader next step

Choose one workflow with two required, independent model subtasks. List each task’s owner, success predicate, deadline, response cleanup, and external effects. Run the probe, then repeat its failure and parent-cancellation cases through your staging adapters. Accept the change only when the complete result returns once on success and all owned local work has ended before a failed or canceled workflow returns.

Sources checked

• Python 3.14: Coroutines and Tasks — fetched November 1, 2026 UTC. Supports TaskGroup membership, group exit, ordinary failures, child cancellation, gather behavior, cancellation propagation, timeout semantics, and version differences.
• Python 3.14: Errors and Exceptions — fetched November 1, 2026 UTC. The multiple-exceptions section supports exception groups, selective except* handling, and propagation of unmatched failures.
• HTTPX: Async Support — fetched November 1, 2026 UTC. Supports async client reuse and closure, context-managed streams, and the developer’s closure responsibility for manual streaming.

These maintained primary references support an evergreen Python reliability task. This article does not announce a new release or provider incident. The all-required workflow policy, illustrative limits, coordinator, logging choices, and staging acceptance criteria are engineering recommendations.