Skip to main content
A backend is where a tool actually runs when it should not run inline in the request. It is the execution engine behind background, scheduled, and distributed tool runs — and it is swappable, so the same tool runs the same way whichever engine you choose.

​
The backend seam

The Backend ABC is the contract; the skeleton re-exports it as the registration seam and ships the callback glue that chains a backend result back into the runtime. Concrete worker backends — for example ones built on Celery, RQ, or arq — are external plugins that implement the ABC and register through tai42_app.backends.register_backend when the manifest names the backend_module. The backend runs as its own process:
A registered backend adds a separate task runtime, so the deployment is now a fleet of more than one process. Keeping those processes converged on a config change is not the backend’s job — it is the app’s own worker bus. Because a backend makes the deployment multi-process, naming a backend_module requires the bus: set TAI_BUS_REDIS_URL or the server refuses to start. It also requires a shared hooks registry: set HOOKS_REDIS_URL (or TAI_DEFAULT_REDIS_URL), whatever the worker count. The backend process raises platform events of its own and matches them against its own hooks registry; an in-memory one there holds none of the hooks registered through the server, so those events would run no hook. tai serve, tai backend, a config reload and a config write all refuse a backend_module over an in-memory hooks registry.

​
Three ways a tool runs off the request path

  • Background — submit a tool run and poll its result, for a run that outlives one request. The background tool-runs HTTP surface loads as a router module.
  • Scheduled — schedule a tool to run later or on a cadence. A schedule names the base tool_name, its tool_kwargs, and a typed cadence (an interval or a crontab). The tool must carry the backend’s schedule_task extension to be scheduled on a cadence — see the schedule-a-tool guide for attaching it.
  • Distributed — the worker backend spreads runs across workers.
Manage schedules from the CLI:

​
The backend tools

Every backend offers the same 22 backend_* tools, tagged backend. The kit builds them once over the backend’s seams, so each tool takes one argument list on every backend; a backend supplies only what its engine reads and writes. The ten schedule tools read and write the backend’s schedule store: A schedule tool’s status reply is {"status": ..., "name": name}, and a name with no schedule replies {"status": "not_found", "name": name}. An update given no new cadence and no next run writes nothing and replies {"status": "skipped", "name": name, "message": "No changes provided; nothing updated."}. What a backend’s store cannot represent is refused on every write: a relative interval on a backend that honours none, and a disabled schedule on a backend that holds none. On such a backend every stored schedule is enabled already, so backend_enable_schedule replies enabled and writes nothing, and backend_disable_schedule raises UnsupportedBackendTool, leaving the schedule as it is. The twelve task and worker tools read the backend’s own views of its tasks and workers: backend_ping_worker, backend_list_active_workers, backend_task_status, backend_task_result, backend_cancel_task, backend_active_tasks, backend_reserved_tasks, backend_scheduled_tasks, backend_registered_tasks, backend_list_failed_tasks, backend_worker_stats and backend_worker_queues. Each replies what the backend reports, in the backend’s own shape and words. backend_task_result replies Task <id> is not ready (status: <status>) while the task is unfinished, and raises a failed task’s failure.
  • worker_name. backend_ping_worker, backend_active_tasks, backend_registered_tasks, backend_worker_stats and backend_worker_queues take an optional worker_name that narrows the call to one worker. A backend that cannot address one worker refuses any worker_name but None.
  • backend_cancel_task(task_id, terminate=False). Has one meaning on every backend. Without terminate, a task that has not started is kept from running, and a running task is left to finish. With terminate=True, a running task is stopped as well. A backend that cannot stop a running task refuses terminate=True, and one that cannot cancel a running task without stopping it refuses terminate=False for a running task. When a worker answers that it cannot stop the task it runs, terminate=True is refused for that task rather than leaving it running behind a reply that says otherwise. Each refusal raises UnsupportedBackendTool. The reply says what was done; a stop the workers did not confirm within the time the backend waits for it is reported as not confirmed, with the status the task reached.
  • What a backend cannot serve. A tool, or a form of one, that a backend cannot serve is still registered, and raises UnsupportedBackendTool naming the backend and the tool when it is called. Each backend’s page in the Plugins section lists what it refuses.
See Backend tools for the kit API that builds them.

​
Queue labels

The <tool>_sync_task and <tool>_async_task branches take an optional queue argument: a label of letters, digits, _ and - naming a worker pool. A task sent with queue: gpu goes to the queue <QUEUE_NAME>.gpu, where <QUEUE_NAME> is the backend’s own queue setting (ARQ_QUEUE_NAME, CELERY_QUEUE_NAME, RQ_QUEUE_NAME). Any other value is refused before anything is queued, and a task sent with no queue stays on <QUEUE_NAME>. A preset can fix queue in its arguments like any other argument. A tool with a queue parameter of its own cannot take either branch. A worker serves one queue. Start it with a label to serve that label’s queue alone:
A worker started without --queue serves <QUEUE_NAME>. A task on a queue no worker serves waits: a sync_task raises TimeoutError after the job budget, and an async_task id stays queued. Recurring schedules carry no queue, so a stack that runs schedules keeps a worker without a label.

​
Chaining a job with a callback

A job does not wait for another job. It chains the next one with callback_kwargs, an option of the sync_task and async_task branches. The callback runs as its own job, only when the job it follows succeeded, on that job’s queue (its label included). On a backend whose callback job reads the job it follows back from the engine, a callback job that finds that job unfinished or failed raises, so a broken chain is a failed job, never a silent one. Inside a job, sync_task and backend_task_result given a timeout raise RuntimeError instead of waiting.

​
The job budget

Every backend bounds a dispatched job by one setting, the dispatch group’s task_timeout (<PREFIX>_TASK_TIMEOUT, 300 seconds by default): it is the longest a job may run on the worker, and how long a synchronous dispatch waits for its result. It covers every job the worker runs — a dispatched task, a schedule fire, a callback. Celery applies it as task_time_limit, RQ’s worker sets it as the timeout of every job it runs (whatever timeout the job was stored with), and arq as the worker’s --job-timeout default. The worker binds it when it starts, so a change converges through a process recycle. A backend never retries a tool body. A tool that wants retries declares a ToolRetryPolicy, applied by the platform’s dispatch seam on every door. What a backend still retries is its own transport — publishing a message on connection loss, reconnecting to its broker.

​
Writing a backend

A backend plugs its engine into three seams declared in tai42_contract.backend, and the kit builds every tool over them. What the seams cannot express, the backend’s engine does not offer, and the tools refuse it the same way on every backend.
  • The task door. A TaskDoor queues a dispatched call on the engine (submit), names its job (task_id) and waits for its result (wait, raising TaskWaitTimeoutError when the job is unfinished at its timeout). Its task_opts declare the backend’s own task options beside the ones every backend honours (countdown, expires, eta, callback_kwargs, queue).
  • The schedule store. A ScheduleStore is the engine’s durable home of schedules: read, read_all, exists, delete, set_enabled, run_now and update, read back as StoredSchedule. Its ScheduleWriter part is the one writer every write door shares (write), and its capabilities say what it can represent: relative_intervals and holds_disabled.
  • The worker inspection. A WorkerInspection answers the task and worker views. The kit’s BackendInspection base refuses each view with UnsupportedBackendTool until the backend overrides it; addresses_workers and terminates say whether the backend can narrow a view to one worker and stop a running task.
The backend registers the tools at import with register_task_branches (the sync_task, async_task and schedule_task BACKEND extensions over its door and writer), register_schedule_tools (over its store) and register_inspection_tools (over its inspection). A schedule’s tool rides inside the stored job kwargs under the backend’s own tool-name key; the kit splits it out into the record’s tool on export and list, and injects it back on import. The kit’s conformance checks certify a backend in its own tests: check_backend_declarations checks its runtimes, and check_tool_surface checks that every backend_* tool is registered under the backend tag and no other one is. tai42_kit.utils.runtime.normalize_schedule parses the friendly specs a schedule tool accepts (seconds, a crontab string, {"type": "interval", "every": 5, "period": "minutes"}) into the contract’s cadence. See Backends for the seams and Backend dispatch for the branch tools, their task options and the queue labels.

​
Routing a tool to a backend

A tool is routed to a backend with a BACKEND-kind extension — the extension kind whose job is to send a tool’s execution through a backend rather than running it inline. The tool’s definition does not change; the extension decides where it runs.
A tool that depends on a managed stdio MCP server’s tools runs on the ASGI, arq, and RQ execution paths, but not inside a Celery prefork worker child: the managed server’s live stdio subprocess connection does not survive the prefork fork, so those tools are unavailable there. Route such a tool to a non-prefork engine, or run it inline on the request path.
A persist-across-jobs backend — a Celery prefork pool, whose forked children hold their own in-memory caches for the life of the child — turns its pool over once after any template-eviction or registry-mutating fleet op reaches it over the worker bus, so a child never keeps serving a stale compiled template or a stale registry from before the change.
A specific worker backend’s setup — its broker, queues, and concurrency — lives in that backend package’s own repository, reachable from the Plugins section. The platform documents the backend contract, not any one engine.
See the schedule-a-tool guide and the HTTP API reference for the tool-run and schedule routes.