tai serve workers, or a tai serve fleet alongside a tai backend runtime — and the same mutation must reach every one of them. The worker bus is what carries it there; the tai fleet group is the operator’s window onto the fleet the bus coordinates.
The bus is what makes a fleet, not a backend
Fleet convergence rides the app’s own worker bus — internal infrastructure you enable withTAI_BUS_REDIS_URL, not a plugin you wire in the manifest. Any multi-process deployment needs it:
tai serve server, a kubernetes-mode server, or a server with a registered backend when the bus is off — see the worker bus. A single-worker, file-mode, backend-free process is the one supported busless shape; it runs on a no-op local bus and is its own whole fleet.
Wire a backend
A backend is a separate concern: a task runtime for background, scheduled, and distributed runs. It is optional, and it does not supply the fleet bus. Name a backend module to add that runtime, and set the bus and a shared hooks registry beside it (HOOKS_REDIS_URL, or TAI_DEFAULT_REDIS_URL): the backend process matches the events it raises against its own hooks registry, so a backend over an in-memory one is refused at any worker count:
examples/fleet/backend_manifest.yaml
tai fleet info reports the registered backend’s identity, or the empty state (present: false, a 200) when none is wired. That empty state is about the backend only — the worker census and fleet reload work through the bus whether or not a backend is registered.
Read the fleet
tai fleet is a thin wrapper over the authed backend and fleet routes — info reads /api/backend for the backend’s identity, while workers (/api/fleet/workers) and reload-config (/api/fleet/reload-config) live under /api/fleet/* for the census and the soft restart:
workers lists every process subscribed to the answering server’s bus — serving HTTP workers and any backend runtime alike — read from the presence census (each skeleton major has a bus of its own). It fails loudly: a store it cannot read is a 500, never a silently empty fleet. Each row is one worker:
- name — the worker’s stable slot (
{kind}-{n}, e.g.serve-1). This is the value you pass to--target, and it stays the same across a restart. - kind —
serveorbackend. - pid — the OS process id of the current life.
- gen — the generation (life counter) for that slot; it advances each time a new process claims the name, so
serve-1gen4is a later life than gen3. - state — what the worker last wrote about itself:
ready(serving),resyncing(running a boot or reconnect resync), orrecycling(leaving gracefully — a recycle or any graceful stop — from the start of its stop until the stop ends, with the drain it announced). A(stale)suffix means the row is past the freshness gate — its state can no longer be trusted (reconnecting or dead, unknown until it beats again or its row expires). - seen-since — how long ago the worker last beat.
- last-op — the last op it applied and that op’s outcome, or
—if it has applied none.
resyncing/recycling rows with a fresh seen-since are workers mid-flight, a dead worker’s row simply drops out when its TTL lapses, and a ready row that has gone quiet turns (stale) rather than being labelled reconnecting on a guess. The full subcommand set:
examples/fleet/fleet_help.sh
Soft-restart the fleet
reload-config is the fleet-wide soft restart: every worker refreshes its environment values and reloads the manifest registries in place, picking up configuration and capability changes without dropping its process. The soft restart carries every hot change in place. A recycle-class value changed in the stored env since a worker bound it (a hand edit of .env, a kubectl edit of the config Secret) is named instead — see the awaiting_respawn list below — and the soft restart then restarts the workers that name one. Restart the whole fleet, or restrict it to named workers with a repeatable --target:
--target takes worker slot names — the name column from tai fleet workers (serve-1, backend-2), not a pid or an opaque id.
The response is the per-worker fleet report — one row per worker with its reload outcome and, in its payload, awaiting_respawn: the recycle-class keys whose value differs from the one that worker bound (at boot, or when the plugin that declares them was loaded). A worker whose row names awaiting_respawn keys converged on everything a soft restart carries, and those keys take effect only when that worker is respawned. So once the reports are in, the soft restart rolls a recycle over the workers it addressed: each one that names such a key is restarted, one at a time — it exits gracefully and the deployment’s supervisor starts it again — and each other one whose state is unknown (its reload did not confirm, it has no row, its reply carries no awaiting_respawn list, or the bus was unreachable) is asked once it is ready and restarted unless its reply confirms it clean or states a refused build. A worker whose reload was refused at its epoch build (the build discarded before its swap) keeps serving its current config: its row is failed, with the payload {"status": "discarded", "awaiting_respawn": [...]}. A discard that itself failed is no refused build: that row carries no payload and leaves the worker’s state unknown. When that list names keys, the roll withholds its restart, since the stored config did not build on it and a restart may fail the same way; it is left running and stays named in the top-level awaiting_respawn, and its own ERROR line says to fix the stored config, or the cause the build error names, before restarting it. The workers that confirmed clean keep running. The worker that ran the reload command is never asked: its own row is its statement, and when its own state is unknown it needs its own restart. Only when it needs one and the roll converged does it restart itself last, after its reply (its self-deferred row). The response’s recycle is that roll (null when the roll had no worker to visit: every addressed worker confirmed no recycle-class key, or the only ones naming one were refused at their epoch build), and its top-level awaiting_respawn names the workers (name and keys) still awaiting a respawn after it; the worker that ran the reload command logs an ERROR line naming them, and a separate one naming the withheld workers. A reload run under a bound (a backend job, or a call under the turn budget) hands its roll to the worker it runs for, which carries it out on its own: roll_deferred is then true, roll_deferred_to names that worker, recycle is null and awaiting_respawn empty, and the roll’s outcome is in that worker’s log. A worker still named after the roll, and not withheld, is restarted through the deployment’s supervisor (kubectl rollout restart, docker compose restart <service>, systemctl restart <unit>); verify the generation bump on tai fleet workers. A worker the reload did not cleanly reach is named, never hidden: departed (it went away), timed_out (it acknowledged but is stuck past its drain budget), missing (alive but never acknowledged), or one of the gap outcomes resyncing (mid-resync), recycling (leaving gracefully, under a recycle or any graceful stop), and stale (past the freshness gate, its state no longer trustworthy). A failed local apply raises; a remote miss is reported and logged. Re-run reload-config to converge the named workers.
How a change reaches the fleet
Fleet reach is not a separate command — it rides every mutating operation. When a tool is reloaded or a capability is removed, the route persists the change, applies it locally, and then broadcasts it on the bus so every worker re-reads the durable state and rebinds; that response embeds the per-worker fleet report, so a propagation that did not converge is visible on the flow that caused it. A preset mutation broadcasts the same way, but its response is the plain preset record — a preset broadcast that does not converge is surfaced server-side as a loud non-convergence ERROR log rather than embedded in the response, and a fleetreload_config reconverges the named workers. See live operations for the mutation surface the bus backs.
Config writes are exclusive across every process that shares one config store, and across the writers of one process: every serve worker, every replica, and the CLI. Each write holds the store’s exclusive write span from its first read of the stored config through its persist, so two writes never interleave, whichever worker or replica receives them. A write is validated against the env the store converges every worker to, not against the writing worker’s own process env. So a manifest !ENV marker on a key that a concurrent write has just dropped is refused as dangling, naming the key, instead of being stored where no worker can resolve it.
A 200 does not mean every worker converged. A fleet broadcast applies the change locally first, then asks every reachable worker to confirm. The local apply is committed before the call returns, so the call returns 200 even when a sibling worker did not confirm — read the body: ok is false when any reachable worker has not converged, reachable tells you the bus was reached, and results[] carries each worker’s outcome (applied means it converged; anything else — resyncing, recycling, stale — names a worker that did not). A client that only checks the HTTP status will miss an unconverged sibling: check ok, recycle_stopped, awaiting_respawn, roll_deferred and span_release_error. A change whose reload names a recycle-class key — a profile apply, an env write, a config reload — adds a second place to read: its recycle rows (an env write’s response carries them under recycle, null when the roll had no worker to visit: every worker’s reload confirmed no recycle-class key, or the only ones naming one were refused at their epoch build or are the writer whose own restart cannot follow as its exit). Each row is recycled once the target’s old life is gone and fresh capacity booted; a timed-out or failed row is where the roll stopped, with a detail naming why, and the targets after it — and the writer’s own recycle — were not attempted. Two rows do not stop the roll: self-deferred (a restart the roll cannot confirm: the writer’s own self-exit after its reply, or a target whose exit is held by its own in-flight change, which the roll does not wait for) and withheld (a target asked once ready whose reload was refused at its epoch build yet reports keys, left running on its current config). recycle_stopped names where the roll stopped and why (null when the roll ran to its end); its name is empty when the bus could not be read before a target was reached. Its awaiting_respawn is the operation’s verdict: the workers (name and keys) still awaiting a respawn after the roll — empty when no worker is left owing one; a stopped roll leaves the workers it did not recycle listed, a withheld worker stays listed, and the writer logs an ERROR line naming them, with a separate one for the withheld workers. A change made under a bound — a backend job’s budget, or the turn budget of a live call — answers before its roll: roll_deferred is true and roll_deferred_to names the worker that carries the roll out on its own, so the body carries no recycle rows and an empty awaiting_respawn; read that worker’s log and tai fleet workers for the roll’s outcome (see How a recycle runs). roll_deferred is false and roll_deferred_to null when the roll ran inside the change. span_release_error is the one place to read for the config store itself: null when the store released its exclusive write span after the persist; otherwise the store’s error. The change landed, and the reload, broadcast and recycle the body reports still ran, but the store may still hold its write span until the writer process exits, so a later write can block or be refused until then; the writer logs one ERROR line with the traceback. A non-2xx is reserved for a local apply that itself failed or a malformed request — a case where the change did NOT land. A roll hand-off that failed raises too, though the change landed: the error names the worker, says the change is persisted and reloaded, and says whether the roll started.
To recover a worker named timed-out or failed: watch tai fleet workers — a target the bus could not confirm may still recycle once its reload returns, and its generation bumps. A failed row or a stop whose detail names the bus is verified on tai fleet workers once the bus answers: a target the recycle op reached may have recycled (its generation bumped). One that did not is restarted through the deployment’s supervisor, then verified on the census.
See also
- The worker bus — the internal fan-out primitive and its boot rules.
- Backends — the optional task-execution runtime.
- Live operations — the reload/remove/update surface that broadcasts across the fleet.
- Deploy — scaling to multiple workers and the durable stores a fleet needs.
- CLI reference — the full
tai fleetsurface.