Skip to main content
The worker bus is how a live change reaches every process in a deployment. When one worker persists a config change — a reloaded tool, a replaced manifest, a soft restart, a settings-profile apply, a compiled-template-cache eviction — the bus carries that change to every other worker on the same bus and collects a per-worker outcome, so the whole fleet converges instead of one process drifting ahead of its siblings. Each skeleton major has a bus of its own, so workers of another major are a separate fleet.

​
It is internal infrastructure, not a plugin

The bus is app-owned internal infrastructure, in the same class as the reload gate. It is not a plugin and sits deliberately outside the plugin system: nothing about it is registrable, swappable, or user-selectable, no manifest field chooses an implementation, and there is exactly one bus. You do not author a bus or name one in a manifest — you configure the single built-in with environment variables, and that is the whole surface. This is a deliberate line. A backend is a plugin — you pick celery, rq, or arq. The bus is not: fan-out is the app’s own concern, so a backend-runtime process receives fleet changes through the bus exactly like a serving HTTP worker, and carries no fan-out surface of its own.

​
Enable it

The bus runs on a plain Redis, configured entirely through TAI_BUS_* environment variables. Set TAI_BUS_REDIS_URL to turn it on:
With TAI_BUS_REDIS_URL unset the bus is off, and the process runs on a no-op local variant: a mutation applies in that one process and reaches no sibling. That is the correct, supported shape for exactly one deployment shape — a single worker, in file config mode, with no backend — and the wrong shape for every other (see Boot rules below). Any plain, module-less Redis runs the bus — redis:7, redis:8, or valkey. No Redis modules are required.

​
Boot rules

Three deployment shapes cannot converge without the bus, so tai serve and tai backend refuse to start without TAI_BUS_REDIS_URL when any holds:
  • More than one server worker. Sibling workers would serve stale config after a reload with no channel to converge on. Under tai serve --workers N (N > 1), each worker refuses to boot busless.
  • A registered backend. A manifest that names a backend_module runs a separate backend-runtime process that must converge with the server processes on every reload. This is re-checked on every reload, so a manifest or env change that adds a backend — or removes the bus while a backend remains — is refused too.
  • TAI_CONFIG_MODE=kubernetes. The kubernetes config mode exists for multi-pod shared config, and a pod cannot see its own replica count, so kubernetes mode always requires the bus.
A fourth shape is refused whatever the deployment: a process that reaches the bus only through a stored copy of the bus URL (TAI_BUS_REDIS_URL, TAI_BUS_REDIS or TAI_DEFAULT_REDIS_URL) or of TAI_BUS_NAMESPACE while another worker on that same bus has its deployment set that key. The stored copy counts nowhere once a worker’s deployment sets the key, and a process cannot stay on a bus its deployment did not give it, so it refuses to start, naming the keys. A process learns which keys the other workers’ deployments set from the census of the bus it reaches, so a stored copy that points it at another bus (another Redis or another TAI_BUS_NAMESPACE) is not refused: that census holds none of the fleet’s workers, and the process runs on the other bus, apart from the fleet. The refusal names TAI_BUS_REDIS_URL. A busless kubernetes boot is rejected on the missing bus, not on a later kubeconfig error, because the kubernetes check reads only boot-fixed env and runs before the config manager is built. The workers check runs in each worker at its boot, after the stored env (the config-dir .env or the config provider’s store) is applied: a bus URL kept only in the stored env counts when no worker’s deployment sets it; a bus URL the deployment’s environment sets keeps its value whatever the stored env holds, on every worker. A worker whose own environment lacks the bus URL while another worker’s sets it is refused at its boot, naming the key, when the stored copy brings it to that worker’s bus: the stored copy counts nowhere, and a worker cannot stay on a bus its deployment did not give it. A stored copy that brings it to another bus is not refused, as above. It counts the workers tai serve hands each worker in TAI_SERVE_WORKERS; a process without that count counts as one worker. A worker that refuses fails its startup, and tai serve exits non-zero. Every reload re-checks it, and a config write (an env change, a manifest change, a profile replace) that would leave a multi-worker server busless is refused. More than one worker also needs shared sub-MCP and hooks stores, checked the same way — see Scale workers. A registered backend also requires a shared hooks registry (HOOKS_REDIS_URL, or TAI_DEFAULT_REDIS_URL), at any worker count. The backend-runtime process matches the events it raises against its own hooks registry, and an in-memory one holds none of the hooks registered through the server, so those events would run no hook. The rule is checked at the same points as the bus rule for a backend: tai serve and tai backend refuse to start, every reload re-checks it, and a manifest or env change that adds a backend over an in-memory hooks registry — or removes the hooks store while a backend remains — is refused, naming HOOKS_REDIS_URL.

​
Namespacing

Every bus key and channel sits under one root, {namespace}:v{major}. The namespace is TAI_BUS_NAMESPACE (default tai), and the major is the major version of the running tai42-skeleton, so a server on skeleton 34.x with the default namespace uses the root tai:v34. The major is a fact of the installed code, not a setting. The control channel is {namespace}:v{major}:bus:control, and the reply channels, the presence keys and their index, the slot-claim keys, and the generation counters sit under {namespace}:v{major}:bus: too. The env-reader descriptions sit under {namespace}:v{major}:env-readers:. Redis pub/sub is server-global — it is not scoped by the numeric database index — so two deployments sharing one Redis would cross-deliver each other’s fleet ops unless they diverge by namespace.
When several independent stacks share one Redis, give each a unique TAI_BUS_NAMESPACE. Two stacks left on the default namespace deliver each other’s reloads and manifest replaces to the wrong fleet.

​
Two skeleton majors on one namespace

Workers of two skeleton majors on one Redis and namespace are two separate fleets: each major has a bus of its own under its own root. Neither reads the other’s presence rows or receives the other’s ops, so every bus op — a reload (the soft restart), a recycle, an MCP or tool reload or removal, a template-cache eviction, a deployment-keys change — reaches only the workers of the major that sent it. Each major numbers its own slots, so both can hold a serve-1. Every census read sees its own major alone: tai fleet workers and the fleet report of a mutation list only that major’s workers, and a save, a reload, an env or schema read, or a profile diff judges deployment ownership by that major’s workers alone. This is the shape of a rolling upgrade across a skeleton major — see Upgrades. The split covers the bus alone. The config store, the template store, and the other shared state the two majors reach — job queues and conversations among them — are not split by major, so both read and write the same copy.
A config change made through a worker of one major is persisted to the shared config store first, so it is stored for both majors; only the live reload that follows is per major. The other major’s workers apply a hot value at their next reload, and a recycle-class value only when their process restarts.A template write or delete lands in the shared template store in the same way, and only its cache eviction is per major. The other major’s workers can keep serving their cached compilation for up to TEMPLATE_CACHE_TTL (default 300 seconds; with no TTL, until an eviction on their own major or a restart drops it).

​
Worker identity

Every process on the bus holds a stable slot name — {kind}-{n}, where kind is serve or backend and n is the lowest free ordinal, so the processes of a fleet are serve-1, serve-2, backend-1, and so on. A worker claims its slot atomically at boot and holds it for the life of the process. The name is what you target a reload at and what every fleet surface labels a row with — stable, not a fresh string each start. The busless local variant names itself serve-1 and is its own fleet of one. Each slot also carries a generation — a life counter that increments every time a process claims that name. A worker that restarts takes the same slot back at the next generation, so serve-1 at generation 4 is a later life of the same slot than serve-1 at generation 3. The pair {name, generation} identifies one life: a reply from a superseded life — a process that lost its slot but has not yet noticed — is discarded on the generation, never mistaken for the current one.

​
The presence value

A worker advertises itself in a presence key whose value is the single source every fleet surface reads: The state is written live by the worker itself:
  • ready — booted, resynced, and serving.
  • resyncing — running its boot or reconnect resync; it converges to ready when the resync finishes.
  • recycling — leaving gracefully: a departure it decided (a recycle among them), or the start of a graceful stop decided outside it (a rolling restart, a scale-down). It is on its way out, not coming back under this life. The row reads recycling, with its drain_seconds, from that moment through the stop’s shutdown handlers; the stop then deletes it, before tearing down the worker’s resources. A stop decided outside starts at its first stop signal, SIGTERM or SIGINT, on either worker kind. A recycle decision and an outside stop are the same leaving: a roll that reaches a worker already stopping finds it leaving, waits out its announced drain, and asks for no second stop, and a roll handed to that worker is refused as already leaving.

​
What a leaving worker announces

The drain_seconds a leaving worker writes is the bound of its stop, read when the stop begins: its runtime’s stop phase, then the app’s lifespan shutdown, whose steps run one after another within the bounds the app’s modules declare — the hook-run attempts twice HOOKS_RUN_SHUTDOWN_GRACE_SECONDS, the credential refreshes’ close wait (30 s, for a refresh to finish its write-back), the states outbox STATES_OUTBOX_SHUTDOWN_GRACE_SECONDS, and the tool-run supervisors TAI_TOOL_RUNS_SHUTDOWN_DRAIN_SECONDS: 100 s at the defaults. A tai serve worker’s stop phase is its graceful-shutdown cap (APP_ARGS_TIMEOUT_GRACEFUL_SHUTDOWN, 10 s by default), so it announces 110 s at the defaults. A tai backend worker’s stop phase is its backend’s drain bound — for the shipped backends, the job budget plus a 30 s grace (arq: its job-completion wait plus the grace) — so a shipped backend announces 430 s at the defaults. A fleet restart waits each worker’s announced drain, plus the start-and-rejoin step, for it to go and its replacement to join. One wait falls outside the announced drain: a Celery worker run with -B waits up to 10 s more, after its drain, for its embedded beat to end (see the Celery backend).

​
A blip does not drop a worker

The presence key carries a TTL (TAI_BUS_HEARTBEAT_TTL) that the heartbeat refreshes at a third of its length. A deliberate shutdown keeps the key, reading recycling, through its shutdown handlers and then deletes it, before tearing down the worker’s resources; a transport blip does not delete it — the worker leaves the key to its TTL and, on reconnect, re-verifies it still holds the slot and beats again under the same name and generation. A short connection wobble therefore leaves presence intact: the worker stays the same life rather than vanishing and returning as a stranger. Only a blip that outlasts the TTL — or a lost claim — ends the life; the next claim of that slot then comes back at a higher generation.

​
Reading a restart in progress

The resyncing and recycling states make a restart legible while it happens — but a written state is only as trustworthy as the heartbeat behind it. Each presence row is judged against the same freshness gate the bus uses to decide who a broadcast can expect an answer from, and a row whose claim has decayed past that gate is stale. So a worker in flux reads a few ways:
  • A fresh resyncing or recycling row is genuinely restarting — the state is live and the heartbeat current.
  • A dead worker’s row vanishes when its TTL lapses; it is simply gone.
  • A ready row that has gone quiet past the freshness gate is ambiguous — it is either reconnecting or already dead, and which is unknown until it beats again or its row expires. It is reported as stale, never labelled “reconnecting” on a guess.
  • A resyncing or recycling row that has decayed past the same gate also reads as stale, not as its written state: a worker that died mid-resync or mid-recycle must not carry a promise it will converge.

​
What the fleet report tells you

A mutation waits for the fleet and returns a per-worker report — one row per worker, keyed on its name, with the op’s outcome. It waits until every worker it expects has answered or is gone, up to the apply window. While it waits it re-reads each unanswered worker’s slot claim once per heartbeat interval: a worker whose life has ended (its slot claim is gone, or a newer life holds its slot) can no longer answer, so it is reported departed then, not at the end of the apply window. A live worker keeps renewing its slot claim while it applies, so a slow apply is waited on to the end of the window. A presence row that lapsed while the slot claim still holds ends nothing: that worker is still waited on. Beyond a plain applied/failed, the report names, honestly, every worker the change did not cleanly reach:
  • missing — the worker never acknowledged the op by the end of the apply window while its slot claim stayed held: it is alive but silent. A silent worker whose slot claim could not be re-read at the end of the window, because the bus read failed, is reported missing too: its life is unknown, and the row’s detail says its life could not be re-read.
  • departed — the life the op was sent to ended before it answered: its slot claim is gone, or a newer life holds its slot (a rolling restart, a worker recycling itself, or a SIGKILL mid-reload).
  • timed_out — the worker acknowledged the op but did not finish applying it inside the apply window, which covers its full drain budget: its reload is stuck past it. Read that worker’s log (the retire names the step that overran) and verify with the fleet census or a fleet reload.
  • resyncing — the worker is mid-resync (a boot or a reconnect) and was not in a ready state to take the op; it converges on its own resync.
  • recycling — the worker is leaving gracefully (a recycle, or any graceful stop); it converges by ending its old life, not by applying this op, and under a supervisor a fresh life then takes its slot. A worker whose stop began after the op went out, and that sent no answer by the end of the apply window, is reported recycling too: its row reads recycling and its slot claim is still held.
  • stale — the worker’s row is past the freshness gate, so its state can no longer be trusted: reconnecting or dead, unknown until it returns or its row expires. A stale row carries no promise it will converge.
A worker that was not in a ready, freshly-beating state is reported as its actual condition rather than silently skipped, so a broadcast stays honest about who it could not expect an answer from. None of these outcomes is hidden: a worker the broadcast could not converge is reported, logged, and surfaced in the UI — a failed propagation is never a fake success. A recycle-class env change — a settings-profile apply or an env write (tai config env set, the Studio editor, a backup restore, a plugin install), and every reload that names a recycle-class key — drives one more thing over the bus: after the reload fans out, the writer censuses the fleet on the bus and orchestrates a process recycle across the serve/backend workers it names; a change made under a bound hands that roll to the worker it runs for, which orchestrates it after the reply (see How a recycle runs). Confirmation is by reality, not by tracking a name to its successor — a recycle is accepted when two facts hold: the targeted old lives are gone (their slots absent, expired, or already advanced to a higher generation), and fresh ready capacity of each kind has booted under the new config, counted per kind to cover the recycled workers. The report never claims a particular new life is a particular old one’s replacement; it confirms the old lives ended and the fleet came back to strength. A target the bus could not confirm (it was still applying the reload, so the recycle op got no acknowledgement) is awaited on the census within the step budget; it may still recycle once its reload returns, and its generation then bumps on tai fleet workers. The roll stops at the first target that does not converge — reported timed-out or failed with a detail — or when the bus cannot be read, and attempts neither the later targets nor the writer’s own recycle; recycle_stopped in the response names where and why. A worker whose reload was refused at its epoch build (the build discarded before its swap) yet reports recycle-class keys is not restarted: it keeps serving its current config, since the stored config did not build on it, and a target asked once ready that answers so is reported withheld without stopping the roll. The writer is never a target: it is never asked at its own roll, needs its own restart when its own state is unknown, and recycles itself after its reply only when it needs a restart and the roll converged. Because that census rides the bus itself, the keys that locate it are ones a recycle can never carry — replacements would register on a new bus, or under a new namespace, that the running census cannot see — so a config change or profile that changes TAI_BUS_REDIS_URL, TAI_DEFAULT_REDIS_URL (the bus URL’s fallback) or TAI_BUS_NAMESPACE is refused on every deployment. Set them in the deployment’s environment and restart the deployment.

​
Template-cache eviction over the bus

A change to stored template content evicts the compiled-template cache fleet-wide, not just on the worker that made the change. A template write, a template delete, a directory delete, and the manual tai templates clear-cache each broadcast a cache eviction over the bus, and every worker drops the affected compiled templates so none serves a stale compilation. The op response carries the same per-worker fanout report the mutation pipeline returns — one row per worker — and a worker that does not confirm the eviction is logged loudly rather than passed over. A directory delete evicts by prefix: every compiled template under the deleted path is dropped, and the eviction is idempotent — it converges even on a partial failure, so a re-driven delete or a worker that missed the first pass lands on the same end state. The eviction rides the bus, so it reaches only the sender’s skeleton major; the note on two majors covers how long the other major can serve its cached copy. On a persist-across-jobs backend — a celery prefork pool, whose forked children hold their own in-memory caches for the life of the child — a template-eviction or any registry-mutating fleet op turns the pool over once after it applies, so a forked child can never take new work with a stale compilation or a stale registry from before the change; a child finishing a job it already holds exits after it and takes no further job.

​
Documented limitations

The boot rules catch the shapes they can detect from inside one process. Some multi-process shapes look identical to a legitimate single busless process, so the rules cannot flag them for you. In every one of these, set TAI_BUS_REDIS_URL yourself:
  • Only tai serve gives the worker count. An external process manager (gunicorn, a supervisor, a container orchestrator) that drives the ASGI factory with its own --workers gives none, and the factory cannot count sibling processes, so each process counts as one worker and the workers rule never refuses. Set the bus, and the sub-MCP and hooks stores, in any multi-process deployment you run outside tai serve.
  • Single-worker fleets look busless. A supervisor running many tai serve -w 1 processes on one host, or several machines sharing one manifest over a network filesystem, pass every boot rule — each process legitimately sees workers=1, file mode, and no backend — yet run busless-stale. Set the bus in any multi-process file-mode deployment.
  • Mixed transports are undetectable. A stdio or sse process sharing a manifest with an http fleet is another shape the rules cannot see.
  • A shared Redis needs a unique namespace. Distinct stacks on one Redis that do not set unique TAI_BUS_NAMESPACE values cross-deliver fleet ops.
The rule of thumb: if two or more processes share a manifest, they need the bus, whether or not a rule fires.

​
Operational knobs

The timing knobs have safe defaults; change them only for a deployment whose reloads legitimately run long. The apply window is never below three drain budgets (TAI_TOOL_RUNS_SHUTDOWN_DRAIN_SECONDS): a reload’s retire drains its in-flight work (served requests, session calls and backend jobs running on the serving loop), its background runs and its retired client pools in sequence, each within the drain budget, then closes its retired checkpoint savers and stores, and a worker draining real work must answer inside the window. A bus-enabled process refuses to boot or reload, and a config change is refused, when the window is below that floor; raise the window with the drain budget, adding the reload’s own time.

​
See also

  • Live operations — the mutation surface the bus fans out.
  • Manage a fleet — inspecting the fleet and driving fleet-wide operations.
  • Backends — the task-execution plugin, which the bus is deliberately not.
  • Config and secrets — the full environment-variable reference.
  • Deploy — the multi-worker and kubernetes deployment shapes the boot rules gate.