Skip to main content
You operate a running server without a restart. Reloading a tool, replacing the manifest, re-probing a failed MCP server, or removing a capability all happen live, through the operations layer — one surface that projects to an HTTP route, a CLI command, and (curated) an MCP tool.

​
The operations layer

Every management capability is an operation — one decorated async function that carries its own metadata: a name, a summary, the typed errors it may raise, and flags such as destructive. An operation is the single source, and three surfaces project from it:
  • an HTTP route under /api/*, the door the CLI and the Studio drive;
  • a CLI command in the tai groups;
  • an MCP tool, registered through the normal tool binding so an extension wraps it, a preset bakes over it, and tai tools run or an agent dispatches it — subject to curation (below).
Because the three surfaces come from one declaration they cannot drift. The management operations cover the live-ops families:
  • Tools — re-register one app tool from its stored definition (reload_tool), drop a tool from the live registry (remove_tool), and run a registered tool by name (run_tool).
  • Config — soft-restart in place by refreshing the environment and reloading the manifest registries (reload_config), and apply a whole settings profile — replace the stored env with a saved band, swap in a fresh serving epoch, and recycle any pooled resource the change tore down (apply_profile).
  • Manifest — replace the whole persisted manifest, fleet-wide (update_manifest). The change is written through to durable config and then broadcast, so it survives a restart and reaches every worker.
  • MCP servers — list the servers a viability check skipped (list_failed_mcps), re-probe and re-attach one server or every failed one (reload_mcp, reload_failed_mcps), and detach a server’s tools by title (deregister_mcp).
  • Manifest sections — add or remove a single mcp, tools, or agents entry by title (an add optionally replacing a colliding title in place), and add or remove names on the api_tools include/exclude lists — each a persisted, hot-reloaded edit to one item, so changing one entry no longer means rewriting the whole section.
  • Tool extensions — attach or detach a single extension combo on a tool, leaving its other combos in place, alongside the whole-list write.
  • Access control — add or remove one scope on a key, and set or remove one tag grant on a role — an item-level edit that never rewrites the whole scope set or grant map.
Drive them from the CLI:

​
Curating the MCP surface

Which operations project as MCP tools is set by the manifest’s api_tools block. Two exclusion tiers guard the surface regardless of what a manifest asks for:
  • Tier 1 — never projectable. A meta-executor — an operation that runs any tool by name, run_tool — is a universal authorization bypass, so it is hardcode-blocked from the MCP surface even when named in api_tools.include. It stays a route, the tai tools run command, and an internal dispatch target.
  • Tier 2 — off by default, includable. An authority-changing operation — one that mints or scopes keys, edits policy, replaces the manifest (update_manifest), or restores unshipped state (a backup import) — is off the default MCP surface but projects when an api_tools.include names it explicitly.
A destructive operation carries a destructiveHint annotation on its projected tool (and an x-destructive marker on its route in the OpenAPI spec); it only projects when api_tools.expose_destructive is on.

​
Authorization on every MCP surface

AuthzMiddleware authorizes every projected-operation dispatch. It runs at tool call time — before the extension chain, so before any backend enqueue — on every MCP-serving instance: the main server and each sub-MCP mount. It resolves the dispatched tool to its base operation and, for a projected operation, runs the access-control check; a denial raises a loud tool error. A tool that is not a projected operation passes straight through — its authorization is a separate concern.

​
The mutation pipeline

A mutating operation runs one pipeline: it validates the change, persists it transactionally to durable config, reloads the calling process, then broadcasts it on the worker bus and awaits a per-worker outcome from every worker on that bus (each skeleton major has a bus of its own). Persist-before-broadcast is why a mutation survives a restart and why the persisted state and the live state cannot drift apart through a supported path. The broadcast reaches every process on the bus — serving HTTP workers and any backend runtime alike — not just the one that received the request. Fan-out is the app’s own concern, carried by the bus; it is not a backend capability. Every mutation response embeds the fleet report, keyed per worker on its name, so a worker the change did not converge is named — departed, timed_out, missing, or one of the gap outcomes resyncing, recycling, and stale for a worker that was not in a ready, freshly-beating state — never a fake success. A failed local apply raises; a remote miss is reported and logged. Inspect the fleet and re-drive a reload with the tai fleet group. Because a mutation reaches every worker, the deployment shapes that run more than one process need the bus: the boot rules refuse a multi-worker tai serve server, a registered backend, or kubernetes mode without TAI_BUS_REDIS_URL. A registered backend also needs a shared hooks registry (HOOKS_REDIS_URL, or TAI_DEFAULT_REDIS_URL), so a mutation that adds a backend over an in-memory hooks registry, or removes the hooks store while a backend remains, is refused at the validate step, naming HOOKS_REDIS_URL; the same holds for the bus. A multi-worker tai serve server also needs shared sub-MCP and hooks stores, so a mutation that would leave it without the bus or with either store in-memory is refused at the validate step too, naming the setting. A settings-profile apply extends this pipeline at the reload step: instead of refreshing in place it builds a fresh serving epoch off to the side under the proposed env and swaps it in atomically, retiring the old one — a failed build is discarded with the old surface still serving. This same epoch build backs a plugin install or upgrade: any reload that rebuilds the serving surface re-imports each plugin package and its route-sibling modules and re-registers the declared routes, so a plugin’s HTTP surface — a router or middleware, a channel’s inbound webhook — comes live on the epoch swap of the reload that changed it rather than waiting for a process restart. Before it commits, an epoch route audit compares the rebuilt route set against the outgoing one and refuses to swap in an epoch that would silently drop a still-declared plugin’s routes; the prior epoch keeps serving and the failure is surfaced. For a changed key whose reload class is recycle, it then orchestrates a process recycle across the fleet. Its report is per worker: each recycled row carries a status and the generation_before it was recycled from, alongside a per-kind list of the fresh lives that have since joined. Confirmation is by reality, not successor identity — a recycle is accepted on two facts: the targeted old lives are gone (their slots absent, expired, or already at a higher generation), and fresh ready capacity of each kind has booted under the new config, counted per kind to cover the recycled workers. The targets are the workers on the census when the apply started together with those on it when the roll begins, so a worker that briefly dropped off the census is waited for rather than skipped. A timed-out or failed row is where the roll stopped: timed-out (within the step budget the old life did not go or fresh capacity did not boot, or the target never returned to ready) or failed (the worker refused the op, or the bus could not be reached while it was in hand), with a detail naming why; the roll also stops when the bus cannot be read before a target is reached. Two rows do not stop the roll. A self-deferred row is a restart the roll cannot confirm: the writer’s own self-exit after its reply, or a target whose exit is held by its own in-flight change (it leaves once that change’s reply is answered; the roll does not wait for it). A withheld row is a target asked once ready whose reload was refused at its epoch build (the build discarded before its swap) yet reports recycle-class keys. A withheld target is left running on its current config: the stored config did not build on it, and a restart may fail the same way. A discard that itself failed is no refused build: it leaves the worker’s state unknown. The response’s recycle_stopped names where the roll stopped and why: its name is empty when the bus could not be read before a target was reached, and it is null when the roll ran to its end. Later targets and the applier’s own recycle are then not attempted, and the call still answers 200 with the report, as every fleet mutation does.

​
Reloading is a first-class state

While a reload is in flight, the routes that dispatch a run or mutate live registries return a retriable 503 carrying a reloading marker and a Retry-After header, rather than serving against a half-swapped registry. A client treats it as a transient condition and retries — the reload is a state the API describes, not an error. The refusal holds on every door that schedules a conversation turn, not only on the API’s own message and event routes: a channel webhook (a vendor’s inbound message, the web chat’s send) that arrives during a reload is answered the same retriable 503, and nothing of it is recorded, so the vendor’s redelivery is processed in full. A conversation door tool reached off the HTTP gate — from a hook, a schedule, an agent’s tool call, or a detached run — fails loudly with the unavailable error kind instead of running, and is retried only where the tool’s retry policy opts in. A plugin route opts into the same refusal by declaring reload_gated on its registration; the route surface enforces it around the handler.

​
Soft restart

reload_config is the soft restart: it refreshes environment values and reloads the manifest registries in place. Combined with live config and manifest updates, it lets a server pick up configuration and capability changes without dropping its process. Across a fleet, tai fleet reload-config soft-restarts every worker on the bus (or only --target ones) and returns the per-worker report — see Manage a fleet. The soft restart carries every hot change in place without dropping a process. A recycle-class value changed in the stored env outside the pipeline (a hand edit of .env, a kubectl edit of the config Secret) is named by the reload: each worker’s report lists it under awaiting_respawn (its value differs from the one the worker bound — at boot, or when the plugin that declares it was loaded). The reload then rolls a recycle over the workers it addressed that name such a key, one at a time. Every other worker whose state the reload left unknown is asked once it is ready, and restarted unless its reply confirms it clean or states a refused build; the workers that confirmed clean keep running. A worker whose reload was refused at its epoch build discards the build and keeps serving its current config; its failed reply carries the payload {"status": "discarded", "awaiting_respawn": [...]}, the recycle-class keys standing off their bound value after the discard. When that list names keys, the roll withholds the worker’s restart, since the stored config did not build on it and a restart may fail the same way: the worker is left running and stays named in awaiting_respawn. The worker that ran the reload command is never asked: its own reload row is its statement, and when its own state is unknown it needs its own restart. It restarts itself last, after its reply, only when it needs its own restart and the roll converged. It logs an ERROR line naming every worker still awaiting a respawn after the roll, to be restarted through the deployment’s supervisor, and a separate ERROR line naming every withheld worker, whose stored config, or the cause its build error names, is fixed before any restart. The response’s recycle reports that roll and awaiting_respawn names the workers (name and keys) still awaiting a respawn after it — withheld workers included, and empty when no worker is left owing one. A reload run under a bound (a backend job, or a call under the turn budget) hands its roll to the worker it runs for: roll_deferred is true, roll_deferred_to names that worker, which carries the roll out on its own, and recycle carries no rows. The response of an env write or a profile apply carries the same fields after its own roll.

​
The kinds summary

A server reports which pluggable kinds it actually resolved at startup. Every kind — identity and accounts providers, monitoring, storage, file loading (files), backend, sandbox, channels, webhook verifiers, config, and Studio plugins — gets one status row, in three states:
  • active — a real plugin is registered and serving the kind.
  • default — nothing is configured, but a built-in fallback is serving in its place (the no-op monitoring recorder, or the file config provider). The kind works; it is just running on the built-in.
  • off — nothing is registered and the kind has no built-in fallback. This is a legal, reported state, not an error.
files is active when the server’s optional file-loading dependencies (the files extra) are installed, and off with the install line as its detail when they are not. The same rows surface three ways: printed as a [kinds] summary block in the server’s startup log (and again after any in-place reload), served at the authed GET /api/system/kinds route, and rendered in the Studio’s System kinds table. Read them from a terminal with:
One state carries an explicit heads-up: when monitoring is running on its default (the no-op recorder — no monitoring plugin installed), the startup summary logs a one-time warning that runs are not being recorded and observability dashboards will read zeros. It fires at most once per process — the point where “monitoring not configured” becomes distinguishable from “no traffic yet” — so a monitoring-less deployment is flagged without spamming the log on every reload. See the observe guide for watching a live server and the HTTP API reference and CLI reference for the management routes and commands.