Skip to main content
You operate a running server without a restart. Reloading a tool, replacing the manifest, re-probing a failed MCP server, or removing a capability all happen live, through the operations layer — one surface that projects to an HTTP route, a CLI command, and (curated) an MCP tool.

​
The operations layer

Every management capability is an operation — one decorated async function that carries its own metadata: a name, a summary, the typed errors it may raise, and flags such as destructive. An operation is the single source, and three surfaces project from it:
  • an HTTP route under /api/*, the door the CLI and the Studio drive;
  • a CLI command in the tai groups;
  • an MCP tool, registered through the normal tool binding so an extension wraps it, a preset bakes over it, and tai tools run or an agent dispatches it — subject to curation (below).
Because the three surfaces come from one declaration they cannot drift. The management operations cover the live-ops families:
  • Tools — re-register one app tool from its stored definition (reload_tool), drop a tool from the live registry (remove_tool), and run a registered tool by name (run_tool).
  • Config — soft-restart in place by refreshing the environment and reloading the manifest registries (reload_config), and apply a whole settings profile — replace the stored env with a saved band, swap in a fresh serving epoch, and recycle any pooled resource the change tore down (apply_profile).
  • Manifest — replace the whole persisted manifest, fleet-wide (update_manifest). The change is written through to durable config and then broadcast, so it survives a restart and reaches every worker.
  • MCP servers — list the servers a viability check skipped (list_failed_mcps), re-probe and re-attach one server or every failed one (reload_mcp, reload_failed_mcps), and detach a server’s tools by title (deregister_mcp).
  • Manifest sections — add or remove a single mcp, tools, or agents entry by title (an add optionally replacing a colliding title in place), and add or remove names on the api_tools include/exclude lists — each a persisted, hot-reloaded edit to one item, so changing one entry no longer means rewriting the whole section.
  • Tool extensions — attach or detach a single extension combo on a tool, leaving its other combos in place, alongside the whole-list write.
  • Access control — add or remove one scope on a key, and set or remove one tag grant on a role — an item-level edit that never rewrites the whole scope set or grant map.
Drive them from the CLI:

​
Curating the MCP surface

Which operations project as MCP tools is set by the manifest’s api_tools block. Two exclusion tiers guard the surface regardless of what a manifest asks for:
  • Tier 1 — never projectable. A meta-executor — an operation that runs any tool by name, run_tool — is a universal authorization bypass, so it is hardcode-blocked from the MCP surface even when named in api_tools.include. It stays a route, the tai tools run command, and an internal dispatch target.
  • Tier 2 — off by default, includable. An authority-changing operation — one that mints or scopes keys, edits policy, replaces the manifest (update_manifest), or restores unshipped state (a backup import) — is off the default MCP surface but projects when an api_tools.include names it explicitly.
A destructive operation carries a destructiveHint annotation on its projected tool (and an x-destructive marker on its route in the OpenAPI spec); it only projects when api_tools.expose_destructive is on.

​
Authorization on every MCP surface

AuthzMiddleware authorizes every projected-operation dispatch. It runs at tool call time — before the extension chain, so before any backend enqueue — on every MCP-serving instance: the main server and each sub-MCP mount. It resolves the dispatched tool to its base operation and, for a projected operation, runs the access-control check; a denial raises a loud tool error. A tool that is not a projected operation passes straight through — its authorization is a separate concern.

​
The mutation pipeline

A mutating operation runs one pipeline: it validates the change, persists it transactionally to durable config, reloads the calling process, then broadcasts it on the worker bus and awaits a per-worker outcome from every worker sharing the manifest. Persist-before-broadcast is why a mutation survives a restart and why the persisted state and the live state cannot drift apart through a supported path. The broadcast reaches every process on the bus — serving HTTP workers and any backend runtime alike — not just the one that received the request. Fan-out is the app’s own concern, carried by the bus; it is not a backend capability. Every mutation response embeds the fleet report, keyed per worker on its name, so a worker the change did not converge is named — departed, timed_out, missing, or one of the gap outcomes resyncing, recycling, and stale for a worker that was not in a ready, freshly-beating state — never a fake success. A failed local apply raises; a remote miss is reported and logged. Inspect the fleet and re-drive a reload with the tai fleet group. Because a mutation reaches every worker, the deployment shapes that run more than one process need the bus: the boot rules refuse a multi-worker server, a registered backend, or k8s mode without TAI_BUS_REDIS_URL. A settings-profile apply extends this pipeline at the reload step: instead of refreshing in place it builds a fresh serving epoch off to the side under the proposed env and swaps it in atomically, retiring the old one — a failed build is discarded with the old surface still serving. This same epoch build backs a plugin install or upgrade: any reload that rebuilds the serving surface re-imports each plugin package and its route-sibling modules and re-registers the declared routes, so a plugin’s HTTP surface — a router or middleware, a channel’s inbound webhook — comes live on the epoch swap of the reload that changed it rather than waiting for a process restart. Before it commits, an epoch route audit compares the rebuilt route set against the outgoing one and refuses to swap in an epoch that would silently drop a still-declared plugin’s routes; the prior epoch keeps serving and the failure is surfaced. For a changed key whose reload class is recycle, it then orchestrates a process recycle across the fleet, on the shapes that support one. Its report is per worker: each recycled row carries a status and the generation_before it was recycled from, alongside a per-kind list of the fresh lives that have since joined. Confirmation is by reality, not successor identity — a recycle is accepted on two facts: the targeted old lives are gone (their slots absent, expired, or already at a higher generation), and fresh ready capacity of each kind has booted under the new config, counted per kind to cover the recycled workers.

​
Reloading is a first-class state

While a reload is in flight, the routes that dispatch a run or mutate live registries return a retriable 503 carrying a reloading marker and a Retry-After header, rather than serving against a half-swapped registry. A client treats it as a transient condition and retries — the reload is a state the API describes, not an error.

​
Soft restart

reload_config is the soft restart: it refreshes environment values and reloads the manifest registries in place. Combined with live config and manifest updates, it lets a server pick up configuration and capability changes without dropping its process. Across a fleet, tai fleet reload-config soft-restarts every worker on the bus (or only --target ones) and returns the per-worker report — see Manage a fleet.

​
The kinds summary

A server reports which pluggable kinds it actually resolved at startup. Every kind — identity and accounts providers, monitoring, storage, backend, channels, webhook verifiers, config, and Studio plugins — gets one status row, in three states:
  • active — a real plugin is registered and serving the kind.
  • default — nothing is configured, but a built-in fallback is serving in its place (the no-op monitoring recorder, or the file config provider). The kind works; it is just running on the built-in.
  • off — nothing is registered and the kind has no built-in fallback. This is a legal, reported state, not an error.
The same rows surface three ways: printed as a [kinds] summary block in the server’s startup log (and again after any in-place reload), served at the authed GET /api/system/kinds route, and rendered in the Studio’s System kinds table. Read them from a terminal with:
One state carries an explicit heads-up: when monitoring is running on its default (the no-op recorder — no monitoring plugin installed), the startup summary logs a one-time warning that runs are not being recorded and observability dashboards will read zeros. It fires at most once per process — the point where “monitoring not configured” becomes distinguishable from “no traffic yet” — so a monitoring-less deployment is flagged without spamming the log on every reload. See the observe guide for watching a live server and the HTTP API reference and CLI reference for the management routes and commands.