Skip to main content
Upgrading is a tag change plus a migration run. A new image can carry pending schema migrations; a one-shot runner applies the chain before the app services start, and each container refuses to serve on an out-of-date database (the boot gate). The image carries no state, so the data volumes are never touched — only the app containers are replaced.

Upgrade the Compose bundle

1

Snapshot the database first

A migration rolls forward only — there are no down-migrations. Before an upgrade that carries schema changes, take a Postgres snapshot (a managed provider’s point-in-time restore, or pg_dump) so a bad release is a restore away. Recovery is restore-then-re-migrate, never a rollback.
2

Pin the new tag

Set TAI_VERSION in .env to the release you are moving to.
TAI_VERSION in .env
3

Pull it

4

Stop the app services

The app containers stop and their volumes stay; Postgres and Redis keep running for the steps below. A server of the old image loads its plugins through the plugin environments the next step rebuilds, so none may run while they are rebuilt.
5

Run the migrations

The one-shot db-migrate service runs tai db migrate, which applies every pending migration across every component — the core chain, each installed plugin that owns one, and each plugin the manifest loads — in order, recording each in the tai_schema_history table. It is idempotent: a run with nothing pending reports so and exits 0. Preview a run first with docker compose run --rm db-migrate db migrate --plan, which lists the pending files without touching the database.
6

Rebuild the plugin environments

A deployment with plugins installed through the marketplace rebuilds their environments against the new image while no app service runs; a server refuses to start on environments that no longer match the image. See Rebuild the plugin environments.
7

Start the app services on the new image

config-init hands config/ to the image’s user, as a full up does before the app services start. --no-deps then recreates the three stopped app services against the new image without restarting Postgres or Redis, so the datastores — and their volumes — stay up.
Read the release notes before pinning. A release that needs a step beyond the migration run — a data backfill, a manual expand/contract cutover — names it.

Rolling upgrades across a skeleton major

A release can move tai42-skeleton to a new major version. When old and new nodes overlap during the upgrade — a rolling update that replaces one node at a time, or hosts upgraded in turn — the two majors run side by side on one Redis and one TAI_BUS_NAMESPACE until the old nodes drain. Every bus key and channel sits under {namespace}:v{major} (see Namespacing), so each major has a bus of its own and the two are separate fleets. To see which skeleton major an image carries, read the constraints file it ships:
It prints one tai42-skeleton==<version> line; the first number of the version is the major. A skeleton before 34 has no major in that root: its keys are {namespace}:bus:* and {namespace}:env-readers:*. An upgrade from it to 34 or later runs two separate fleets too, since the two layouts share no key. While both run, tai fleet workers calls GET /api/fleet/workers on whichever server answers, so it lists the fleet of that server’s major. Behind a load balancer that spreads requests across old and new nodes, the listing can show either fleet from one call to the next, and both can show a serve-1. A config change is stored for both majors, but its live reload, its fleet report, and every other bus op reach only the major that sent it. The database, the config store, and the job queues stay shared — see Two skeleton majors on one namespace for when the other major takes a change up and what else is shared. A job on a shared queue may be picked up by a backend worker of either major, and the release notes name any step a major needs beyond the migration run. Plugins installed through the marketplace do not overlap this way. Every node loads them from the one plugins directory, and tai plugins rebuild switches that directory’s environments to builds for the new image. Stop every node of the old image before the rebuild, and start nodes of the new image after it.

Move the Langfuse stack to 4.x

The compose/langfuse/ stack runs Langfuse 4.x, and the monitoring plugin (tai42-monitoring-langfuse 6.0.0) needs a 4.x server. The move has no migration variables: it replaces the stack’s data.
1

Stop the stack and remove its volumes

This removes langfuse-postgres-data, langfuse-clickhouse-data, langfuse-clickhouse-logs, langfuse-minio-data and langfuse-redis-data (prefixed tai-langfuse_ under the default project name, or the -p name you use).
2

Pull the 4.x images

3

Start the stack

Data recorded under 3.x is not carried over. The LANGFUSE_INIT_* values re-create the same organisation, project and keys, so config/.env and the collector’s credential need no change.

Recreate the states schema

A release can rewrite the states baseline in place. On an existing deployment the boot gate then refuses to start with:
and tai db migrate refuses too:
The step runs on the database the states component is bound to (TAI_DB_BINDING_STATES, the default database unless set; in the Compose bundle the postgres service, user and database ${POSTGRES_USER:-tai} / ${POSTGRES_DB:-tai}). It drops every states table: state_outbox, state_writes, state_applied_ops, state_subject_aliases, state_records, state_attachments, state_templates and state_declarations, the pending saves among them, and the catalog version sequence state_catalog_versions. Their data is lost.
1

Stop the app services

Compose prints each container as it stops.
2

Drop the states schema and its history rows

psql prints BEGIN, DROP TABLE, DROP SEQUENCE, DELETE 1 and COMMIT. On any error psql stops, the transaction rolls back and nothing is dropped.
3

Apply the baseline

It prints Applied states 0001_baseline. and Applied 1 migration(s).
4

Start the services

The gate passes and the services start.

The boot gate

Each serve and backend container checks, at startup and on every reload, that every schema chain it depends on is fully applied: the core skeleton and states chains, and the chain of every plugin the manifest loads, however that plugin is installed, editable installs included. tai setup --recover runs the same check before it imports any manifest module. A core chain, or a plugin chain under the plugin’s own name, whose component database is not configured is not checked. A plugin chain on a declared component binding is always checked: an unconfigured bound database refuses boot and names the missing setting. A pending migration or a rewritten (checksum-mismatched) chain is a loud refusal, not a half-migrated serve:
The fix is the message: run the migration step above, then let the container restart. If you recreate the app services before running db-migrate, they refuse to boot with exactly this message — run the migrations and they come up. Running db-migrate before the app services (the step order above) avoids the refusal entirely. A checksum mismatch on the states chain is fixed by recreating the states schema. A reload the gate refuses answers 500 with this message as its error, and the server keeps serving its current configuration. tai setup --recover prints it as Error: … and exits non-zero.

Runtime-installed plugins

An image upgrade replaces the core only. Plugins added at runtime through the marketplace live in the plugins directory (TAI_PLUGINS_DIR), not in the image, so they do not update with a new image tag — a freshly pinned image boots with the same plugin versions the install store records. The directory holds one environment per kind of plugin: the plugin layer, shared by every in-process plugin, and one virtual environment per mcp-server package.

Rebuild the plugin environments

Those environments were built against the previous image. Rebuild them against the new one after the migrations, with every app service stopped and before any starts on the new image:
tai plugins rebuild runs with the deployment’s environment and manifest, as tai serve does, so a setting kept only in the env store applies to it. It holds the fleet-wide lock the installer holds, checks every environment against the install store and the running image, rebuilds each one that does not match (fetching a GitHub-sourced plugin’s archive again when the directory lost it), deletes the other builds and the server environments no plugin is recorded in, and prints one row per environment: its environment (layer, or server <package>), its result — up to date, rebuilt, or removed — and a detail, which for a rebuilt environment is the check’s finding. When it rebuilt a plugin layer that holds plugins, it ends with running servers load the rebuilt plugin layer on their next start. It switches each rebuilt environment’s current link to the new build and deletes the old builds, and a running server loads its plugins through current, so a server of the old image left running would load the new image’s builds: stop every server first, as the steps above do. A server loads the rebuilt environments on its next start. It needs the database settings, TAI_PLUGINS_DIR, pip, and the package index, and a failure prints Error: … (with pip’s report on a failed build) and exits 1.

The plugin check at boot

Each serve and backend process checks, once at startup and after the schema check, that the plugins directory matches the install store and the running image. It only reads. A mismatch is a loud refusal that names the fix:
The same check refuses when a plugin in the layer provides a package the image already ships at another version (plugin environment conflicts with this image), when a plugin’s own requirements are not met by the image (plugin environment is inconsistent with this image), and when a server package’s environment has no interpreter or records no Python version. Each of these ends Run 'tai plugins rebuild' against this image. It also refuses when the directory does not match the install store (plugin environment is out of date: the plugins directory <dir> does not match the install store (…), naming each plugin it is missing or holds unrecorded), and when an environment’s current build has no readable build record. Every build carries its record, tai42-build.json, written last: the plugins it was built from, the plugins it left out and why, the Python version and the platform. A current build without one is incomplete, and a record that is not the document a build writes does not read:
These end Run 'tai plugins rebuild' against this deployment, then start again. A server package’s environment reads plugin environment of server package <package> in place of plugin environment. When packaged plugins are recorded but TAI_PLUGINS_DIR is unset, the refusal reads installed plugins need TAI_PLUGINS_DIR and names them: set it to the directory they were installed into. With no skeleton database configured there is no install store, so the check is skipped and logged. tai doctor reports the same check per environment (plugins: layer, plugins: server <package>).

Plugins the new core does not support

A plugin whose declared tai42-contract range does not cover the new image’s contract is left out of the rebuilt layer and recorded in the build record as excluded, with the reason: the rebuild lists it as left out <ref> <version>: <reason>, an install or update that rebuilds the layer adds the same line to its notes, the installed inventory shows it as incompatible, and tai plugins upgrade --all moves it to a version that supports the new core, or reports that none is published. While the manifest still names one of its modules, as an install leaves it, the server refuses to start (the Warning below): remove the plugin with tai plugins uninstall <ref> --offline, or drop its entries from the manifest, start the server, then install a version that fits. A layer that still holds such a plugin, or that leaves out one the image now supports, is out of date to the boot check, and tai plugins rebuild puts it right.

Update the plugins

Update them as a step of their own, after the image upgrade:
This resolves each installed plugin to the newest version compatible with the core you are now running and builds it into the plugins directory. A router or middleware plugin — and a channel’s inbound webhook route — comes live on the next reload / config-apply, not on a container restart: the epoch rebuild that a reload runs re-imports the plugin package and its route-sibling modules and re-registers the declared routes, so an upgraded router or middleware is serving once that reload lands. The image pins its own core versions as a pip/uv constraints file at /opt/tai/constraints.txt and sets PIP_CONSTRAINT/UV_CONSTRAINT to it, so any install — a marketplace install, python -m pip install by hand, or a derived image’s build step — that would move a core package (tai42-contract, tai42-kit, tai42-skeleton, tai42-cli, or any other package the image ships) off the image’s version is refused by the resolver, never silently downgraded or upgraded. The marketplace builds the plugin layer against the same file and never lets a plugin replace a package the image provides; it refuses such an install with a 409 that carries pip’s report. A hand install fails loudly the same way:

A plugin that blocks the upgrade

A plugin whose own dependency pin rejects the new image cannot be rebuilt: tai plugins rebuild stops with pip’s report, the server refuses to start, and the running-server uninstall cannot be reached. Remove the plugin from the host instead:
tai plugins uninstall <ref> --offline runs with no server, on the deployment host, with the deployment’s environment (the same variables tai serve runs with, TAI_PLUGINS_DIR included) and the server package installed beside the CLI, as it is in the image. It holds the same fleet-wide lock the installer and tai plugins rebuild hold. It changes only the environment the plugin lives in: for an in-process plugin it rebuilds the plugin layer from the other installed plugins, and for an mcp-server package it withdraws and deletes that package’s own environment. It then strips the plugin’s entries from the manifest through the same validated config write the running server uses and removes its row from the install store. A server that is running keeps what it loaded until its next start. Every other server package’s environment is left as it was, so after an image upgrade it can still be built for the previous image. Run the rebuild again before you start the app services, so it brings those environments up to date and the boot check passes:
Then start the app services and install a version of the removed plugin that fits the new image. The venv is root-owned, so a direct install into it runs as root — a derived image’s USER root build step, or docker exec -u 0 into a running container. The marketplace builds under TAI_PLUGINS_DIR and is unaffected. After a manual or derived-image install, run python -m pip check to confirm the environment is consistent.
A manifest-declared plugin that is incompatible with the new core — or otherwise cannot load — aborts boot loudly: the process exits naming the module, its kind, and the reason, rather than coming up missing a plugin the manifest asked for. Fix or update the plugin, point the manifest at a working one, or remove it — from the manifest, or with tai plugins uninstall <ref> --offline — then boot again. A plugin that cannot load during a live config reload aborts that reload instead: the previous generation keeps serving and the reload call returns the failure (a non-2xx naming the module, which tai config reload prints before exiting non-zero). Keep the plugins directory on persistent storage so the installed set survives the container recreation an upgrade performs.

See also