Upgrade the Compose bundle
1
Snapshot the database first
A migration rolls forward only — there are no down-migrations. Before an
upgrade that carries schema changes, take a Postgres snapshot (a managed
provider’s point-in-time restore, or
pg_dump) so a bad release is a restore
away. Recovery is restore-then-re-migrate, never a rollback.2
Pin the new tag
Set
TAI_VERSION in .env to the release you are moving to.TAI_VERSION in .env
3
Pull it
4
Stop the app services
5
Run the migrations
db-migrate service runs tai db migrate, which applies every
pending migration across every component — the core chain, each installed
plugin that owns one, and each plugin the manifest loads — in order, recording each in the tai_schema_history
table. It is idempotent: a run with nothing pending reports so and exits 0.
Preview a run first with docker compose run --rm db-migrate db migrate --plan,
which lists the pending files without touching the database.6
Rebuild the plugin environments
7
Start the app services on the new image
config-init hands config/ to the image’s user, as a full up does before
the app services start. --no-deps then recreates the three stopped app services
against the new image without restarting Postgres or Redis, so the datastores
— and their volumes — stay up.Read the release notes before pinning. A release that needs a step beyond the
migration run — a data backfill, a manual expand/contract cutover — names it.
Rolling upgrades across a skeleton major
A release can movetai42-skeleton to a new major version. When old and new nodes
overlap during the upgrade — a rolling update that replaces one node at a time, or
hosts upgraded in turn — the two majors run side by side on one Redis and one
TAI_BUS_NAMESPACE until the old nodes drain. Every bus key and channel sits under
{namespace}:v{major} (see Namespacing), so each
major has a bus of its own and the two are separate fleets.
To see which skeleton major an image carries, read the constraints file it ships:
tai42-skeleton==<version> line; the first number of the version is
the major.
A skeleton before 34 has no major in that root: its keys are {namespace}:bus:* and
{namespace}:env-readers:*. An upgrade from it to 34 or later runs two separate
fleets too, since the two layouts share no key.
While both run, tai fleet workers calls GET /api/fleet/workers on whichever server
answers, so it lists the fleet of that server’s major. Behind a load balancer that
spreads requests across old and new nodes, the listing can show either fleet from one
call to the next, and both can show a serve-1.
A config change is stored for both majors, but its live reload, its fleet report, and
every other bus op reach only the major that sent it. The database, the config store,
and the job queues stay shared — see
Two skeleton majors on one namespace for when the other major
takes a change up and what else is shared. A job on a shared queue may be picked up by
a backend worker of either major, and the release notes name any step a major needs
beyond the migration run.
Plugins installed through the marketplace do not overlap this way. Every node loads
them from the one plugins directory, and
tai plugins rebuild switches that directory’s
environments to builds for the new image. Stop every node of the old image before
the rebuild, and start nodes of the new image after it.
Move the Langfuse stack to 4.x
Thecompose/langfuse/ stack runs Langfuse 4.x, and the monitoring plugin
(tai42-monitoring-langfuse 6.0.0) needs a 4.x server. The move has no migration
variables: it replaces the stack’s data.
1
Stop the stack and remove its volumes
langfuse-postgres-data, langfuse-clickhouse-data,
langfuse-clickhouse-logs, langfuse-minio-data and langfuse-redis-data
(prefixed tai-langfuse_ under the default project name, or the -p name you use).2
Pull the 4.x images
3
Start the stack
LANGFUSE_INIT_* values re-create
the same organisation, project and keys, so config/.env and the collector’s
credential need no change.
Recreate the states schema
A release can rewrite the states baseline in place. On an existing deployment the boot gate then refuses to start with:tai db migrate refuses too:
states component is bound to
(TAI_DB_BINDING_STATES, the default database unless set; in the Compose bundle the
postgres service, user and database ${POSTGRES_USER:-tai} / ${POSTGRES_DB:-tai}).
It drops every states table: state_outbox, state_writes, state_applied_ops,
state_subject_aliases, state_records, state_attachments, state_templates and
state_declarations, the pending saves among them, and the catalog version sequence
state_catalog_versions. Their data is lost.
1
Stop the app services
2
Drop the states schema and its history rows
BEGIN, DROP TABLE, DROP SEQUENCE, DELETE 1 and COMMIT. On any
error psql stops, the transaction rolls back and nothing is dropped.3
Apply the baseline
Applied states 0001_baseline. and Applied 1 migration(s).4
Start the services
The boot gate
Eachserve and backend container checks, at startup and on every reload, that
every schema chain it depends on is fully applied: the core skeleton and states
chains, and the chain of every plugin the manifest loads, however that plugin is
installed, editable installs included. tai setup --recover runs the same check
before it imports any manifest module. A core chain, or a plugin chain under the
plugin’s own name, whose component database is not configured is not checked. A plugin
chain on a declared component binding
is always checked: an unconfigured bound database refuses boot and names the missing
setting. A pending migration or a rewritten (checksum-mismatched) chain is a loud
refusal, not a half-migrated serve:
db-migrate, they refuse
to boot with exactly this message — run the migrations and they come up. Running
db-migrate before the app services (the step order above) avoids the refusal
entirely. A checksum mismatch on the states chain is fixed by
recreating the states schema.
A reload the gate refuses answers 500 with this message as its error, and the
server keeps serving its current configuration. tai setup --recover prints it as
Error: … and exits non-zero.
Runtime-installed plugins
An image upgrade replaces the core only. Plugins added at runtime through the marketplace live in the plugins directory (TAI_PLUGINS_DIR), not in the image, so they do not update with a new image
tag — a freshly pinned image boots with the same plugin versions the install
store records. The directory holds one environment per kind of plugin: the
plugin layer, shared by every in-process plugin, and one virtual environment
per mcp-server package.
Rebuild the plugin environments
Those environments were built against the previous image. Rebuild them against the new one after the migrations, with every app service stopped and before any starts on the new image:tai plugins rebuild runs with the deployment’s environment and manifest, as
tai serve does, so a setting kept only in the env store applies to it. It holds
the fleet-wide lock the installer holds, checks every environment against the
install store and the running image, rebuilds each one that does not match
(fetching a GitHub-sourced plugin’s archive again when the directory lost it),
deletes the other builds and the server environments no plugin is recorded in, and prints
one row per environment: its environment (layer, or server <package>), its
result — up to date, rebuilt, or removed — and a detail, which for a rebuilt
environment is the check’s finding. When it rebuilt a plugin layer that holds plugins,
it ends with running servers load the rebuilt plugin layer on their next start. It switches each
rebuilt environment’s current link to the new build and deletes the old builds, and
a running server loads its plugins through current, so a server of the old image
left running would load the new image’s builds: stop every server first, as the
steps above do. A server loads the rebuilt environments on its next start. It needs
the database settings,
TAI_PLUGINS_DIR, pip, and the package index, and a failure prints Error: …
(with pip’s report on a failed build) and exits 1.
The plugin check at boot
Eachserve and backend process checks, once at startup and after the
schema check, that the plugins directory matches the install store
and the running image. It only reads. A mismatch is a loud refusal that names the
fix:
plugin environment conflicts with this image),
when a plugin’s own requirements are not met by the image (plugin environment is inconsistent with this image), and when a server package’s environment has no
interpreter or records no Python version. Each of these ends Run 'tai plugins rebuild' against this image.
It also refuses when the directory does not match the install store (plugin environment is out of date: the plugins directory <dir> does not match the install store (…), naming each plugin it is missing or holds unrecorded), and when an
environment’s current build has no readable build record. Every build carries its
record, tai42-build.json, written last: the plugins it was built from, the plugins
it left out and why, the Python version and the platform. A current build without
one is incomplete, and a record that is not the document a build writes does not
read:
Run 'tai plugins rebuild' against this deployment, then start again. A
server package’s environment reads plugin environment of server package <package>
in place of plugin environment. When packaged plugins are recorded but
TAI_PLUGINS_DIR is unset, the refusal reads installed plugins need TAI_PLUGINS_DIR and names them: set it to the directory they were installed into.
With no skeleton database configured there is no install store, so the check is
skipped and logged. tai doctor reports the same check per environment (plugins: layer, plugins: server <package>).
Plugins the new core does not support
A plugin whose declared tai42-contract range does not cover the new image’s contract is left out of the rebuilt layer and recorded in the build record as excluded, with the reason: the rebuild lists it asleft out <ref> <version>: <reason>, an
install or update that rebuilds the layer adds the same line to its notes, the
installed inventory shows it as incompatible, and
tai plugins upgrade --all moves it to a version that supports the new core, or
reports that none is published. While the manifest still names one of its modules,
as an install leaves it, the server refuses to start (the Warning below): remove
the plugin with tai plugins uninstall <ref> --offline, or drop its entries from
the manifest, start the server, then install a version that fits. A layer that
still holds such a plugin, or that leaves out one the image now supports, is out
of date to the boot check, and tai plugins rebuild puts it right.
Update the plugins
Update them as a step of their own, after the image upgrade:/opt/tai/constraints.txt and sets PIP_CONSTRAINT/UV_CONSTRAINT to it, so any
install — a marketplace install, python -m pip install by hand, or a derived
image’s build step — that would move a core package (tai42-contract, tai42-kit,
tai42-skeleton, tai42-cli, or any other package the image ships) off the image’s
version is refused by the resolver, never silently downgraded or upgraded. The
marketplace builds the plugin layer against the same file and never lets a plugin
replace a package the image provides; it refuses such an install with a 409 that
carries pip’s report. A hand install fails loudly the same way:
A plugin that blocks the upgrade
A plugin whose own dependency pin rejects the new image cannot be rebuilt:tai plugins rebuild stops with pip’s report, the server refuses to start, and the
running-server uninstall cannot be reached. Remove the plugin from the host
instead:
tai plugins uninstall <ref> --offline runs with no server, on the deployment
host, with the deployment’s environment (the same variables tai serve runs
with, TAI_PLUGINS_DIR included) and the server package installed beside the
CLI, as it is in the image. It holds the same fleet-wide lock the installer and
tai plugins rebuild hold. It changes only the environment the plugin lives in:
for an in-process plugin it rebuilds the plugin layer from the other installed
plugins, and for an mcp-server package it withdraws and deletes that package’s
own environment. It then strips the plugin’s entries from the manifest through the
same validated config write the running server uses and removes its row from the
install store. A server that is running keeps what it loaded until its next start.
Every other server package’s environment is left as it was, so after an image
upgrade it can still be built for the previous image. Run the rebuild again before
you start the app services, so it brings those environments up to date and the
boot check passes:
USER root build step, or docker exec -u 0 into a running container. The
marketplace builds under TAI_PLUGINS_DIR and is unaffected. After a manual or
derived-image install, run python -m pip check to confirm the environment is
consistent.
See also
- Verify artifacts — confirm the new tag before you run it.
- Back up and restore — capture state before an upgrade.

