Config and secrets

​

The provider seam

from tai42_skeleton.config import ConfigManagerFactory

manager = ConfigManagerFactory.create()

​

Secrets are referenced, not embedded

• A scalar value written !ENV ${VAR} is the convention wherever a secret rides as a plain string leaf — a manifest value, a Studio secret field, and a preset’s baked fixed_kwargs. The runtime keeps the marker verbatim and resolves it from the environment at use time; the !ENV ${VAR:default} form falls back to default when the variable is absent. The Studio SecretRefField and MCP-tab secret flow write this marker for a pasted secret automatically (see Settings profiles).
• A secret_env field carries the env var name where the binding is a config object rather than a scalar — a hook or webhook verifier binding names secret_env, and an OAuth connector provider a client_secret_env.

​

How a config change is written

• Transactional writes. Every config write runs inside the provider’s exclusive_write() span, held from the first read through the persist. A provider must block until the write is exclusive across every process sharing the store and within the holding process itself, so two spans over one store are never held at once. It must release the span when the holding process dies, and must raise rather than let a write proceed unserialized. A release the provider cannot complete raises from the span’s exit; the writes made inside the span stand, the write’s reload and broadcast still run, and the store may still hold the span until the holding process dies — the response’s span_release_error and an ERROR log carry the provider’s error. The span is entered and exited on worker threads, and the exiting thread may differ from the entering one, so a provider must not bind the span to the thread that entered it. The file provider takes a POSIX record lock on .config-write.lock (mode 0600) in the config directory, plus a process-local lock, since a record lock does not exclude its own process; it is not supported on Windows.
• Comment preservation. A manifest edit keeps the surrounding comments, key order, and formatting of the untouched document. An automated edit does not reflow a file a human maintains by hand.

​

Deployment values and stored values

​

Settings profiles

tai config profile diff production
tai config profile apply production

• hot — re-read in place on the next serving generation. The common case; no resource is torn down.
• recycle — the pooled resource behind the key (a Redis or broker connection, the bus) must be torn down and rebuilt, which needs the supervisor that starts the affected process again. Every env write rolls that recycle across the fleet (see How a recycle runs below). A recycle-class value changed in the stored env outside these writes (a hand edit of .env, a kubectl edit of the config Secret) is named by the next reload — each worker’s awaiting_respawn — and that reload (a config reload, or the fleet soft restart tai fleet reload-config) rolls the same recycle over the workers that name it, all but those whose restart it withholds (see How a recycle runs below). A worker compares each recycle-class value with the one it bound — at boot, or when the plugin that declares the setting was loaded — so a plugin loaded after boot with its value already in place awaits nothing. A stored recycle-class value a worker withdraws when another worker’s deployment turns out to set the key is rolled as a recycle by the worker that joined (see How every process learns what the deployment owns above). The response of an env write or a profile apply names under awaiting_respawn the workers (name and keys) still awaiting a respawn after its roll — withheld workers included — empty when no worker is left owing one, and when the roll was handed off (roll_deferred, below).
• excluded — deployment / boot identity: the config mode, the transport, the manifest path, the readiness sentinel, the backend liveness settings (TAI_BACKEND_LIVENESS_PATH, TAI_BACKEND_LIVENESS_INTERVAL, TAI_BACKEND_LIVENESS_GRACE), the launcher’s server options (bind, workers, graceful-shutdown timeout). These are set by the launcher and have no live-reload path, so a profile that carries one is refused — at save and at apply — naming the key. The refusal reads the profile’s declared keys, never the effective env, so the applier legitimately carrying the whole excluded band across the replace is not itself rejected. A stored env that carries an excluded-band key is refused at boot and at reload, naming the key.

• The reload’s report decides whom the roll restarts. A worker that reports a recycle-class key off the value it bound is restarted; a worker other than the writer whose state the report leaves unknown (its reload did not confirm, it has no row, its reply carries no awaiting_respawn list, or the bus was unreachable) is asked once it is ready, and restarted unless its reply confirms it clean or states a refused build; a worker that confirmed clean is left running. A worker whose reload was refused at its epoch build (the build discarded before its swap) keeps serving its current config, and its failed reply carries {"status": "discarded", "awaiting_respawn": [...]}; when it reports keys, its restart is withheld, since the stored config did not build on it and a restart may fail the same way. It is left running, never asked, named in awaiting_respawn, and its own ERROR line says to fix the stored config, or the cause the build error names, before restarting it. A discard that itself failed is no refused build: it leaves the worker’s state unknown. A worker asked once ready that states a refused build with keys is a withheld row of the roll, which does not stop the roll. The writer, the worker that took the change, is never asked at its own roll: its own reload row is its statement, and when its own state is unknown it needs its own restart. A reload that names its targets decides over those targets only. The roll restarts the workers one at a time: for each it waits the drain the worker announces on its census row as its stop begins (its runtime’s stop phase plus the app’s shutdown bounds, see What a leaving worker announces), plus the start-and-rejoin step (TAI_TOOL_RUNS_SHUTDOWN_DRAIN_SECONDS).
• The recycle rebuilds the affected pooled resource, and the response of every env write — a profile apply, tai config env set, a backup restore, a plugin install — carries the recycle rows naming which workers recycled. The roll runs inside the change, unless the change hands it off (below), so the response answers once the roll ends. The worker that took the write, only when it needs its own restart and the roll converged, recycles itself after replying (its self-deferred row in the recycle rows), through whichever door the write came, when that door runs under no bound: the HTTP response, or a tool call (MCP tools/call, tai tools run, an agent, a preset, a hook run in that worker’s process). The exit is graceful, so the reply finishes before the worker stops and its supervisor starts it again. A background tool run (POST /api/tool-runs) runs on the serve worker that took it, so that serve worker recycles once the run has ended. A change under a bound hands its roll off instead, its writer’s restart included (below).
• A change made under a bound fixed before it started hands its roll off instead, since one worker’s drain alone can outlast the bound. Two bounds count: a backend job’s budget (every tool a tai backend worker runs as a job: a tool queued on the backend with an async_task branch or run there with a sync_task branch, a schedule fire, a hook dispatched through the backend, a callback’s tool), and the turn budget of a live caller’s call (MCP tools/call, tai tools run, an agent run, a conversation turn, a preset run through any of these) when TAI_TURN_TIMEOUT_SECONDS is set. Such a change hands its roll to the worker it runs for — the tai backend worker that ran the job, or the worker serving the call — and answers at once: roll_deferred is true, roll_deferred_to names that worker, recycle carries no rows and awaiting_respawn is empty. That worker carries the roll out on its own, outside every bound: it waits each worker’s drain and step as above and, when it needs its own restart, restarts itself last, once the roll converged. The roll’s outcome is in that worker’s log, and the restarts show on tai fleet workers as generation bumps. A hand-off that fails fails the change loudly, naming the worker: the change is persisted and reloaded, and the error says the roll did not start (the worker refused it, or is already leaving) or, when no answer came, that whether it started is unknown. A change under a bound hands off even when the only restart is that worker’s own, so its exit never waits inside the bound. Every response that reports a roll carries the two fields; roll_deferred is false and roll_deferred_to null when the roll ran inside the change.
• A worker joining the bus starts the same roll when a join withdraws a stored recycle-class value the workers still run on: it restarts the workers that reported such a key, leaves running those whose reload was refused at their epoch build (their restart withheld), and asks the workers whose state the announcement left unknown once they are ready, one at a time. It recycles itself last when its own withdraw left one and the roll converged. A roll that stops is logged as an error with its partial report, and the workers it did not reach are named as awaiting a respawn.
• A change that sets a key the deployment’s environment sets is refused, as described in Deployment values and stored values.
• A change to the bus URL is refused on every deployment — the fleet census rides the bus, so recycling the bus out from under a live apply is intrinsically unobservable. Removing a stored copy of a bus URL the deployment sets is no change, so it is allowed.

​

The plugin model

tai config mode
tai config env set KEY=value

​

Model providers

​

Named databases

​

Admin identity

​

Component bindings

​

Default connection namespace

1. the value written to the config store (manifest env / init);
2. the field’s own environment variable;
3. the field’s own .env entry;
4. a file secret for the field (secrets_dir);
5. the matching TAI_DEFAULT_* environment variable;
6. the matching TAI_DEFAULT_* .env entry;
7. the class-level default.

​

No hidden defaults: absent means OFF

​

What OFF looks like

• Reads answer empty. A collection read returns 200 with an empty list; a read of a named entity returns 404, byte-identical to a genuine miss.
• Writes refuse with a named reason. A mutation returns 501 with the JSON body {"error": …, "code": "<feature>-not-configured"} — the code is machine-readable so a caller can branch on it, and the message names the env var that turns the feature on.
• Readiness skips it. /ready omits an unconfigured feature entirely: an OFF feature never fails a readiness probe and is never probed.
• Introspection labels it. GET /api/system/kinds lists the feature with state: "off" and a detail naming the variable to set; the boot summary shows the same row.
• Boot logs one line. Startup emits a single INFO line per gated feature naming what is off and which env prefix turns it on — no traceback, no repetition.
• Public doors stay uniform. An unauthenticated door keeps answering with its own standard 404, so an OFF feature opens no new oracle.

​

OFF behavior per feature

​

Local-dev recipe

TAI_DEFAULT_REDIS_URL=redis://localhost:6379/0
TAI_DATABASE_DEFAULT_PG_HOST=localhost
TAI_DATABASE_DEFAULT_PG_PORT=5432
TAI_DATABASE_DEFAULT_PG_DB=postgres
TAI_DATABASE_DEFAULT_PG_USER=postgres
TAI_DATABASE_DEFAULT_PG_PASSWORD=postgres

​

Settings reference

​

Core environment

​

Worker bus

​

Connector token encryption

​

Rotating CONNECTORS_KEK

1. Carry the old key. Set CONNECTORS_KEK_PREVIOUS to the current key’s value and set CONNECTORS_KEK to the new key. Every stored blob still decrypts (via the ring), and every new/refreshed token is written under the new key.
2. Converge the store. Run tai connectors reencrypt-tokens (route POST /api/connectors/tokens/reencrypt). It scans every stored blob — expired sessions included — re-encrypts each one still under a previous key with the current key, and reports {scanned, reencrypted, skipped, failed, failed_connection_ids, cas_retries}. It is idempotent (a blob already under the current key is skipped) and compare-and-set safe (a concurrent token refresh is never clobbered). A blob no configured key can open is counted in failed and named — never silently dropped.
3. Retire the old key. Once the sweep reports failed: 0, clear CONNECTORS_KEK_PREVIOUS. The old key is no longer needed and its exposure window closes.

​

Proxy pool (tai42-toolbox)

​

Timeouts and limits

​

Connection timeouts and retries

• Postgres. _PG_RETRY_ATTEMPTS re-attempts establishing a connection when a pool is built: after a failed connect it waits and tries again, with an exponential backoff that starts at 16 ms and doubles to a 512 ms cap. A statement is never re-run. With n retries, building a pool waits at most (n + 1) × _PG_CONNECT_TIMEOUT plus the backoff.
• Redis. <PREFIX>RETRY_ATTEMPTS re-tries a failed connect and the command, writes included, with the same backoff. A subscription whose connection drops is reconnected and re-subscribed inside the same retry, so a drop shorter than the retry raises no error to the subscriber; a message published while it was disconnected is not delivered, as Redis pub/sub keeps no backlog. So each subscriber catches up after such a reconnect: the celery Redis result backend reads the stored state of every task it still awaits, so a result written meanwhile still arrives; the worker bus runs its resync (presence resyncing, the resync, then ready); and a turn waiting on a conversation thread retries the thread at once. A fleet operation’s replies have no stored state, so when the publisher’s reply subscription reconnects this way it logs a WARNING, keeps collecting the replies sent after the reconnect, and every worker left without a final reply carries the cause in its verdict (reply collection transport error: the reply subscription reconnected …). When the celery result backend’s retry runs out, the wait raises the connection error, and the next wait, on any task, connects again, re-subscribes every task still awaited and reads their stored state. <PREFIX>RETRY_ON_TIMEOUT alone (with RETRY_ATTEMPTS at 0) re-tries a timed-out command once, at once.
• Every retry is logged. Each retried failure logs one WARNING naming the connection — Redis connection <SETTING> failed (attempt 1 of 3): …; retrying in … or Postgres connection '<pool>' failed (attempt 1 of 3): … — and when the attempts run out the last error is raised; a Postgres one carries the note Postgres connection '<pool>' failed after 3 attempt(s). With the defaults (0) a connection makes one attempt and fails at once.
• A connection with its own connection string takes its retry and timeouts from its own namespace, never from the base one: the model-provider store and checkpointer (LLM_PROVIDER_STORE_* / LLM_PROVIDER_CHECKPOINT_*: RETRY_ATTEMPTS, RETRY_ON_TIMEOUT, SOCKET_TIMEOUT, SOCKET_CONNECT_TIMEOUT, PG_RETRY_ATTEMPTS), the celery Redis result backend (CELERY_RESULT_BACKEND_*) and the celery Redis broker (CELERY_BROKER_*, read and write sides alike). With them unset, that connection makes one attempt. The celery broker never retries a read kombu polls: a worker’s task and fanout reads, and a control command’s reply read in any process. A drop there raises at once: a worker’s consumer restart reconnects, and a control call fails with the connection error, so a retry of that call reconnects.