Skip to main content
Backend plugin · listing tai42/backend-arq

​
Install

​
Permissions

​
Provides

​
arq

Backend — arq execution backend — background tool runs and schedules over Redis. The arq backend runs queued tool calls, schedules, and agent turns over a single Redis — no separate broker. It is the default the distribution bundle selects.

​
Enable it

manifest.yml
A config that registers a task backend also needs the worker bus and a shared hooks registry: tai serve, tai backend, a config reload and a config write all refuse it unless TAI_BUS_REDIS_URL and HOOKS_REDIS_URL are set (each falls back to TAI_DEFAULT_REDIS_URL). The backend process matches the events it raises against its own hooks registry, so an in-memory one would run none of the hooks registered through the server.

​
Configuration

The ARQ_ env group. ARQ_REDIS_URL and ARQ_REDIS_MAX_CONNECTIONS fall back to the shared TAI_DEFAULT_REDIS_URL / TAI_DEFAULT_REDIS_MAX_CONNECTIONS namespace when unset; a set ARQ_ value always wins.

​
Run the worker

The worker and the enqueue side both connect with these settings. The worker’s connect at start follows ARQ_RETRY_ATTEMPTS: with the default 0, a worker started before Redis is reachable fails at once. A running worker rides a Redis outage only within that same retry — arq’s poll loop has no reconnect of its own, so a longer outage ends the worker. --job-timeout defaults to ARQ_TASK_TIMEOUT; an explicit flag wins. The queue label, burst mode, result retention, and poll delay are worker CLI flags, not env vars. worker is the only tai backend subcommand this backend accepts — there is no beat or dashboard process, and anything else is refused by name.

​
Liveness

The worker reports each pass of arq’s poll loop into a liveness file at TAI_BACKEND_LIVENESS_PATH, promising the next pass within --poll-delay. tai42-backend-probe run in the worker’s environment exits 0 while that promise is not overdue by more than TAI_BACKEND_LIVENESS_GRACE seconds, and 1 otherwise, so it serves as the container’s exec liveness probe:
docker-compose.yml
Jobs run as tasks on the same loop, so the passes go on while jobs run; a job that blocks the loop stops them, and the probe then reports not alive. The worker removes the file when it exits. Each worker on one filesystem needs its own TAI_BACKEND_LIVENESS_PATH.

​
Send a task to a chosen worker

The sync_task and async_task branches take an optional queue argument, a label of letters, digits, _ and -. A job sent with queue: gpu goes to the queue <ARQ_QUEUE_NAME>.gpu; any other value is refused before anything is written, and no queue keeps the job on ARQ_QUEUE_NAME. A preset can fix queue in its arguments like any other argument. A worker started with --queue <label> consumes that label’s queue alone; a worker without the flag consumes ARQ_QUEUE_NAME. One worker serves one queue, so a stack that runs labelled tasks runs one more worker per label. In the compose file, beside the backend service:
docker-compose.yml
A job on a queue no worker consumes waits: a sync_task raises TimeoutError after ARQ_TASK_TIMEOUT, and an async_task id stays queued. Schedules carry no queue, so they need a worker without a label. The task tools find a job by id on whichever queue holds it, and backend_reserved_tasks lists due job ids keyed by queue.

​
Worker lifecycle

Everything around the arq worker is the shared backend lifecycle every backend gets, not this plugin’s own: the launch waits for the app’s boot self-resync before the worker consumes (a worker running against a half-built tool registry would fail its jobs permanently), the first SIGTERM or SIGINT asks arq for a warm drain, a repeated one escalates to a cold stop, and the drain is awaited to completion before the process tears down. arq is deliberately built with its own signal handlers disabled. The host owns the process’s signal disposition and composes every subscriber onto one handler per signal; arq’s install would replace that outright and take the server’s teardown with it, which is what makes a recycle drain cleanly instead of severing in-flight jobs.

​
Schedule recovery

On startup a watchdog restarts enabled schedules whose pending job was lost, never found, or completed without rescheduling. A schedule it cannot recover (its record throws every tick) is recorded as an operator-visible error state rather than only logged: its backend_list_schedules row carries error (message, the failure detail, and at_iso, an ISO-8601 timestamp), and backend_get_schedule carries recovery_error and recovery_failed_at. They are null on a healthy schedule and are cleared on any later tick that finds the schedule healthy or restarts it.

​
Quirks

  • Every process of one stack sets the same ARQ_QUEUE_NAME: labelled queues and the record of them (<ARQ_QUEUE_NAME>:queues) are named from it.
  • A tool with its own queue parameter cannot take a sync_task or async_task branch.
  • ARQ_MANIFEST_KEY, ARQ_TASK_TIMEOUT, and ARQ_TOOL_NAME_ARG mirror the host’s own BACKEND_ group. Change them in lockstep or the worker and the server stop agreeing. ARQ_MANIFEST_KEY and ARQ_TOOL_NAME_ARG pin wiring built at boot, so a change to either converges through a process recycle rather than in place.
  • arq cannot honour a relative interval: a schedule with relative: true is refused on create, update and import.
  • The schedule store holds the typed cadence; a hash in an earlier format raises ScheduleStoreFormatError on list, export, get and update until the keys matching ArqSettings.arq_schedule_pattern are deleted and the schedules re-created.
  • A dispatched tool runs once; arq re-runs a job cut by a cold stop, bounded by --max-tries.
  • ARQ_JOB_COMPLETION_WAIT must cover the longest job, and the deployment’s termination grace period must be at least as long, or a drain still severs running work. arq cancels running jobs on signal unless this is non-zero.

​
See also