Skip to main content
Backend plugin · listing tai42/backend-rq

​
Install

​
Permissions

​
Provides

​
rq

Backend — RQ execution backend — background tool runs and schedules over Redis. RQ is Redis-only like arq, with a worker, a scheduler, and the RQ dashboard.

​
Enable it

manifest.yml
A config that registers a task backend also needs the worker bus and a shared hooks registry: tai serve, tai backend, a config reload and a config write all refuse it unless TAI_BUS_REDIS_URL and HOOKS_REDIS_URL are set (each falls back to TAI_DEFAULT_REDIS_URL). The backend process matches the events it raises against its own hooks registry, so an in-memory one would run none of the hooks registered through the server.

​
Configuration

The RQ_ env group. RQ_REDIS_URL and RQ_REDIS_MAX_CONNECTIONS fall back to the shared TAI_DEFAULT_REDIS_URL / TAI_DEFAULT_REDIS_MAX_CONNECTIONS when unset; a set RQ_ value always wins. The retry and socket settings are read from RQ_ alone. Every Redis connection of the backend — the tools, the enqueue path, the worker and the beat — is built from them.

​
Run the processes

The worker runtime runs RQ’s work loop on a dedicated thread, so the process’s event loop stays free for worker-bus ops. Its lifecycle is the shared one every backend gets: it waits for the app to finish booting before it consumes, then SIGTERM (or SIGINT) drains it warm — no new jobs, the current one finishes, the process exits after teardown — and a second signal kills the running work-horse. An idle worker stops within one dequeue poll (5s). beat and dashboard pull no queued work, so they run on the event loop and keep whatever signal handling their own CLI installs. Nothing else runs in those processes, so that is deliberate. A waiting worker that loses Redis keeps reconnecting through RQ’s own loop, logged at ERROR, outside RQ_RETRY_ATTEMPTS. beat takes rq-scheduler’s own options that are not about the connection (-b/--burst, -i/--interval, -v/--verbose, --quiet, --path, --pid, -j/--job-class) and -u/--url. Its connection comes from the RQ_ settings, like the worker’s: the URL falls back to TAI_DEFAULT_REDIS_URL, and -u/--url replaces only the URL. The scheduler loop has no reconnect of its own, so the beat rides a Redis outage only within RQ_RETRY_ATTEMPTS; a longer one ends it. The dashboard opens its own Redis connection from its --redis-url option, outside the backend’s connection settings, so the RQ_ retry and timeouts do not apply to it.

​
Liveness

The worker and beat processes report each pass of their loop into a liveness file at TAI_BACKEND_LIVENESS_PATH. tai42-backend-probe (shipped with tai42-kit) run in a process’s environment exits 0 while the promise its liveness file holds is not overdue by more than TAI_BACKEND_LIVENESS_GRACE seconds, and 1 otherwise, so it serves as that container’s exec liveness probe. Each process on one filesystem needs its own TAI_BACKEND_LIVENESS_PATH, and removes its file when it exits.
  • worker reports on each RQ heartbeat: every dequeue poll (at most 5s apart) and every job boundary. While a forked work-horse runs a job, the worker’s horse monitor reports; on the solo and gevent pools, where the job runs in the worker process, a monitor thread reports once a second until the job ends. It is an OS thread under gevent too, so a job that holds the CPU does not starve it. Those pools cannot end every job at RQ_TASK_TIMEOUT (one blocked in a C call, or under gevent one that never yields), so the monitor stops reporting once a job runs past it, and the probe ends the worker. A report that fails (the file is owned by another live process, or cannot be written) stops the worker before its next job, and the launch exits non-zero.
  • beat reports each pass of the scheduler loop, promising the next within its -i/--interval.
  • dashboard serves HTTP, so it is probed over HTTP instead: GET /favicon.ico answers 200 while the dashboard serves; under --url-prefix the path is /<prefix>/favicon.ico. With --username set, that path sits behind the same basic auth as the rest of the dashboard, so the probe sends the Authorization header.
docker-compose.yml

​
Send a task to a chosen worker

The sync_task and async_task branches take an optional queue argument, a label of letters, digits, _ and -. A job sent with queue: gpu goes to the queue <RQ_QUEUE_NAME>.gpu, its callback with it; any other value is refused before anything is written, and no queue keeps the job on RQ_QUEUE_NAME. A preset can fix queue in its arguments like any other argument.
A worker started with --queue <label> serves that label’s queue alone, and moves its countdown / eta jobs onto it; a worker without the flag serves RQ_QUEUE_NAME. A stack that runs labelled tasks runs one more worker per label. A job on a queue no worker serves waits: a sync_task raises TimeoutError after RQ_TASK_TIMEOUT, and an async_task id stays queued. Recurring schedules carry no queue: beat puts them on RQ_QUEUE_NAME, so they need a worker without a label.

​
Quirks

  • A tool with its own queue parameter cannot take a sync_task or async_task branch.
  • RQ has no disabled-schedule state, so backend_list_schedules always reports a schedule as enabled, and disabling one deletes it.
  • backend_registered_tasks and backend_list_failed_tasks are not implemented.
  • The rq scheduler cannot honour a relative interval: a schedule with relative: true is refused on create, update and import.
  • The worker runs every job — enqueued, or fired by the scheduler — under its own RQ_TASK_TIMEOUT, whatever timeout the job was stored with, so a change converges through a worker recycle.

​
See also