backend-celery

​

Install

tai plugins install tai42-backend-celery

​

Permissions

​

Provides

​

celery

​

Enable it

backend_module: tai42_backend_celery

​

Configuration

​

Redis connections

• The plugin picks the Redis result backend and broker transport classes itself, so a URL from CELERY_RESULT_BACKEND, CELERY_BROKER_URL, CELERY_BROKER_READ_URL or CELERY_BROKER_WRITE_URL goes through them alike. CELERY_LOADER and --loader do not change the app’s loader.
• The Redis URL forms taken are redis://, rediss:// and redis+socket:// (and redis://, rediss:// and unix:// for CELERY_REDBEAT_REDIS_URL). Every other Redis form — sentinel://, a ; failover list, unix:// or socket:// for the result backend or the broker, redis-cluster:// or redis-sentinel:// for RedBeat — stops the app at startup with an error naming the setting. A non-Redis broker or result backend (AMQP, rpc://, …) is left to celery, with celery’s own retries.
• With RETRY_ATTEMPTS unset, a Redis connection makes one attempt: a result write fails on the first error, a worker whose Redis read side is not reachable at start exits, and a publish over a Redis write side fails at once. Celery’s own startup connect retry is switched off when the read side is Redis, and its publish retry when the write side is Redis, each side on its own; the namespace’s RETRY_ATTEMPTS takes their place.
• A running worker that loses its Redis broker keeps reconnecting through celery’s own loop (broker_connection_max_retries), logged at ERROR.
• CELERY_BROKER_RETRY_ATTEMPTS never retries a read kombu polls: a worker’s task and fanout reads, and a control command’s reply read (inspect(), control) in any process. A drop there raises at once. A worker’s drop restarts celery’s consumer, which reconnects; a control call fails with the connection error, and a retry of that call reconnects.
• A result wait whose Redis result backend drops is reconnected and re-subscribed by CELERY_RESULT_BACKEND_RETRY_ATTEMPTS; after each reconnect it reads the stored state of every task it still awaits, so a result written meanwhile still arrives. When that retry runs out, the wait raises the connection error, and the next wait, on any task, connects again, re-subscribes every task still awaited and reads their stored state.
• kombu’s own connect step retries once after 2 s, and its channel open retries within broker_connection_timeout; neither is logged.
• flower’s events, inspector and results go through the plugin’s app and the settings above. Its broker and queue-length views open their own Redis connection per request, outside these settings, with no retry.

​

Run the processes

tai backend worker
tai backend beat
tai backend flower

​

Liveness

• worker reports once a second from a consumer bootstep on the timer of Celery’s event loop, so a report proves the loop is passing. While the broker connection is down the loop does not pass, so the file goes stale; the reports resume once the consumer reconnects. Per pool:
  • prefork: the jobs run in the pool children and the loop goes on while they run; a child stuck in a job is the pool’s to end, by CELERY_TASK_TIMEOUT.
  • solo: the job runs on the loop’s own thread, so the running job reports once a second from its own asyncio loop. A job that blocks that loop stops the reports. The pool cannot end a job, so a job past CELERY_TASK_TIMEOUT holds a wedged slot: the worker then reports nothing, so the probe fails and the orchestrator restarts it.
  • threads: the loop goes on while the jobs run in the pool’s threads. Once every thread holds a job past CELERY_TASK_TIMEOUT the worker reports nothing; one such job beside free threads keeps it reporting.
  • gevent, eventlet: the timer runs on its own green thread, so there a report proves the process and that thread.
• worker -B embeds a beat, which reports each tick into its own file, <TAI_BACKEND_LIVENESS_PATH>-beat, with its own pid. The worker reports nothing while its beat’s promise is overdue, so the one probe on the worker’s file covers both. When the worker exits, even before it ever reached its broker, it ends its beat and removes both files.
• beat reports each tick of RedBeat’s scheduler, promising the next within the interval the beat sleeps (at most --max-interval). A beat that cannot hold RedBeat’s lock waits for it before its first tick and reports nothing while it waits, so run one beat (standalone or embedded with -B). The app names the reporting scheduler as its beat_scheduler; --scheduler is refused by tai backend worker and tai backend beat.
• flower serves HTTP, so it is probed over HTTP instead: GET /healthcheck answers 200 while Flower serves, without authentication even when Flower’s own auth is on. Under --url_prefix, the path is /<prefix>/healthcheck.

services:
  backend:
    healthcheck:
      test: ["CMD", "tai42-backend-probe"]

​

Worker lifecycle

​

Quirks

• CELERY_MANIFEST_KEY and CELERY_TOOL_NAME_ARG pin wiring built at boot — a forked child reads the env key it was given and a queued job carries the kwarg name its producer used — so a change to either converges through a process recycle rather than in place: the recycled process exits and the deployment’s supervisor starts it again.
• On macOS, prefer the solo or threads pool — forking around the system resolver deadlocks.
• backend_list_failed_tasks is not implemented: Celery keeps no queryable failed-task index.
• A dispatched tool runs once: the backend never retries a tool body. A tool that wants retries declares a ToolRetryPolicy. CELERY_TASK_TIMEOUT is bound into Celery’s task_time_limit when the worker starts, so a change converges through a process recycle. The prefork pool enforces it; the solo and threads pools cannot end a job, and there the worker’s liveness stops reporting once every slot holds a job past the budget (see Liveness).
• Schedules use the contract’s typed cadence; this backend honours an interval’s relative flag and stores schedules in RedBeat’s own format.

​

See also

• Backends — the contract all three shipped backends implement.
• Schedule a tool — putting work on a schedule.