Migrate from Flower to z4j

This guide walks through replacing Flower with z4j on a production Celery deployment. Both tools observe the same Celery broker events, so you can run them side by side during the cut-over and turn Flower off only when you are confident z4j covers your workflow.

End state: persistent task history in Postgres, retry/cancel/bulk actions from the dashboard, schedule management within the documented Beat support boundary, RBAC with invitations, an HMAC-chained audit log, and the same dashboard for any other engines you add later (RQ, Dramatiq, Huey, arq, taskiq).

Plan a dedicated change window; duration depends on your deployment and validation.

Who this guide is for

You are running Celery 5.3 or newer (z4j-celery alone accepts Celery 5.2.2 and later) with Flower in front of it, and you have outgrown Flower for at least one of these reasons:

• Worker crashes leave tasks "started" forever because Flower has no reconciler.
• You want task history in a separate database with z4j's retention and search tools. Flower also offers optional persistence.
• You need project roles, a shared bulk-action workflow or supported schedule management.
• Your stack is going multi-engine (RQ alongside Celery, for example) and you want one dashboard.
• An auditor asked for an HMAC-chained audit log of who retried which job and when.

If none of that applies and you are happy with Flower, do not migrate. Flower is a fine tool inside the scope it was built for.

Prerequisites

• Celery 5.3 or newer (z4j-celery alone accepts 5.2.2 and later; z4j-celerybeat needs 5.3).
• A reachable Postgres or SQLite path for z4j. Production should use Postgres; SQLite is fine for evaluation.
• The same broker URL and result backend the workers already use. No data migration is required.
• Permission to add the z4j-celery package to the worker venv (plus z4j-django in a Django project), one import line or INSTALLED_APPS entry to the code the worker loads, and one container or process for z4j.

Phase 1: Install z4j alongside Flower

Flower keeps running unchanged. z4j is added in parallel. Neither tool interferes with the other.

# Option A: pip + bundled SQLite, fastest path for evaluation
pip install z4j
z4j serve

# Option B: docker compose, recommended for anything other than a laptop
git clone https://github.com/z4jdev/z4j
cd z4j
docker compose up -d

The first boot prints a setup URL to stderr. Open it, create the first admin, and you have an empty dashboard.

Verify the install:

z4j check    # config + DB connectivity + the stored migration revision (no head compare)
z4j status   # user/project/agent/task counts, version

At this point z4j is running but has no agents and no events. Flower is still your source of truth.

Phase 2: Add the z4j-celery agent to your workers

Install the adapter into the venv your Celery workers use:

pip install z4j-celery

Set these environment variables on the worker process. Z4J_BRAIN_URL, Z4J_TOKEN, Z4J_HMAC_SECRET and Z4J_PROJECT_ID are required; Z4J_AGENT_NAME is optional and unset by default:

export Z4J_BRAIN_URL="https://z4j.example.com"
export Z4J_TOKEN="<bearer-from-the-dashboard>"
export Z4J_HMAC_SECRET="<hmac-from-the-dashboard>"
export Z4J_PROJECT_ID="default"          # required: slug of the project the agent was minted in
export Z4J_AGENT_NAME="celery-prod-1"    # optional, unset by default

Mint the token and HMAC secret in the dashboard on the project's Agents page, New agent. The dashboard prints them once; store them in your secrets manager.

Make the worker import z4j_celery. Celery does not discover the package on its own: installing it and setting the variables starts nothing until the worker process imports it.

• Django project: pip install z4j-django as well, add z4j_django to INSTALLED_APPS and define a Z4J settings dict (Z4J = {} when every value comes from the environment), as in the Django quickstart. z4j_django imports z4j_celery for you, and also runs an agent in the web process.

• Any other project: add one line to the module that defines your Celery app (the module celery -A loads):

  import z4j_celery  # registers z4j's worker_ready bootstrap

  In a Flask or FastAPI application, the worker importing an application module that imports z4j_flask or z4j_fastapi has the same effect; see Celery.

Restart one worker with Celery logging at INFO (-l info). You should see this line in the worker log (Celery adds its own timestamp and process prefix; the logger name is z4j.adapter.celery.worker_bootstrap):

INFO: z4j worker bootstrap: agent runtime started (celery_app=..., framework=...)

The message is logged at INFO, so Celery's default WARNING level hides it even while the agent is running. The line means the agent runtime started inside the worker, not that z4j accepted it. If you log at INFO and do not see it, the agent did not start. Common causes:

• The worker never imports z4j_celery (see the step above).
• The worker was started as celery beat or celery inspect, not celery worker, or through a launcher the bootstrap does not recognize. The bootstrap only starts an agent under celery worker.
• The Z4J_* values are missing from the worker environment. The bootstrap then logs z4j worker bootstrap: no Z4J_* env config found, skipping auto-start at INFO.
• Z4J_DISABLED is set to a truthy value.
• Startup raised an error, logged as z4j worker bootstrap: failed to start agent runtime - worker will continue without z4j observability.

An unreachable Z4J_BRAIN_URL or a wrong Z4J_TOKEN still produces the line, so confirm the worker slot is online in the dashboard. With a wrong token z4j logs bearer rejected and closes the connection with code 4401, and the agent retries on a backoff schedule; with an unreachable URL, check DNS and TLS from inside the worker pod or container.

Once the first worker reports in, verify its task state in the z4j dashboard, then restart and verify the remaining workers.

For retry without explicit argument overrides, configure Celery's result_extended=True (CELERY_RESULT_EXTENDED=True with Django's CELERY namespace) and a result backend. This stores original arguments in that backend under its own access controls and retention policy. z4j refuses to reconstruct missing arguments from redacted event history; alternatively supply both args and kwargs explicitly in the retry request.

Phase 3: Validate parity over a representative business cycle

Run Flower and z4j side by side for one full business cycle. The goal is to make sure z4j shows the same operational picture you trusted Flower for, plus the new things you came for.

A reasonable validation checklist:

• Live tasks appear in z4j as they appear in Flower.
• Task counts per queue match between the two dashboards.
• A task you intentionally fail shows up with its traceback in z4j (z4j-celery sends at most the last 4096 characters of a traceback, so a long one keeps its innermost frames and the exception line).
• A task you intentionally retry from z4j actually re-runs.
• A scheduled task fires at its expected time.
• The audit log entry for the retry shows your user, the timestamp, and the action.
• A task stranded in started by a worker restart is corrected by reconciliation (every 300 s the brain probes tasks stuck longer than 15 min and rewrites its own snapshot to the engine's state; it never re-runs a task).
• Secret arguments are redacted by default in the z4j detail view.

If any of these fail, file an issue at github.com/z4jdev/z4j/issues with the agent log and z4j log. Do not turn Flower off until they all pass.

Phase 4: Cut over

Three things to do, in order:

1. Move any operator playbooks or runbooks that reference flower.example.com to point at z4j.example.com.
2. Stop the Flower process or remove the flower service from docker-compose.yml. Keep the entry commented out in git for one release cycle in case you need to roll back.
3. Tell the team. Include a one-paragraph diff so they know what changed and where the new buttons are.

z4j records events from the time its agent connects. Preserve Flower's persistence file if you enabled it and need its earlier history; z4j does not import that file.

Rollback plan

If validation finds a problem, use the rollback steps below. z4j and Flower remain independent during the overlap period.

# Re-enable Flower with its previous configuration and persistence file.
docker compose up -d flower

# Stop the z4j agent: set Z4J_DISABLED=1 in each worker's environment
# and restart the workers. They keep running normally; only the agent
# stops phoning home.

To remove the package as well, first delete the import z4j_celery line you added, then run pip uninstall -y z4j-celery. Without the package that import fails and the worker cannot load its Celery app. A Django project that relies on z4j_django instead can uninstall z4j-celery directly.

Operators can keep using Flower exactly as before. z4j retains the history it has already recorded; you can come back to it later.

Frequently asked questions

Will z4j double-process my tasks?

No. Passive capture never enqueues anything: the agent reads broker events, worker signals, and the result backend. Operator actions you trigger from the dashboard (retry, cancel, bulk retry, purge, restart, rate limit) do act on the broker and workers through the agent, and each is audit-logged; a retry re-enqueues the original task via send_task, so it runs again by design. Running z4j alongside Flower or any other observer is safe.

Does z4j handle Celery chords, groups, and chains?

Yes. The z4j-celery adapter understands chord/group/chain parents and renders them as a tidy-tree DAG with runtime badges per node. Flower shows these as a flat parent/child list, which is one of the things you may have come here to fix.

What about Celery Beat?

Follow the Celery Beat guide for installation, a long-lived scheduler agent and exact release support. Native create/edit/delete with source confirmation needs the z4j server, z4j-core, z4j-bare and z4j-celerybeat from the same release. Database schedules use django-celery-beat and DatabaseScheduler; application-defined static entries are read-only.

Do I need to change my broker, result backend, or worker code?

Your broker and result backend stay as they are: CELERY_BROKER_URL, CELERY_RESULT_BACKEND, task definitions and the celery worker command do not change. Capture is passive; only the operator actions you trigger from the dashboard act on the broker and workers. The one code change is making the worker import z4j_celery: an import z4j_celery line in the Celery app module, or z4j_django in INSTALLED_APPS (see Phase 2). Retry without explicit arguments also needs result_extended=True and a result backend.

Will my secret task arguments leak to the dashboard?

Secrets are scrubbed by default. The redaction layer recursively walks args and kwargs and replaces values that match common credential patterns (password, token, secret, api_key, authorization, etc.). You extend the patterns on the agent side: set Z4J_REDACTION_EXTRA_KEY_PATTERNS and Z4J_REDACTION_EXTRA_VALUE_PATTERNS in the worker environment, pass the matching install_agent(...) arguments, or use the redaction dict the Django and Flask adapters accept in their Z4J settings. There are no project-level redaction settings in the dashboard.

Does z4j work with django-celery-beat / redbeat?

The Beat guide covers database and static sources. RedBeat is not a write target. Installing the package in a worker alone does not start a scheduler agent; configure the documented long-lived scheduler process.

What happens to historical tasks Flower had cached?

z4j starts a separate history when its agent connects. Flower can retain history when configured with persistence. Keep that data if needed: z4j does not provide a Flower history importer.

How much overhead does the z4j agent add?

The agent buffers events in a crash-safe on-disk SQLite store, one file per process (~/.z4j/buffer-<pid>.sqlite by default; Z4J_HOME moves the directory), and ships them to z4j in batches over a single WebSocket. Measure agent overhead and brain/database throughput with a representative workload before setting production capacity expectations.

Can I still use Flower's HTTP API for scripts I have already written?

z4j ships a REST and WebSocket API documented under API reference. Map each call you scripted against Flower's API to its z4j counterpart in that reference rather than assuming a one-to-one equivalent; operator actions through the z4j API leave an audit-log entry recording who called them.

Is there a paid version with different features?

No. z4j is open source under split licensing: z4j is AGPL v3, the adapters are Apache 2.0. Every feature is in the open-source release. There is no commercial gate, no telemetry, and no phone-home. The only version-check request is the operator-initiated Settings, System, Check for updates fetch of versions.json from GitHub; set Z4J_VERSION_CHECK_URL to an empty string to disable it.

Common pitfalls

Check these integration pitfalls during validation:

• Worker started without celery worker. The z4j-celery worker bootstrap only starts an agent under that command; celery inspect, celery purge and other short-lived sub-commands never start one. celery beat starts an agent only in a Django project with z4j_django installed, as the long-lived scheduler agent the Celery Beat guide describes.
• Reverse proxy strips the WebSocket upgrade. The agent uses a long-lived WebSocket. nginx, Cloudflare, and Traefik all need explicit upgrade handling. See the TLS setup guide for working configs.
• One token for every host. Processes that share a token stay distinguishable: each connects as its own worker slot, and task rows record the Celery worker name. They still appear under one agent, and revoking that token disconnects all of them. Give each worker host its own agent token when you want per-host revocation and a per-host agent row.
• HMAC clock skew. The HMAC frames embed a timestamp. If the worker clock drifts by more than 60 seconds from the z4j clock, z4j rejects the frames. Run NTP everywhere.
• Missing Z4J_PROJECT_ID. The agent refuses to start with missing required Z4J settings: project_id (or Z4J_PROJECT_ID); there is no fallback to a default project. Set it to the slug of the project whose agent token you minted.

Next steps

• The Celery adapter reference covers what events z4j captures and how reconciliation works.
• The Production checklist walks through hardening for self-hosting.
• The Comparison page shows how z4j stacks up against other dashboards if you are still evaluating.
• Marketing landing for the high-level story: Flower vs. z4j on z4j.com.