Skip to content
z4j docsLatestv1.13.2

Service-user deployments

Production deployments often run the host process under a dedicated service user - www-data for nginx + gunicorn, app for systemd-managed uvicorn, or an ephemeral uid via DynamicUser=yes. This page covers what changes about the agent in those environments and how to debug when it doesn't come online.

What we handle for you

Section titled “What we handle for you”

Service users typically don't have a writable $HOME. www-data resolves to /var/www, nobody to /nonexistent, DynamicUser to a transient mount. The agent's on-disk SQLite buffer would normally land in ~/.z4j/, which the process can't create.

The agent detects this at startup and relocates the buffer to a per-uid tmp directory:

$TMPDIR/z4j-{uid}/buffer-{pid}.sqlite     # directory 0700, file 0600

Two WARNINGs are logged at startup: the first names the unwritable home and the fallback directory, the second the requested buffer file and the relocated one. The buffer is still crash-safe (SQLite WAL mode), still per-process (no counter drift), and still bounded - only the location moved.

When you want a persistent buffer location

Section titled “When you want a persistent buffer location”

The tmp fallback works but is wiped on tmpfs clears. If your host has a tmpfs /tmp, set the canonical z4j state directory so events buffered during a brain outage survive a host reboot:

Terminal window
# 1. Create a writable dir owned by the service user
sudo mkdir -p /var/lib/picker/.z4j
sudo chown www-data:www-data /var/lib/picker/.z4j


# 2. Point z4j at it via systemd unit override
sudo systemctl edit gunicorn

Add to the override:

[Service]
Environment=Z4J_HOME=/var/lib/picker/.z4j
Terminal window
sudo systemctl daemon-reload
sudo systemctl restart gunicorn

Z4J_HOME relocates all per-host z4j state, including the per-process buffer-{pid}.sqlite files and agent pidfiles. The removed Z4J_BUFFER_PATH, Z4J_BUFFER_DIR, and Z4J_RUNTIME_DIR variables cause a startup refusal; do not use them for relocation.

The doctor command

Section titled “The doctor command”

When an agent doesn't come online, run the framework-side doctor first. It probes the buffer directory, DNS, TCP and TLS to the brain, and local runtime startup, synchronously and without starting the persistent connection - so you get specific failure reasons instead of "the dashboard shows unknown".

The doctor's buffer probe resolves the buffer location the way the agent does: the configured directory (Z4J_HOME, or ~/.z4j when it is unset) first, then the per-uid tmp fallback. When the agent would fall back, the probe passes and names the fallback directory, so under a service user with no writable home the doctor and the agent agree. It fails only when neither location is writable.

Django:

Terminal window
sudo -u www-data /srv/picker/venv/bin/python /srv/picker/picker/manage.py z4j_doctor

Flask:

Terminal window
sudo -u www-data /srv/picker/venv/bin/python -m z4j_flask doctor

FastAPI:

Terminal window
sudo -u www-data /srv/picker/venv/bin/python -m z4j_fastapi doctor

Standalone Celery (no framework):

Terminal window
sudo -u www-data /srv/picker/venv/bin/python -m z4j_bare doctor

Always run the doctor as the same user the service runs under, with the same Z4J_HOME (for example sudo -u www-data env Z4J_HOME=/var/lib/picker/.z4j /srv/picker/venv/bin/python -m z4j_bare doctor). Otherwise Path.home() or Z4J_HOME resolves differently, the buffer-path probe reads a different writable status, and the result doesn't reflect what the actual service sees.

Sample output

Section titled “Sample output”

A Django run as www-data with Z4J_HOME=/var/lib/picker/.z4j:

z4j-doctor (django)
===================
  brain_url:   https://tasks.example.com/
  project_id:  picker
  agent_name:  picker_django
  buffer_path: /var/lib/picker/.z4j/buffer-7281.sqlite
  transport:   auto


  [OK]   buffer_path  OK: buffer dir /var/lib/picker/.z4j is writable
  [OK]   dns          OK: tasks.example.com -> 198.51.100.42
  [OK]   tcp          OK: TCP connect to tasks.example.com:443
  [OK]   tls          OK: TLS TLSv1.3 to tasks.example.com (cert CN='tasks.example.com')
  [OK]   websocket    OK: local agent transport runtime started for https://tasks.example.com/; remote connection and authentication were not verified


  engines auto-detected: celery

The same run without Z4J_HOME passes the buffer probe on the tmp fallback and says so:

  [OK]   buffer_path  OK: buffer dir /var/www/.z4j is not writable, so the agent uses the per-uid tmp fallback /tmp/z4j-33; set Z4J_HOME to a persistent writable directory to keep the buffer across reboots

Add --no-websocket to skip the last probe, which starts and stops the local transport runtime and never reaches the brain. Add --json for scripting.

What each probe catches

Section titled “What each probe catches”
Probe Catches
buffer_path No writable location for the buffer: neither the configured directory (Z4J_HOME, or ~/.z4j when it is unset) nor the per-uid tmp fallback. A buffer the agent would place on the fallback passes, with the fallback directory in the message
dns Bad brain hostname, split-DNS, stale /etc/hosts
tcp Egress firewall blocks z4j port, NAT timeout, no route
tls Cert hostname mismatch, expired cert, untrusted CA, intermediate cert missing
websocket Local transport start only: hmac_secret missing, malformed or too short, buffer initialisation failure, adapter signal wiring. It does not test the proxy's Upgrade: passthrough, the token, the project_id or the protocol version; those failures appear in the agent log once the runtime connects

systemd unit checklist

Section titled “systemd unit checklist”

Common problems that the doctor will surface:

  • Environment= vs EnvironmentFile= - env vars set in your shell or a project .env file don't propagate to the service. Either declare them in the unit override (sudo systemctl edit <service>) or set EnvironmentFile=/path/to/.env so systemd loads it.
  • ProtectHome=true / PrivateTmp=true - these restrict where the process can write. The agent's tmp fallback still works under PrivateTmp=true (it is a per-service tmpfs mount). For durable state, grant the service user write access to its configured Z4J_HOME; do not disable service hardening merely to make an unrelated home directory writable.
  • User= vs running as root - token files and buffer paths created while testing as root won't be readable when the service drops to its real user. Always test the doctor as the service user.
Section titled “Related”