Troubleshooting

Run the doctor first

Before anything else, run the framework-side doctor as the same user the service runs under. It loads the agent configuration, then probes the buffer directory, DNS, TCP and TLS to the brain, and local startup of the agent runtime, in that order, and stops at the first failure with a specific reason. It does not verify a connection to the brain or authentication with it, so a wrong token or project id, a wire-protocol mismatch, or a proxy that drops the WebSocket upgrade still passes it; the lists below cover those.

Stack	Command
Django	`python manage.py z4j_doctor`
Flask	`python -m z4j_flask doctor`
FastAPI	`python -m z4j_fastapi doctor`
Bare Python	`python -m z4j_bare doctor`

Add --no-websocket to skip the last probe, which starts and stops the local transport runtime and never reaches the brain. Add --json for scripts. Exits 0 on all-green, 1 on any failure.

Agent shows offline

Run the doctor - it surfaces the most common failures with a specific reason. The list below covers the rest.
Check the agent's app logs for the z4j.runtime.* loggers at boot (messages begin z4j agent ... or z4j buffer: ...). Look for handshake errors.
Verify Z4J_BRAIN_URL is an https:// URL in production. The agent derives wss:// from it. The WebSocket transport accepts a plain http:// URL only for localhost, 127.0.0.1 or ::1, or with dev mode on; the long-poll transport refuses http:// for every host, loopback included, unless dev mode is on.
Verify the token is the one shown at mint time (tokens are not recoverable - re-mint if lost).
Egress firewall: the agent's host must reach z4j on TCP 443 (or wherever your proxy is).
Proxy WebSocket passthrough: ingress-nginx passes the upgrade through on its own; give it long timeouts with the nginx.ingress.kubernetes.io/proxy-read-timeout and proxy-send-timeout annotations set to "3600" (see Kubernetes). A hand-written nginx location needs these three directives:
```
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
```

Buffer relocated under gunicorn / uvicorn service users

A WARNING reading z4j buffer: /var/www/.z4j is not writable; falling back to /tmp/z4j-33. Set Z4J_HOME to a persistent writable location to silence this warning. (or any path that's not the running user's writable home) in the service log, followed by a second WARNING, z4j buffer: requested /var/www/.z4j/buffer-<pid>.sqlite but the parent dir is not writable; using /tmp/z4j-33/buffer-<pid>.sqlite instead.; no PermissionError reaches the log. The service user has an unwritable $HOME. The agent auto-relocates the buffer to $TMPDIR/z4j-{uid}/buffer-{pid}.sqlite and logs those two WARNINGs instead of crashing. See service-user deployments.

Events don't appear

Agent is online (Agents page shows online)?
Engine is auto-detected (Engines column on the Agents page)?
For Django: INSTALLED_APPS includes z4j_django after any Celery apps?
For Flask: z4j.init_app(app) (or Z4J(app)) ran in the app factory? It takes only the app; configuration comes from app.config or the environment.
For FastAPI: agent is inside the lifespan context manager?
Task names registered (Workers, open the worker, Registered tasks)?

The banner counts agents that have connected at least once and whose last advertised wire protocol_version is lower than the brain's current protocol. It is about the wire protocol, not the package version. Package versions drive the separate per-row badges on the Agents page, compared against the brain's versions snapshot: update available when a newer package of the same major exists, incompatible for a different major. Re-deploy the named agents with pip install -U z4j-*.

Audit chain `verify` fails

Someone modified the audit_log table directly, or a backup restore is incomplete.

Read the MISMATCHES (<count>) block the command prints: one line per finding, capped at 100 with a closing line naming how many further findings are not shown. Rows outside the authenticated generations are reported as their own finding.
If known-intentional (e.g., planned DB surgery), document the break externally. The chain from there is not recoverable.
If unexpected, treat as a compromise event - preserve the DB, alert security, investigate.

`409 conflict` on mint-token

The body carries "error": "conflict" and the message an agent with that name already exists in this project: another agent already has that (project_id, name). Pick a different name or delete the old agent first.

Schedule controls missing or failing for a scheduler

The scheduler adapter advertises the operations it supports (list, create, update, delete, enable, disable, trigger_now) and the dashboard offers only those; there is no read_only flag. celery-beat always advertises list, adds trigger_now when it knows the Celery app, and adds create, update, delete, enable and disable only when a writable source such as django-celery-beat is available. A static beat_schedule (for example under PersistentScheduler) never advertises the writes, so the dashboard does not offer them. See schedulers overview.

Password reset email not arriving

Email channel configured? Use the channel test endpoint (POST /api/v1/projects/{slug}/notifications/channels/{channel_id}/test) or the Test button on the dashboard's Notifications, Channels page.
From: domain has SPF / DKIM / DMARC?
Sender reputation good? Gmail drops many SMTP senders silently.

Very high CPU

Check rate(z4j_events_ingested_total[1m]) broken down by project and engine - is one agent emitting millions of events?
Redaction patterns not looping on giant payloads? - the redactor truncates any single value above 8192 bytes before matching.
Hot task - check z4j_task_duration_seconds by task_name. There is no per-endpoint HTTP histogram; get request latency from your reverse proxy.

`503 agent_offline` on a retry

The target agent was offline when z4j dispatched the command. The command still stays pending and is delivered if the agent reconnects before its deadline (Z4J_COMMAND_TIMEOUT_SECONDS, default 60). Check it on the Commands page before retrying: a second retry sent while the first is pending can run the task twice. Pick a different agent for the same engine only after the first command has timed out.

When stuck

Include z4j logs (with X-Request-Id), agent logs, and a description of the sequence - file at github.com/z4jdev/z4j/issues.