Troubleshooting

Run the doctor first

Before anything else, run the framework-side doctor as the same user the service runs under. It probes everything below in order and reports the first specific failure.

Stack	Command
Django	python manage.py z4j_doctor
Flask	python -m z4j_flask doctor
FastAPI	python -m z4j_fastapi doctor
Bare Python	python -m z4j_bare doctor

Add --no-websocket to skip the WS round-trip when z4j is intentionally offline. Add --json for scripts. Exits 0 on all-green, 1 on any failure.

Agent shows offline

1. Run the doctor - it surfaces the most common failures with a specific reason. The list below covers the rest.
2. Check the agent's app logs for [z4j] lines at boot. Look for handshake errors.
3. Verify Z4J_BRAIN_URL uses wss:// in production, ws:// in local dev.
4. Verify the token is the one shown at mint time (tokens are not recoverable - re-mint if lost).
5. Egress firewall: the agent's host must reach z4j on TCP 443 (or wherever your proxy is).
6. Proxy WebSocket passthrough: nginx-ingress needs proxy_set_header Upgrade and proxy_set_header Connection.

Agent silently fails to start under gunicorn / uvicorn

PermissionError: ... /var/www/.z4j (or any path that's not the running user's writable home) in the service log. The service user has an unwritable $HOME. The agent auto-relocates the buffer to $TMPDIR/z4j-{uid}/buffer-{pid}.sqlite and logs a single WARNING instead of crashing. See service-user deployments.

Events don't appear

1. Agent is online (Agents page shows online)?
2. Engine is auto-detected (agent drawer → engines list)?
3. For Django: INSTALLED_APPS includes z4j_django after any Celery apps?
4. For Flask: z4j.init_app(app, ...) was called on the app factory?
5. For FastAPI: agent is inside the lifespan context manager?
6. Task names in registry (agent drawer → registry)?

"Agent version mismatch" banner

You've upgraded z4j to a newer major than the agent. Re-deploy agents with pip install -U z4j-*. Agents up to one major version behind still work but may lack new features.

Audit chain verify fails

Someone modified the audit_log table directly, or a backup restore is incomplete.

1. Identify first_broken_id.
2. If known-intentional (e.g., planned DB surgery), document the break externally. The chain from there is not recoverable.
3. If unexpected, treat as a compromise event - preserve the DB, alert security, investigate.

409 conflict_duplicate_name on mint-token

Another agent already has that (project_id, name). Pick a different name or delete the old agent first.

Schedules showing read_only

The scheduler backend doesn't support writes (e.g. celery-beat with PersistentScheduler, rq-scheduler currently). See schedulers overview.

Password reset email not arriving

1. Email channel configured? Use the channel test endpoint (POST /api/v1/projects/{slug}/notifications/channels/{channel_id}/test) or the Test button on the dashboard's Notifications, Channels page.
2. From: domain has SPF / DKIM / DMARC?
3. Sender reputation good? Gmail drops many SMTP senders silently.

Very high CPU

1. Check rate(z4j_events_ingested_total[1m]) broken down by project and engine - is one agent emitting millions of events?
2. Redaction patterns not looping on giant payloads? - the redactor has a 2 MiB payload cap.
3. Hot task - check z4j_task_duration_seconds by task_name. There is no per-endpoint HTTP histogram; get request latency from your reverse proxy.

503 agent_offline on a retry

The target agent was offline when z4j dispatched the command. The command still stays pending and is delivered if the agent reconnects before its deadline (Z4J_COMMAND_TIMEOUT_SECONDS, default 60). Check it on the Commands page before retrying: a second retry sent while the first is pending can run the task twice. Pick a different agent for the same engine only after the first command has timed out.

When stuck

Include z4j logs (with X-Request-Id), agent logs, and a description of the sequence - file at github.com/z4jdev/z4j/issues.