Grafana

z4j ships five Grafana dashboards in the deploy/grafana/ directory of the source repository on GitHub. They are not in the PyPI sdist or the container image, so clone the repository or download the files directly. They are plain JSON, schema-compatible with Grafana 10.4+, and they work against any Prometheus that is scraping the brain's /metrics endpoint.

Dashboards

File	Title	Purpose
z4j-overview.json	z4j -- Brain overview	First stop. Stat panels for agents online, brain RSS, DB pool utilisation, deadlock rate. Task throughput by final state. Task duration p50 / p95 / p99. Queue depth by project. Background-task error flags. Swallowed exceptions by module.
z4j-tasks.json	z4j -- Tasks	Tail latency or failure rate climbing? Open this. Stacked task throughput by state, failure-rate ratio, full duration heatmap, top-10 failing / slow (p99) / by-volume / retried task names.
z4j-agents.json	z4j -- Agents and commands	Agent + worker counts per project. Command dispatch flow by status and action. Late-result counter (tuning signal for Z4J_COMMAND_TIMEOUT_SECONDS). Live WebSocket connection count. In-memory state by subsystem.
z4j-notifications.json	z4j -- Notifications	Send rate by channel type and status. Failure-rate table (per channel, red rows are channels currently failing). Cooldown skip rate per trigger. 24h channel mix donut. Blocked-by-SSRF / host-lock rate.
z4j-scheduler.json	z4j-scheduler	Only relevant when the scheduler companion is deployed. Leader status, fire throughput by terminal state, fire latency p50/p99, tick drift, per-schedule top-N.

Each brain dashboard exposes a $project template variable (multi-select, all by default) so multi-tenant deployments can scope panels to a single project without editing JSON.

Importing

Manual

1. Grafana, Dashboards, New, Import.
2. Upload the JSON.
3. Pick your Prometheus datasource.
4. Save.

Provisioned (production)

Mount deploy/grafana/ into the Grafana container and provision a file-based dashboard provider. A complete example sits in deploy/grafana/README.md alongside the dashboards in the repository. Grafana picks up changes within updateIntervalSeconds of a JSON edit, so the dashboards become a normal infra-as-code artefact.

For the Kubernetes Grafana Helm chart, drop the JSONs into a ConfigMap and point dashboardsConfigMaps at it.

Prometheus scrape config

Brain (default port 7700, fail-secure metrics auth):

scrape_configs:
  - job_name: z4j-brain
    metrics_path: /metrics
    static_configs:
      - targets: ["brain.internal:7700"]
    authorization:
      type: Bearer
      credentials: "<Z4J_METRICS_AUTH_TOKEN>"

Z4J_METRICS_AUTH_TOKEN is auto-minted on first boot and persisted to $Z4J_HOME/secret.env. On trusted-LAN deployments you can flip Z4J_METRICS_PUBLIC=true and drop the authorization block; the brain logs a loud WARNING at startup naming the risk.

Scheduler companion (only if z4j-scheduler is deployed). It serves /metrics from its own API port, default 7800:

  - job_name: z4j-scheduler
    metrics_path: /metrics
    static_configs:
      - targets: ["scheduler.internal:7800"]
    # Drop this block if Z4J_SCHEDULER_METRICS_AUTH_TOKEN is unset.
    authorization:
      type: Bearer
      credentials: "<Z4J_SCHEDULER_METRICS_AUTH_TOKEN>"

The route is only mounted when Z4J_SCHEDULER_METRICS_ENABLED is true; with it off, /metrics returns 404.

Why five and not one

Five small dashboards over one mega-dashboard is deliberate:

• Each dashboard fits a single screen at 1080p without horizontal scroll.
• The split matches the operator's natural drill path: overview, then per-area (tasks / agents / notifications), then per-row (the top-N panels link the operator to the exact task name or channel that is misbehaving).
• You can permission them independently in Grafana folders -- the on-call team often does not need write access to the scheduler dashboard, for example.

Suggested alerts

A full table sits in deploy/grafana/README.md alongside the dashboards, with starter thresholds for:

• zero agents online
• DB pool saturated
• sustained Postgres deadlocks
• a self-watch background task failing (audit retention, WAL checkpoint, etc.)
• task failure rate above 5%
• late command results (timeout-sweeper mistune)
• notification channel failing above 10%
• swallowed-exception spike by module
• brain not reporting metrics at all (catch-all liveness)

The thresholds are starting points; tune to your fleet's normal baseline.

Legacy

An older four-panel snapshot (RSS slope, deadlock rate, DB pool utilisation, and events context) predates this set and is kept under docs/perf/ in the source tree for reproducibility. deploy/grafana/z4j-overview.json covers the same signals plus much more; use that.