Monitoring

Health endpoints

GET /api/v1/health          # liveness, public
GET /api/v1/health/ready    # readiness, public
GET /api/v1/health/system   # build and platform detail, authenticated
GET /api/v1/health/deep     # per-subsystem report, authenticated

Liveness

/api/v1/health does no I/O and returns exactly:

{ "status": "ok" }

Nothing else. The endpoint is publicly reachable so container runtimes can poll it cheaply, and it deliberately carries no version field: publishing the brain version on an unauthenticated endpoint lets an attacker pin known CVEs to your install. Version detail lives on /health/system, behind authentication.

Readiness

/api/v1/health/ready returns 200 once lifespan startup has finished and a SELECT 1 round-trips within two seconds. Otherwise it returns 503 with {"status": "unready", "reason": "starting"} or {"status": "unready", "reason": "database"}. That is the whole check: it does not compare the schema against the migration head, and it does not look at agent connectivity. Pair it with a Kubernetes readinessProbe.

Per-subsystem detail

/api/v1/health/deep (authenticated) reports one entry per subsystem and returns 503 when any of them failed. Every response carries a coverage object saying that its verdict is not the one the boot path would give the same database, in both directions: it checks less than startup does, and it can also fail on a deadline startup does not have. A check that could not complete is reported as failed rather than omitted, because an omission reads as healthy.

The audit_chain check authenticates the chain-state row and reports scope: "state-only" on every answer, clean or not. It does not read audit_log, so a tampered audit row passes it. Do not wire an audit-integrity alert to this endpoint; that is the scheduled verifier's job, below.

/api/v1/health/system (authenticated) is build and platform detail: version, Python, OS, database version and size, package versions. It is not a subsystem report.

Audit-chain verification

Verification walks every retained row, so it is not something a health probe can do on each poll. It runs in two places instead:

• On demand, via z4j audit verify (CLI, dashboard button).
• On a schedule, via a built-in leader-gated worker. It is off by default. Set Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true to turn it on, and Z4J_AUDIT_CHAIN_VERIFY_INTERVAL_SECONDS to change the cadence (default 86400, floor 900, ceiling 604800).

A failed verification is logged at error level and counted. It does not stop the brain, because refusing to serve would destroy your ability to investigate the thing that just tripped.

With the worker enabled, two metrics carry the result:

• z4j_audit_chain_verifications_total{outcome} (counter): runs by outcome. clean, failed (the chain did not verify), or error (the run could not complete, which is a different problem and must not be read as evidence of tampering).
• z4j_audit_chain_rows_verified (gauge): rows walked by the most recent run.

Neither metric is emitted while the worker is disabled, so an alert on them is also an alert on having switched it off.

Remember what a clean result covers. It says nothing outside z4j's own write path touched the log. A role that can write audit_log and audit_chain_state directly can roll the log back to an earlier state the brain itself signed, and this verification will report it clean. To detect that you need a head exported somewhere that role cannot write, checked with z4j audit verify --known-head. See HMAC audit chain.

Metrics

Scrape /metrics (Prometheus format, token-gated). Full metric list: metrics API.

Alerts to set

Alert	Trigger
Brain down	up{job="z4j"} == 0 for 2m
Audit chain broken	increase(z4j_audit_chain_verifications_total{outcome="failed"}[1d]) > 0
Audit chain unwatched	increase(z4j_audit_chain_verifications_total[2d]) == 0
Agents dropping	sum(z4j_agents_online) falls by > 20% in 5m
Task failure rate	rate(z4j_tasks_total{state="task.failed"}[5m]) > 0.1
Task backlog growing	engine-specific (Celery/RQ/...)
Background task failing	max(z4j_background_task_error_active) == 1 for 15m
High HTTP 5xx	Not available from /metrics. The brain exports no HTTP request counter; alert on 5xx at your reverse proxy.

Logs

JSON to stdout. Fields:

• ts, level, logger, msg
• request_id (per HTTP request)
• user_id (when authenticated)
• project_id
• agent_id (when relevant)

Ship with Fluent Bit / Vector / Loki / Datadog.

Error tracking / APM

Two optional integrations ship with the brain. Both are off until you configure them.

• Sentry. Install z4j[sentry] and set Z4J_SENTRY_DSN. Unhandled exceptions in HTTP handlers, background workers and domain code are captured, and every event passes through a redaction pass before the SDK ships it. See Sentry.
• OpenTelemetry. Install z4j[otel] and set Z4J_OTEL_EXPORTER_OTLP_ENDPOINT. FastAPI requests, SQLAlchemy queries and outbound httpx calls are traced and exported over OTLP. Traces only; metrics stay on /metrics. See OpenTelemetry.

Neither is required. Application logs go to stdout as JSON and can be shipped by your log pipeline (Fluent Bit / Vector / Loki / Datadog) instead. There is no bundled APM agent beyond the OTLP exporter above.

Agent telemetry loss

Open Agents → Health to inspect the latest 100 retained status samples. The view keeps the newest sample for each buffer and adapter runtime; it does not add repeated cumulative counters. Each report shows its authenticated worker identity and the time the agent sent it, not the time its counters were read. A report buffered during an outage is stamped when it is finally sent after the reconnect, so its Reported time can be well after the loss it describes. The Brain stores at most 12 status samples per minute from one agent connection or long-poll upload and drops the rest, so part of a large backlog can be missing. An empty view, an older agent or a disabled status stream means accounting is unavailable, not that no telemetry was lost.

• event_records: records inside discarded, readable event batches.

• command_results: discarded command-result frames; task execution may still have happened. Investigate the command's outcome before retrying work.

• other_frames: discarded heartbeat, status, registry and other control frames.

• unclassified_frames: discarded event batches whose records could not be read.

• capacity_evicted_frames and content_rejected_frames: why frames were discarded. Content rejection also covers frames the agent refuses to send because they are unreadable, are not a signed frame type, or are larger than the Brain accepts. These overlap the categories above; do not add them to event counts.

• adapter_events: event records lost from a supported adapter's in-memory capture queue, before reaching the buffer. Celery and RQ report this counter. An omitted adapter has no supported counter, rather than a measured zero.

Buffer counters are stored with the buffer file, including across reopening. Deleting a drained buffer starts a new buffer identity; adapter counters start a new scope with the runtime. Samples follow event-history retention and are best effort, not a permanent audit of every previous process or deleted buffer. A buffer deletion and its loss accounting commit together; acknowledgements do not increment the counters. Under sustained overflow, loss caused by emitting a health frame becomes visible in a subsequent sample.

A positive heartbeat report also emits the structured Brain warning "z4j agent reports telemetry loss; inspect agent health", at most once per minute per connection when the reported counters change. This works even with Z4J_AGENT_STATUS_DISABLED=1; that switch removes the status history used by the Health view. The older dropped_events heartbeat field remains a compatible, capped total of buffer event records and supported adapter event loss; use the structured fields for precise categories. No new Prometheus counter is implied by these reports.

Check transport connectivity, persistent content-rejection messages and capture backpressure. Increase buffer limits only after checking disk capacity and the expected outage window. Loss of lifecycle telemetry does not itself establish loss or duplication of task execution.