Sentry

z4j brain ships an optional Sentry hook. With a DSN configured, unhandled exceptions inside HTTP handlers, background workers, and domain code are captured to a Sentry project of your choosing. The integration is off by default, opt-in via a single env var, and runs every event it sends (errors, log events and performance transactions) through a redaction pass that strips Authorization headers, cookies, webhook URLs, and OAuth-style query tokens before the SDK ships them.

What the brain captures

Source	Without Sentry	With Sentry
Unhandled exception in an HTTP handler	500 response, structured stdout log	Same, plus a Sentry event built from that log record, with the traceback rendered into its message
Background worker crash (registry, audit retention, schedule fires)	Structlog error event	Same, plus a Sentry event; the brain adds no worker-name tag of its own
logger.exception(...) in domain code	Structlog error event	Same, plus Sentry event with the call site
Successful requests	Counted in Prometheus, no log line	Optionally captured as transactions when Z4J_SENTRY_TRACES_SAMPLE_RATE > 0, redacted like every other event

Migrations (alembic upgrade head) and one-shot CLI commands (z4j init, z4j audit verify, z4j reset-mfa) deliberately do NOT init Sentry. Failures on those paths already land on the operator's terminal and in the audit log; adding a network round trip for every CLI invocation would be net negative.

Enabling

Install the optional dependency, then set the DSN:

pip install 'z4j[sentry]'

# In your env file or systemd unit
Z4J_SENTRY_DSN=https://<public-key>@<instance>/<project-id>

Restart the brain. On boot the JSON log carries a line like this:

{"event": "z4j observability.sentry: Sentry initialised (environment=production, release=z4j@<version>, traces=0.000, profiles=0.000)", "level": "info", "timestamp": "<UTC ISO 8601>"}

If sentry-sdk is not installed when the DSN is set, the brain logs a single WARNING explaining what to install and continues running without Sentry. There is no fallback retry, no init backoff, and no exception that propagates out of create_app.

Settings

All settings are prefixed Z4J_ and read from your env file or the process environment.

Variable	Default	Notes
Z4J_SENTRY_DSN	unset	When unset OR empty, every other knob below is ignored. SecretStr at the Pydantic layer, so the value is never echoed in startup logs or in a Pydantic validation traceback.
Z4J_SENTRY_ENVIRONMENT	unset	Override the environment tag Sentry attaches to events. Defaults to Z4J_ENVIRONMENT (production / staging / dev). Use this when several brains route into one Sentry project and you want to distinguish them (e.g. staging-eu vs staging-us).
Z4J_SENTRY_TRACES_SAMPLE_RATE	0.0	Fraction of requests captured as transactions, in [0.0, 1.0]. Default 0 keeps Sentry on error-only mode. 0.05 is a reasonable opening bid; raise once you have a feel for event volume + your Sentry quota.
Z4J_SENTRY_PROFILES_SAMPLE_RATE	0.0	Fraction of in-transaction code that is profiled, in [0.0, 1.0]. Bounded above by Z4J_SENTRY_TRACES_SAMPLE_RATE (no transaction, no profile). Leave at 0 unless traces are already on.
Z4J_SENTRY_SEND_DEFAULT_PII	false	Forward identifying data (IP, username, raw cookies) to Sentry. Default false. The brain's redaction pass strips Authorization headers, cookies, OAuth tokens, and webhook URLs from every event even when this is on, performance transactions included, so flipping it on still leaves credentials redacted.

Out-of-range sample rates fail validation at startup. A typo like Z4J_SENTRY_TRACES_SAMPLE_RATE=1.5 raises a Pydantic ValidationError before the FastAPI app is built; you see the error on the terminal, not at runtime.

What gets redacted

Every error, log and transaction event is passed through z4j_brain.observability.sentry.scrub_event before reaching the SDK transport. The SDK calls before_send for error and log events only, so the brain registers the scrubber as both before_send and before_send_transaction. The scrubber strips:

• Request headers (case-insensitive): Authorization, Proxy-Authorization, Cookie, Set-Cookie, X-Z4J-Signature, X-Z4J-Audit-Signature, X-Z4J-API-Key, X-API-Key, X-Auth-Token, X-CSRF-Token, X-CSRFToken, plus the IP-chain set X-Forwarded-For / X-Real-IP / Forwarded / CF-Connecting-IP / True-Client-IP / Fastly-Client-IP / X-Cluster-Client-IP / Remote-User. The header name stays so the Sentry event still shows "Authorization was set"; the value is replaced with [REDACTED by z4j].
• Request env (CGI-style): request.env dict from the WSGI/ASGI integration. HTTP_* keys map back to header names and are redacted via the same allowlist. Operator-added keys whose name matches a credential pattern (password, secret, token, api_key, etc.) have their values redacted.
• Query parameters: token, access_token, refresh_token, id_token, api_key, apikey, key, secret, password, code (covers OAuth callbacks and MFA verification codes), signature, sig, session, csrf. Values stripped, parameter names kept.
• Request cookies: the entire request.cookies block, wholesale.
• Request body: the entire request.data block, wholesale. Webhook payloads outbound from the notification dispatchers can carry workflow tokens, and the brain prefers a coarse strip over a per-channel allowlist.
• User block (event["user"]): email, username, ip_address are redacted even when send_default_pii=true so the brain's PII-redaction promise holds. The id field (a UUID) is kept so Sentry issue-grouping works.
• logentry: event["logentry"].message, formatted, and params (dict OR positional list). Top-level event["message"] is scrubbed identically.
• Exception values and stacktraces: every event["exception"]["values"][i]["value"] (the str(exc), which routinely includes the offending URL) is scrubbed. Each stacktrace frame is walked: vars (locals) via the value-key pattern; context_line / pre_context / post_context (the surrounding source lines) via URL scrubbing; filename / abs_path / module redacted when the path itself matches a credential pattern.
• Threads: same stacktrace shape as exception values; same scrubber pass.
• Transaction: when the transaction string is a raw URL (unmatched route case) it is URL-scrubbed.
• Spans + contexts.trace: a span's description goes through the free-form URL scrubber described below (an outbound HTTP span is described as <METHOD> <url>). The URL attributes data.url, data.http.url and data.url.full are URL-scrubbed, and the query-string attributes data.http.query and data.url.query have the query keys above scrubbed. The contexts.trace block gets the same treatment for its description and data.
• Outbound URLs in breadcrumbs (the SDK's httpx integration records one per request): data.url and the breadcrumb message go through the URL scrubber, and data.http.query through the query-key scrubber. The URL scrubber replaces the whole path with /[REDACTED by z4j] and drops the query when the host is a token-in-path host (Slack, Discord, Microsoft Teams, PagerDuty), or when the path has a credential-bearing shape on any host: /services/<id>/<id>/... (Slack), /workflows/... and /triggers/... (Slack workflows, Azure Logic Apps), /api/webhooks/... (Discord), /webhookb2/... and /webhook/... (Teams connectors), /bot<id>:... (Telegram), /integration/<key>/enqueue (PagerDuty). For every other URL the query string is re-scrubbed with the same key set above and the host and path stay visible. URLs embedded in free-form text get the same treatment (a token-in-path host URL becomes <scheme>://<host>/[REDACTED by z4j]), but only in stack-frame context lines, exception values, span descriptions and the transaction string; breadcrumb messages and log entries get the whole-string URL scrubber only.
• extra / tags / contexts blocks: any key matching password, passwd, _pass, secret, token, api_key, auth, signature, private_key, bot_token, webhook_url, integration_key, recovery_code, or mfa_secret (substring, case-insensitive) has its value redacted. Nested dicts and lists of dicts are walked up to 32 levels deep; deeper structures are replaced with [REDACTED by z4j] (a hostile event with 1000-deep nesting cannot crash the scrubber and bypass redaction by recursion).

The redactor is a pure function, exercised by tests/unit/test_sentry_observability.py. The tests check every header and query key in the current allowlists and pin the most sensitive ones by name, so dropping one of those fails a test; dropping any other entry shrinks the checked set instead.

Webhook URL caveat

The Slack, Discord, and Microsoft Teams channels store their destinations in a webhook_url config field, and the Telegram channel stores a bot_token. The scrubber redacts those values wherever they appear under a key matching webhook_url or bot_token. When a dispatcher sends, the SDK records the request URL in a breadcrumb, and in a span when the send runs inside a sampled transaction (a channel test from the dashboard, for example); send_default_pii does not gate either. The dispatchers connect to the IP they resolved, so that URL names the IP rather than the webhook host, and the path shapes listed above are what redact it. They cover every URL the built-in channels send to; the PagerDuty channel carries its routing key in the request body, not the URL. A generic webhook, or the audit webhook, whose secret sits in a path the shapes do not match keeps that path in breadcrumb and span URLs and has only its query string scrubbed, so put such a secret in a header or under one of the query keys above.

Release tags

The brain reads its installed package version and uses z4j@<version> as the Sentry release tag. This is what powers Sentry's "first seen in" attribution and lets you correlate a spike of issues with a specific deploy. If the package version cannot be resolved (e.g. an editable install in a packaging mode that does not register metadata), the release tag is omitted and the boot line reads release=unset; Sentry handles that fine but you lose the issue-to-release correlation.

Disabling

Unset Z4J_SENTRY_DSN and restart the brain. The SDK does not need to be uninstalled; an unset DSN is a complete no-op. No further outbound connections are attempted.

Workers

Adapter-side workers (z4j-celery, z4j-django, etc.) do NOT initialise their own Sentry client; only the brain process does. If you want Sentry on the worker side, install sentry-sdk in the worker's venv and call sentry_sdk.init(...) from your worker entrypoint. The brain's scrubber does not run on that path, so review what your worker sends.