OpenTelemetry

z4j brain ships an optional OpenTelemetry hook. With an OTLP endpoint configured, FastAPI HTTP server requests, SQLAlchemy queries, and outbound httpx calls are traced and exported to a collector of your choosing. The integration is off by default and opt-in via a single env var. The SDK is loaded lazily, so a missing package or an SDK failure cannot prevent boot; a malformed setting is refused at startup like any other.

What gets traced

Source	Span kind	Default sample rate
FastAPI HTTP requests	server	0 (set Z4J_OTEL_TRACES_SAMPLER_ARG=0.05 to collect 5%)
SQLAlchemy queries against the brain's primary engine	client (DB)	Inherits the parent span's sampling decision; with no parent span (background workers) the brain's own ratio applies
Outbound httpx calls (notification dispatchers, audit forwarder, version check)	client (HTTP)	Inherits the parent span's sampling decision; with no parent span (background workers) the brain's own ratio applies
WebSocket dispatch, command issuance, task ingestion	no dedicated spans	The agent HTTP endpoints and the WebSocket connections appear only as generic server spans. The wire protocol carries no trace-context header, so cross-process spans would not connect.

/health* and /metrics are excluded by default. They carry too much background traffic for any sampling budget to be meaningful; tracing them swamps the operator's collector with noise. Flip Z4J_OTEL_INCLUDE_HEALTH=true to re-enable.

Enabling

Install the optional dependency, then set the endpoint:

pip install 'z4j[otel]'

# In your env file or systemd unit
Z4J_OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io/v1/traces
Z4J_OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=<your-api-key>
Z4J_OTEL_TRACES_SAMPLER_ARG=0.05

Restart the brain. On boot the JSON log carries a line like this, with the endpoint reduced to its scheme, host and port:

{"event": "z4j observability.otel: OpenTelemetry initialised (endpoint=https://api.honeycomb.io, protocol=http/protobuf, sampler_arg=0.050, include_health=False)", "level": "info", "timestamp": "<UTC ISO 8601>"}

If the SDK is not installed when the endpoint is set, the brain logs a single WARNING explaining what to install and continues running without OTel.

Settings

All settings are prefixed Z4J_ and read from your env file or the process environment.

Variable	Default	Notes
Z4J_OTEL_EXPORTER_OTLP_ENDPOINT	unset	When unset OR empty, OpenTelemetry stays off and the other settings below have no effect (they are still validated at startup). SecretStr at the Pydantic layer, so it is masked in settings dumps, and the initialised log line prints only its scheme, host and port.
Z4J_OTEL_PROTOCOL	http/protobuf	One of http/protobuf, http (alias), grpc. gRPC additionally needs pip install opentelemetry-exporter-otlp-proto-grpc.
Z4J_OTEL_EXPORTER_OTLP_HEADERS	unset	Comma-separated key=value pairs in the OpenTelemetry OTEL_EXPORTER_OTLP_HEADERS format. Keys and values are URL-decoded and trimmed (write a comma inside a value as %2C), keys are lowercased, and the result is passed to the exporter as its headers for either protocol. A value that does not parse (an entry without =, an empty or invalid header name, a control character) refuses startup with an error that names the setting and the entry's position but never prints the value. The standard place to set x-honeycomb-team, authorization, etc. SecretStr.
Z4J_OTEL_SERVICE_NAME	z4j-brain	Resource attribute service.name. Multi-brain deployments set this to distinguish them in the collector UI.
Z4J_OTEL_SERVICE_NAMESPACE	z4j	Resource attribute service.namespace. Groups every z4j service together in the collector.
Z4J_OTEL_ENVIRONMENT	unset	Resource attribute deployment.environment. Defaults to Z4J_ENVIRONMENT (production / staging / dev).
Z4J_OTEL_TRACES_SAMPLER_ARG	0.0	TraceIdRatioBased sampler argument, 0.0..1.0. Default 0 = no traces sampled. Wrapped in ParentBased(remote_parent_sampled=ALWAYS_OFF, remote_parent_not_sampled=ALWAYS_OFF): a request that carries an inbound traceparent header is never recorded, so a spoofed header cannot force sampling; requests without one are sampled at this ratio. If your proxy or service mesh injects traceparent, strip it before the brain or no requests are traced. Trace context does not propagate across processes.
Z4J_OTEL_INCLUDE_HEALTH	false	When true, the whole default exclude list is dropped, so /health*, /metrics and /api/v1/auth are traced like any other endpoint. Default false. The default list is /health, /api/v1/health, /api/v1/health/, /metrics, /api/v1/auth (auth routes carry credentials in POST bodies). Re-add /api/v1/auth through Z4J_OTEL_EXCLUDED_URL_PATTERNS to keep it out.
Z4J_OTEL_EXCLUDED_URL_PATTERNS	empty	Comma-separated entries to additionally exclude, layered on top of the default list. Each entry is a regular expression searched anywhere in the request URL, so a plain path matches as a substring.

Out-of-range sampler args fail validation at startup. A typo like Z4J_OTEL_TRACES_SAMPLER_ARG=1.5 raises a Pydantic ValidationError before the FastAPI app is built, and a malformed Z4J_OTEL_EXPORTER_OTLP_HEADERS raises a ConfigError at the same point. z4j config validate reports either one when the value is in the file it checks.

Resource attributes

Every span carries:

service.name        = z4j-brain (or your override)
service.namespace   = z4j (or your override)
service.version     = <z4j package version>     (omitted on editable installs without metadata)
deployment.environment = <Z4J_ENVIRONMENT>      (or otel_environment override)

The build_resource_attributes helper is exposed and unit-tested so the attribute set is pinned at the source rather than the collector side.

Collectors

Example endpoint shapes:

• Honeycomb: https://api.honeycomb.io/v1/traces + x-honeycomb-team header.
• Grafana Tempo: https://tempo-prod-04-prod-us-east-0.grafana.net/tempo + basic-auth header. HTTP or gRPC.
• Local Jaeger (jaegertracing/all-in-one:latest): http://localhost:4318/v1/traces (HTTP) or localhost:4317 (gRPC).
• Local OpenTelemetry Collector (otel/opentelemetry-collector-contrib): http://localhost:4318/v1/traces.

The brain ships traces over OTLP and nothing else. Metrics do not export over OTLP: the Prometheus /metrics endpoint is the canonical metric surface, and dual-exporting would only create reconciliation work.

Threat model

The OTLP exporter ships span attributes (URLs, DB statement fragments, response codes, header names if the instrumentation captures them) to a collector outside the brain. By default the brain makes no outbound traffic except to configured notification, audit and export destinations and the version check an admin triggers by hand; enabling OTel adds the collector. Review what the auto-instrumentations attach BEFORE pointing at a multi-tenant collector. In particular:

• The FastAPI instrumentation attaches the request path and method. Path parameters become span attributes; if you embed an opaque token in the URL it will appear in spans. Move it to a header.

• The SQLAlchemy instrumentation runs with enable_commenter=False so it does not embed SQL comments in your queries. The statement text itself is captured; sensitive queries (the brain has none by default, but operator-added schema might) leak by name.

• The httpx instrumentation captures outbound URLs. Slack, Discord and Teams webhooks and the Telegram bot API carry credentials in the URL path; without scrubbing, the OTLP exporter ships those URLs to whatever collector you configure. The brain registers its scrubber as all four httpx instrumentation hooks: request_hook and response_hook for httpx.Client, and async_request_hook and async_response_hook for httpx.AsyncClient, which the notification dispatchers and the audit forwarder use. It rewrites http.url and url.full to <scheme>://<host>/[redacted by z4j], sets http.target and url.path to /[redacted by z4j], and blanks url.query when either of these holds:

  • A hostname of the request matches the credential-bearing set: outlook.office.com, *.webhook.office.com, *.logic.azure.com, hooks.slack.com, discord.com, discordapp.com, *.slack.com, *.discordapp.com, *.discord.com, *.pagerduty.com, the exact host events.pagerduty.com, and the host of Z4J_AUDIT_WEBHOOK_URL when that is set, registered when OTel initialises. The dispatchers connect to the IP they resolved, so the URL host is an IP; the hook also checks the Host header and TLS SNI name they send, and <host> in the redacted URL is the hostname they dialled.
  • The path has a credential-bearing shape, whatever the host: /services/<id>/<id>/... (Slack), /workflows/... and /triggers/... (Slack workflows, Azure Logic Apps), /api/webhooks/... (Discord), /webhookb2/... and /webhook/... (Teams connectors), /bot<id>:... (Telegram), /integration/<key>/enqueue (PagerDuty).

  The hook is fail-closed: if the scrubber itself raises (a future SDK upgrade renames request.url etc.) the URL is overwritten to a generic https://[unknown]/[redacted by z4j] marker and a WARNING is logged, so the unscrubbed URL never reaches the OTLP exporter. Any outbound URL that matches neither rule is exported as the httpx instrumentation records it, so a secret in the path or query of a generic webhook URL can reach the collector; put such a secret in a header.

Disabling

Unset Z4J_OTEL_EXPORTER_OTLP_ENDPOINT and restart. The SDK does not need to be uninstalled; an unset endpoint is a complete no-op.

Workers and the scheduler

Adapter-side workers (z4j-celery, z4j-django, etc.) and the scheduler companion do NOT initialise their own OTel SDK; only the brain process does. Cross-process trace context propagation, so a span starting on an agent connects to a span on the brain, would need a wire-protocol header that the protocol does not carry.