Audit webhook forwarding

The brain can optionally forward every audit-log row to an external webhook as it is written. It forwards what the audit log holds, which is successful control-plane actions plus a narrow band of denials (write requests against a project's schedule endpoints). Every other authorization denial is never an audit row in the first place, so it can never reach the mirror; pair the forwarder with your reverse-proxy access logs if your SIEM needs refused attempts too. The primary use cases:

SIEM ingest: Splunk HEC, Datadog Logs, Sumo Logic, Elastic. The receiver gets the stored audit-row fields plus the row HMAC.
Out-of-band tamper detection, with the limit below understood: the receiver keeps a copy on a separate trust boundary, so an attacker who compromises the brain's database cannot reach back and rewrite what the receiver already holds. What the forwarder cannot promise is that the receiver holds everything, because it drops rows rather than block the brain. Read Not a substitute for a durable sink before relying on this for evidence.
Compliance evidence: SOC 2 / ISO 27001 auditors often want audit data in a logging stack they already control.

The brain's primary HMAC-chained audit log remains the source of truth. The forwarder is a best-effort mirror -- if the receiver is down, the row is dropped and a metric increments, but the brain's own write succeeds.

Enabling

# In your env file or systemd unit
Z4J_AUDIT_WEBHOOK_URL=https://siem.example.com/ingest
Z4J_AUDIT_WEBHOOK_HMAC_SECRET=<32+ byte random string>

Replace the example with a hostname whose DNS records resolve to public addresses. Private, loopback, link-local, and other blocked address classes are always rejected; the HTTP opt-in changes the permitted scheme, not the destination-address policy.

Restart the brain. On boot you should see:

INFO z4j.brain.domain.audit_forwarder: forwarder started (url=https://siem.example.com/ingest, buffer=1000)

If the URL fails the SSRF pre-flight (for example loopback, RFC1918, or plaintext HTTP without Z4J_NOTIFICATIONS_WEBHOOK_ALLOW_HTTP), a startup WARNING fires and every forwarded row is then dropped at dispatch. There is no private-address bypass. Fix the URL and restart.

Settings

Variable	Default	Notes
`Z4J_AUDIT_WEBHOOK_URL`	unset	Receiver URL. Empty / unset disables the forwarder entirely. SecretStr at the Pydantic layer so a path-embedded token does not land in startup logs.
`Z4J_AUDIT_WEBHOOK_HMAC_SECRET`	unset	REQUIRED when the URL is set. At least 32 bytes. The brain refuses to start if the URL is set without an HMAC secret -- an unauthenticated mirror is worse than no mirror because downstream parsers may trust it implicitly. Mint with `python -c "import secrets; print(secrets.token_urlsafe(48))"`.
`Z4J_AUDIT_WEBHOOK_TIMEOUT_SECONDS`	`10.0`	Per-row POST timeout, range 1.0..120.0. A slow receiver does NOT block the brain's audit write path; the forwarder runs in a background drain task.
`Z4J_AUDIT_WEBHOOK_BUFFER_SIZE`	`1000`	In-memory queue size between the audit-write hook and the drain task. Spikes above this drop rows with a WARNING + a swallowed-exception metric bump. Raise on high-volume brains.

Wire format

The receiver gets a POST request with the row as canonical JSON:

POST /your/path HTTP/1.1
Host: siem.example.com
Content-Type: application/json
X-Z4J-Audit-Signature: sha256=<hex>
X-Z4J-Audit-Timestamp: 1715515200
X-Z4J-Audit-Schema: 1

{"action":"user.password_changed","api_key_id":null,"event_id":null,"id":"...","metadata":{"key":"val"},"occurred_at":"2026-05-12T12:00:00.000000+00:00","outcome":"allow","prev_row_hmac":"...","project_id":null,"result":"success","row_hmac":"...","source_ip":"192.0.2.10","target_id":"user-1","target_type":"user","user_agent":"z4j-cli/1","user_id":"..."}

Fields are emitted in JSON-sorted-keys order so the signature is reproducible. The body matches the brain's internal audit row, plus a row_hmac field. To re-verify that chain in your own pipeline a receiver needs Z4J_AUDIT_CHAIN_SECRET, the dedicated audit-chain key, not Z4J_SECRET. An earlier version of this page named the wrong one, which would not have worked and would have handed an external receiver the master key that derives every agent's frame-signing key and encrypts stored TOTP secrets. Share the audit-chain key alone, and only with a receiver you would trust to verify your audit history.

Verifying the signature

Each POST carries TWO headers:

X-Z4J-Audit-Signature: sha256=<hex> -- the HMAC digest
X-Z4J-Audit-Timestamp: <unix_seconds> -- when the signature was minted

The signature is computed over the bytes <timestamp>.<body>. Folding the timestamp into the HMAC input gives replay-resistance: a captured POST replayed later still has its original signature, but the timestamp is stale so a receiver enforcing a skew window rejects it.

Python receiver (with replay-defence dedupe):

import hmac, hashlib, json, time

SKEW_SECONDS = 300  # 5 minute window; tune to your fleet

# Production: replace this in-memory set with a Redis SETNX or a DB
# unique-index insert. Audit row IDs are UUIDs (~10^-37 collision
# probability), so a permanent dedupe table is bounded by your audit
# retention window.
_seen_ids: set[str] = set()

def verify(body: bytes, headers: dict, secret: bytes) -> bool:
    sig = headers.get("X-Z4J-Audit-Signature", "")
    ts = headers.get("X-Z4J-Audit-Timestamp", "")
    if not sig or not ts:
        return False
    # Reject stale / future timestamps before the constant-time compare
    # so an attacker cannot use the verify call itself as a clock oracle.
    try:
        ts_int = int(ts)
    except ValueError:
        return False
    if abs(int(time.time()) - ts_int) > SKEW_SECONDS:
        return False
    digest_input = ts.encode("utf-8") + b"." + body
    expected = "sha256=" + hmac.new(secret, digest_input, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return False
    # Replay-defence: once the signature verifies, decode the body
    # and reject duplicates on row id. The brain's audit-row id is
    # a UUID, mint-once-per-row. A signed POST replayed inside the
    # 5-minute window has the SAME id; reject it here so the
    # downstream pipeline never inserts the row twice.
    try:
        row_id = json.loads(body).get("id")
    except Exception:
        return False
    if not row_id or row_id in _seen_ids:
        return False
    _seen_ids.add(row_id)
    return True

If verification fails for any reason, reject the request with 401 Unauthorized. Do NOT parse the body before verifying the signature.

HMAC keys across brain replicas

Brain replicas feeding the same receiver normally share one Z4J_AUDIT_WEBHOOK_HMAC_SECRET. The signature covers <timestamp>.<body>, not a brain identity, and the signed audit-row id is the correct deduplication key. A reverse-proxy-injected source header may be useful operational metadata, but it is not cryptographically authenticated by this signature and must not be used as proof of which replica emitted a row.

Use distinct secrets only when replicas send through separately authenticated routes and the receiver can select the right verification key from trusted out-of-band routing context. The wire format itself contains no key ID or signed replica identity.

SSRF protection

Every dispatch runs through the same DNS-pin protection as the generic webhook notification channel:

Scheme must be https:// (or http:// if Z4J_NOTIFICATIONS_WEBHOOK_ALLOW_HTTP=true; this does not permit private destinations)
Hostname resolved to one or more IP addresses
Each IP checked against the blocked set (loopback, RFC1918, link-local, cloud metadata, CGNAT, IPv4-mapped IPv6, 6to4, NAT64, benchmark)
The validated IP is pinned for the TCP connect; Host header + TLS SNI extension stay set to the original hostname so vhost routing and TLS certificate validation still work

A configured URL that resolves to a blocked IP is rejected at dispatch time and the row is dropped with a swallowed-exception metric bump under module=audit_forwarder, site=ssrf_or_dns.

Backpressure

The forwarder owns an asyncio.Queue of size Z4J_AUDIT_WEBHOOK_BUFFER_SIZE. The audit-write hook calls enqueue(row) which is non-blocking:

Queue has space: row is queued, hook returns True.
Queue is full: row is dropped, hook returns False, audit_forwarder.dropped_count increments, a WARNING is logged (deduplicated to every 100 drops).

This is by design. The brain's primary audit-log write must NEVER be slowed down by a misbehaving mirror; the source of truth is in the database. If you see steady-state drops, either raise the buffer size or unblock the receiver.

Observability

The forwarder exposes three counters that show up in the z4j_swallowed_exceptions_total metric under module=audit_forwarder:

site=queue_full -- rows dropped at enqueue because the queue was saturated
site=ssrf_or_dns -- rows dropped at dispatch because the URL failed the SSRF pre-flight (host changed DNS records, IP now blocked)
site=send_one -- rows dropped at dispatch because the receiver returned a non-2xx OR the underlying HTTP call raised

A panel in z4j-notifications.json renders these alongside the notification dispatch counters.

Threat model

The forwarder is an authenticated, best-effort mirror. Append-only retention is a property the receiver must enforce. It does not replace the brain's primary audit log; it complements it. Specifically:

The mirror can lag (the receiver is not consulted on the audit write path).
Individual rows can be dropped (queue saturation, receiver down, DNS change).
An attacker who compromises the brain's HMAC secret can forge rows to the mirror, so the receiver should treat the brain as one of several sources, not as a trusted oracle.

Operators wanting cryptographic non-repudiation should pair the forwarder with a receiver that re-signs every row under its own secret on receipt, so the chain extends beyond the brain's trust boundary.

Not a substitute for a durable sink

The threat model is explicit that a role with write access to the brain's database can delete recent audit rows and roll the authenticated chain state back with them, and that verification then reports the shortened history as clean. The remedy it points to is evidence held outside the database. This forwarder is the obvious thing to reach for, and on its own it will not carry that weight.

The reason is on this page already: rows are dropped when the receiver is unavailable, when its DNS answer changes, and when the queue fills. Those drops are deliberate, because the brain's audit write must not wait on a mirror. The consequence is that a gap in the receiver's copy is not evidence of anything. It could be an outage, a slow receiver, a redeploy -- or the rows an attacker removed. Nothing in the mirror distinguishes them, which is precisely the question you would be asking it.

Evidence meant to survive a compromised database role needs three properties this forwarder does not provide:

Durable. Delivery is retried until it is acknowledged, and unsent rows survive a brain restart. This forwarder buffers in memory and drops on saturation.
Append-only at the sink. The receiver refuses edits and deletions in its own right, under credentials the brain never holds -- an object-store bucket with a retention lock, a WORM log service, an account whose retention policy the brain's operator cannot relax.
Gap-detecting. The receiver can tell "nothing happened" from "something did not arrive". A per-row sequence number or the chain link itself, checked on receipt, turns a silent gap into an alarm.

Two ways to get there, both of which can use this forwarder as one part:

Make the sink authoritative about gaps. Have the receiver track prev_row_hmac continuity and alert on a break or a stall. The forwarder still drops rows; the difference is that a drop now pages someone instead of incrementing a counter nobody reads.
Anchor the chain head, on a schedule, from a job that retries. Export the current head to the durable sink, then verify against it later with z4j audit verify --known-head. This is the smaller and more reliable mechanism, because a single head covers every row beneath it, so one successful export per interval bounds how far the log can be rolled back without detection. See HMAC audit chain.

Use the forwarder for SIEM ingest and for operational visibility, which is what it is good at. Do not let it stand in for the durable, append-only, gap-detecting sink the threat model calls for.

Disabling

Unset Z4J_AUDIT_WEBHOOK_URL and restart. The forwarder is not constructed when no URL is set; no background task starts, no queue is allocated.