Audit exports and the chain head anchor
The synchronous export on the audit page refuses result sets above its in-memory caps (50,000 rows for CSV and JSON, 25,000 for XLSX). A brain that writes more than a few hundred audit rows a day keeps a trail the download cannot return whole. Export jobs remove that limit: the dashboard or the API queues a job, the brain's export-jobs worker streams the rows page by page into an export sink, and the trail of any size lands as one object an auditor can take away.
The same sink serves a second purpose. The HMAC chain cannot, on its own,
prove anything against a role that can write both audit_log and
audit_chain_state (see HMAC audit chain).
The documented mitigation is a chain head recorded somewhere that role
cannot reach, checked later with z4j audit verify --known-head. With a
sink configured, the brain writes that head on a schedule by itself.
Configuring a sink
Section titled “Configuring a sink”Set Z4J_EXPORT_SINK to local. The default, none, disables both
exports and the head export: the API answers 409 to a job request and the
worker is not started.
Local directory
Section titled “Local directory”Z4J_EXPORT_SINK=localZ4J_EXPORT_SINK_PATH=/var/lib/z4j/exportsThe directory must exist and must not be a symlink. Objects are written
through a temporary file in the same directory and renamed into place, so
a reader never sees a partial file and a failed job leaves nothing behind.
Files are created with mode 0640, and the sink refuses to write through
a symlink at any level below the base, so an unprivileged account on the
same host cannot redirect an export. Jobs land under
audit/<project-slug>/<timestamp>-<job-id>.<format>.
A finished local-sink job can be downloaded from the dashboard or from
GET /api/v1/projects/{slug}/audit/export-jobs/{job_id}/download by an
auditor or an admin of the project. The job row records the object's key
below the directory, not a path, and the download does not trust the row
to name the file: it rebuilds the path from Z4J_EXPORT_SINK_PATH, refuses
any step that is .. or otherwise not a plain key, resolves both the
directory and the file and requires the one to be inside the other, follows
no symlink at any level, and serves the handle it opened. A row edited to
point anywhere else answers 404, so a database role that can write
export_jobs cannot turn the download into a read of another file on the
host. The location the API reports is still the full path, rebuilt from
the directory in effect.
Shipping exports to other storage
Section titled “Shipping exports to other storage”The brain writes to the directory and nowhere else. Getting the files to
object storage, another host or an offsite backup is your own tool's job,
run on its own schedule against Z4J_EXPORT_SINK_PATH:
# for example, rclone to any remote it supportsrclone copy /var/lib/z4j/exports remote:z4j-evidence/z4j-exportsFor the head anchor, copy to storage that the brain's database role cannot shorten: a destination with retention or immutability turned on, or at least versioning, written by credentials that can add and not delete. The point of the anchor is that the role which can rewrite the audit tables cannot also rewrite the anchor; a copy the same host can delete is better than nothing and worse than one it cannot.
Queueing and reading a job
Section titled “Queueing and reading a job”From the dashboard, the audit page's Exports panel queues a job in the format picked from its Queue export menu, with the current filters, and lists the 20 most recent jobs with their status, size, sink location and, for the local sink, a download button.
From the API:
# Queue (auditor or admin on the project; API keys need audit:read)curl -X POST "$BRAIN/api/v1/projects/default/audit/export-jobs" \ -H "Authorization: Bearer $Z4J_API_KEY" -H "Content-Type: application/json" \ -d '{"format": "csv", "since": "2026-01-01T00:00:00Z"}'
# Pollcurl "$BRAIN/api/v1/projects/default/audit/export-jobs/$JOB_ID" \ -H "Authorization: Bearer $Z4J_API_KEY"A job moves from queued to running and then to done or failed.
While running, row_count advances after every page; when done, location
names the object (a path) and size_bytes its
length; when failed, error says why. Filters, columns and bytes match the
synchronous export for the same request. Every creation, completion and
failure is an audit row of its own (audit.export_job.created,
audit.export_job.completed, audit.export_job.failed).
What the worker does
Section titled “What the worker does”The export-jobs worker runs in every brain process that has a sink and is
leader-gated like the other periodic workers, so one replica drains the
queue. It polls every Z4J_EXPORT_JOBS_POLL_INTERVAL_SECONDS (default 5)
and claims jobs oldest first. Rows are fetched Z4J_EXPORT_JOBS_PAGE_SIZE
at a time (default 2000) and encoded as they arrive; the whole result is
never in memory. XLSX is spooled to a private temporary file in constant
memory and then streamed to the sink, which is why its only bound is the
worksheet format's row limit.
A job whose row has not moved for fifteen minutes belongs to a process that is gone (progress bumps the row on every page); the next tick marks it failed with that reason, and it can be queued again. Should the process turn out not to be gone (a replica that lost its leader lock mid-write but kept writing), the failure that was recorded stands: the late completion appends nothing to the audit trail and removes the file it wrote rather than leaving an orphan no row refers to. A sink that cannot be written fails the job with the error text and is reported in the log; it never stops the brain.
Scheduled head export
Section titled “Scheduled head export”Z4J_AUDIT_HEAD_EXPORT_INTERVAL_SECONDS=3600With a sink configured and the audit-chain key set, the worker writes the
current authenticated chain head to the sink on that cadence (at least 60
seconds; 0, the default, disables it). Two objects are written each time:
a stable key, audit-head/current.json, that a check can always read, and a
dated copy beside it, audit-head/<timestamp>.json, so a copy kept with
retention or versioning holds every head ever anchored. The envelope is byte for
byte what z4j audit export-head prints: six keys, canonical JSON, one
trailing newline, authenticated against the configured keyring before it is
written, and refused (logged, not written) when the state row does not
authenticate.
Check the chain against the anchor the same way the CLI documents, reading the file the worker wrote:
z4j audit verify --known-head "$(cat /var/lib/z4j/exports/audit-head/current.json)"Check against the copy your tool shipped off the host, not only the one beside the brain, since the off-host copy is the one the database role cannot reach.
CURRENT_MATCH means the chain still ends at the anchored head;
VERIFIED_ANCESTOR means it has grown past it; UNPROVABLE means the log
no longer contains it, which is the finding this anchor exists to produce.
No audit row is written for a head export, because writing one would move
the head the export just anchored.
The head export authenticates the state row; it does not walk every row
first. Pair it with Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true for the scheduled
walk, so a chain that fails to verify is noticed within an interval rather
than anchored as it stands.
Settings
Section titled “Settings”| Variable | Default | Description |
|---|---|---|
Z4J_EXPORT_SINK |
none |
none or local. |
Z4J_EXPORT_SINK_PATH |
- | Directory for the local sink. Required with local. |
Z4J_EXPORT_JOBS_POLL_INTERVAL_SECONDS |
5 |
Queue poll cadence, 1 to 300. |
Z4J_EXPORT_JOBS_PAGE_SIZE |
2000 |
Rows fetched per page while streaming, 100 to 20000. |
Z4J_AUDIT_HEAD_EXPORT_INTERVAL_SECONDS |
0 |
Head export cadence; 0 off, otherwise 60 to 604800. Needs a sink. |
Troubleshooting
Section titled “Troubleshooting”409 no export sink is configured: setZ4J_EXPORT_SINKand restart the brain.the s3 export sink was removed(logged once at startup, the reason on every job, and409to a new job): the environment still setsZ4J_EXPORT_SINK=s3. Set it tolocalwithZ4J_EXPORT_SINK_PATH, and copy the directory to your storage with your own tool. TheZ4J_EXPORT_SINK_S3_*variables are ignored and can be deleted. Jobs the S3 sink wrote earlier still list with theirs3://location.export path is missing or unsafe(orexport directory is missing or unsafewhen the brain starts): the local sink found a symlink or a missing directory at or below its base directory and stopped; replace it with a real directory.export target is not a regular fileis the same refusal for the file itself.- A job stays
queued: no replica holds the export-jobs leader lock, or the worker is not started becauseZ4J_EXPORT_SINKisnoneon the replica you are looking at; check the startup log's worker list. download is served for the local sink only: the job was written somewhere the local sink does not serve (an older S3 job, or a job from before the sink changed); fetch the object at the job'slocation.