Skip to content

Audit exports and the chain head anchor

The synchronous export on the audit page refuses result sets above its in-memory caps (50,000 rows for CSV and JSON, 25,000 for XLSX). A brain that writes more than a few hundred audit rows a day keeps a trail the download cannot return whole. Export jobs remove that limit: the dashboard or the API queues a job, the brain's export-jobs worker streams the rows page by page into an export sink, and the trail of any size lands as one object an auditor can take away.

The same sink serves a second purpose. The HMAC chain cannot, on its own, prove anything against a role that can write both audit_log and audit_chain_state (see HMAC audit chain). The documented mitigation is a chain head recorded somewhere that role cannot reach, checked later with z4j audit verify --known-head. With a sink configured, the brain writes that head on a schedule by itself.

Set Z4J_EXPORT_SINK to local. The default, none, disables both exports and the head export: the API answers 409 to a job request and the worker is not started.

Terminal window
Z4J_EXPORT_SINK=local
Z4J_EXPORT_SINK_PATH=/var/lib/z4j/exports

The directory must exist and must not be a symlink. Objects are written through a temporary file in the same directory and renamed into place, so a reader never sees a partial file and a failed job leaves nothing behind. Files are created with mode 0640, and the sink refuses to write through a symlink at any level below the base, so an unprivileged account on the same host cannot redirect an export. Jobs land under audit/<project-slug>/<timestamp>-<job-id>.<format>.

A finished local-sink job can be downloaded from the dashboard or from GET /api/v1/projects/{slug}/audit/export-jobs/{job_id}/download by an auditor or an admin of the project. The job row records the object's key below the directory, not a path, and the download does not trust the row to name the file: it rebuilds the path from Z4J_EXPORT_SINK_PATH, refuses any step that is .. or otherwise not a plain key, resolves both the directory and the file and requires the one to be inside the other, follows no symlink at any level, and serves the handle it opened. A row edited to point anywhere else answers 404, so a database role that can write export_jobs cannot turn the download into a read of another file on the host. The location the API reports is still the full path, rebuilt from the directory in effect.

The brain writes to the directory and nowhere else. Getting the files to object storage, another host or an offsite backup is your own tool's job, run on its own schedule against Z4J_EXPORT_SINK_PATH:

Terminal window
# for example, rclone to any remote it supports
rclone copy /var/lib/z4j/exports remote:z4j-evidence/z4j-exports

For the head anchor, copy to storage that the brain's database role cannot shorten: a destination with retention or immutability turned on, or at least versioning, written by credentials that can add and not delete. The point of the anchor is that the role which can rewrite the audit tables cannot also rewrite the anchor; a copy the same host can delete is better than nothing and worse than one it cannot.

From the dashboard, the audit page's Exports panel queues a job in the format picked from its Queue export menu, with the current filters, and lists the 20 most recent jobs with their status, size, sink location and, for the local sink, a download button.

From the API:

Terminal window
# Queue (auditor or admin on the project; API keys need audit:read)
curl -X POST "$BRAIN/api/v1/projects/default/audit/export-jobs" \
-H "Authorization: Bearer $Z4J_API_KEY" -H "Content-Type: application/json" \
-d '{"format": "csv", "since": "2026-01-01T00:00:00Z"}'
# Poll
curl "$BRAIN/api/v1/projects/default/audit/export-jobs/$JOB_ID" \
-H "Authorization: Bearer $Z4J_API_KEY"

A job moves from queued to running and then to done or failed. While running, row_count advances after every page; when done, location names the object (a path) and size_bytes its length; when failed, error says why. Filters, columns and bytes match the synchronous export for the same request. Every creation, completion and failure is an audit row of its own (audit.export_job.created, audit.export_job.completed, audit.export_job.failed).

The export-jobs worker runs in every brain process that has a sink and is leader-gated like the other periodic workers, so one replica drains the queue. It polls every Z4J_EXPORT_JOBS_POLL_INTERVAL_SECONDS (default 5) and claims jobs oldest first. Rows are fetched Z4J_EXPORT_JOBS_PAGE_SIZE at a time (default 2000) and encoded as they arrive; the whole result is never in memory. XLSX is spooled to a private temporary file in constant memory and then streamed to the sink, which is why its only bound is the worksheet format's row limit.

A job whose row has not moved for fifteen minutes belongs to a process that is gone (progress bumps the row on every page); the next tick marks it failed with that reason, and it can be queued again. Should the process turn out not to be gone (a replica that lost its leader lock mid-write but kept writing), the failure that was recorded stands: the late completion appends nothing to the audit trail and removes the file it wrote rather than leaving an orphan no row refers to. A sink that cannot be written fails the job with the error text and is reported in the log; it never stops the brain.

Terminal window
Z4J_AUDIT_HEAD_EXPORT_INTERVAL_SECONDS=3600

With a sink configured and the audit-chain key set, the worker writes the current authenticated chain head to the sink on that cadence (at least 60 seconds; 0, the default, disables it). Two objects are written each time: a stable key, audit-head/current.json, that a check can always read, and a dated copy beside it, audit-head/<timestamp>.json, so a copy kept with retention or versioning holds every head ever anchored. The envelope is byte for byte what z4j audit export-head prints: six keys, canonical JSON, one trailing newline, authenticated against the configured keyring before it is written, and refused (logged, not written) when the state row does not authenticate.

Check the chain against the anchor the same way the CLI documents, reading the file the worker wrote:

Terminal window
z4j audit verify --known-head "$(cat /var/lib/z4j/exports/audit-head/current.json)"

Check against the copy your tool shipped off the host, not only the one beside the brain, since the off-host copy is the one the database role cannot reach.

CURRENT_MATCH means the chain still ends at the anchored head; VERIFIED_ANCESTOR means it has grown past it; UNPROVABLE means the log no longer contains it, which is the finding this anchor exists to produce. No audit row is written for a head export, because writing one would move the head the export just anchored.

The head export authenticates the state row; it does not walk every row first. Pair it with Z4J_AUDIT_CHAIN_VERIFY_ENABLED=true for the scheduled walk, so a chain that fails to verify is noticed within an interval rather than anchored as it stands.

Variable Default Description
Z4J_EXPORT_SINK none none or local.
Z4J_EXPORT_SINK_PATH - Directory for the local sink. Required with local.
Z4J_EXPORT_JOBS_POLL_INTERVAL_SECONDS 5 Queue poll cadence, 1 to 300.
Z4J_EXPORT_JOBS_PAGE_SIZE 2000 Rows fetched per page while streaming, 100 to 20000.
Z4J_AUDIT_HEAD_EXPORT_INTERVAL_SECONDS 0 Head export cadence; 0 off, otherwise 60 to 604800. Needs a sink.
  • 409 no export sink is configured: set Z4J_EXPORT_SINK and restart the brain.
  • the s3 export sink was removed (logged once at startup, the reason on every job, and 409 to a new job): the environment still sets Z4J_EXPORT_SINK=s3. Set it to local with Z4J_EXPORT_SINK_PATH, and copy the directory to your storage with your own tool. The Z4J_EXPORT_SINK_S3_* variables are ignored and can be deleted. Jobs the S3 sink wrote earlier still list with their s3:// location.
  • export path is missing or unsafe (or export directory is missing or unsafe when the brain starts): the local sink found a symlink or a missing directory at or below its base directory and stopped; replace it with a real directory. export target is not a regular file is the same refusal for the file itself.
  • A job stays queued: no replica holds the export-jobs leader lock, or the worker is not started because Z4J_EXPORT_SINK is none on the replica you are looking at; check the startup log's worker list.
  • download is served for the local sink only: the job was written somewhere the local sink does not serve (an older S3 job, or a job from before the sink changed); fetch the object at the job's location.