Issues and failure fingerprints

A thousand failed task runs are usually a handful of actual bugs. z4j fingerprints every task failure and groups tasks by fingerprint into issues: one row per distinct logical failure on a project, with its occurrence count, whether it is still happening, and which engines it spans.

What a fingerprint is

A fingerprint is a stable hash of a failure's shape:

• the exception string as the adapter reports it (a bare class name from some adapters, the class with a summary from others; z4j does not hash a separate message field),
• plus the deepest five traceback frames, each reduced to basename:line:function (the file's base name, not its full path).

The deepest frames are used because they are the most distinctive; the top of a deep stack is mostly framework boilerplate. A long traceback can be cut before it is hashed. z4j-celery sends at most the last 4096 characters of a traceback, which keeps the innermost frames and the exception line. Redaction keeps the first 8192 bytes of any longer value, so for a longer traceback from another adapter the fingerprint uses the deepest frames that survived the cut, not the frames nearest the raise. Volatile noise is stripped before hashing, so the same bug cannot splinter into many fingerprints: hex memory addresses, UUIDs, object reprs, and bare long numeric ids are all normalised away. A traceback with no parseable Python frames falls back to hashing its noise-stripped text.

The result is a 32-hex-character value, computed once when the task.failed event is ingested and stored on the task row. It is pure and deterministic: the same failure hashes identically across processes, restarts, and brain replicas.

Line numbers are deliberately kept: a code change that shifts the failing line produces a new fingerprint. A new deploy's bug is a new issue, even when the exception class is the same.

Open vs recovered

A task keeps its fingerprint from its most recent failure, even after a later retry succeeds. That split is what makes an issue's status meaningful:

• ongoing: at least one task carrying the fingerprint is currently in the FAILURE state.
• recovered: every task that ever hit the fingerprint has since succeeded (or otherwise left the failure state).

An issue's first/last-seen timestamps are anchored on when the tasks failed, not on when they finished, so an old failure that recovered recently does not masquerade as a fresh problem.

The issues endpoint

GET /api/v1/projects/{slug}/issues

Role: viewer. An issue is operational data about tasks a member can already read (no who-did-what), so any project member can list them.

Query params: engine, status (ongoing | recovered), hours (time window, 1..8760), cursor, limit (default 50, max 200).

{
  "items": [
    {
      "fingerprint": "9f2c4a1e8b7d63051a2f4c8e9b0d7a31",
      "status": "ongoing",
      "occurrences": 412,
      "open_count": 37,
      "recovered_count": 375,
      "first_seen": "...",
      "last_seen": "...",
      "engine_count": 2,
      "engines": ["celery", "rq"],
      "sample_exception": "IntegrityError",
      "sample_task_name": "billing.close_invoice"
    }
  ],
  "next_cursor": null
}

occurrences counts the tasks that hit the fingerprint; open_count is how many of them are failing right now. Issues are ordered by last_seen, newest first. The engine and hours filters apply to the aggregation itself, so the counts and the engines list always agree with the filter.

Fingerprints as automation conditions

fingerprint is an exact-match condition key in automation rules. That turns "this one known bug keeps failing" into a precise, narrow trigger:

1. List the project's issues and copy the fingerprint of the failure you care about.
2. Create a task.failed rule with a fingerprint condition set to that value, and the action you want (notify, or a governed retry for a known-transient failure).
3. Arm it as dry-run first and watch the audit log to confirm it matches exactly the events you expect.

Because the rule keys on the noise-stripped failure shape rather than on task name or exception substrings, it will not fire on unrelated failures that happen to share a task name, and it will stop matching when a deploy moves the failing line (new fingerprint, new issue), which is usually what you want from a bug-scoped rule.

See also

• Automation rules for the full rule grammar and governance model.
• Redaction for how exception and traceback data is scrubbed before it is stored (fingerprints are computed from the scrubbed form).