A copy of spec/RESULTS.md, rendered from the file in the repository.

Test result format (v0)

Status: under review, together with the Testfile format. Feedback is welcome via GitHub issues.

The Testfile specification has two parts:

1. the Testfile format (TESTFILE.md) — the input: how a project describes its tests, and
2. this document — the output: how a recorded test run looks on disk.

The two parts are deliberately independent: different tools can generate results and different tools can consume them. A runner (any implementation) produces the layout below; viewers — a history CLI, a TUI, a web UI, CI tooling — only ever read it. A result consumer must not assume which runner produced a run.

Layout

Runs are recorded next to the Testfile in a .testfile/ folder. Every run is one self-contained folder:

.testfile/
  .gitignore             # "*" - the folder ignores itself
  runs/
    <run-id>/
      run.yaml           # the run's record (this document's core)
      junit.xml          # the same result as JUnit XML, for CI tooling
      tests/<slug>.log   # merged stdout+stderr per test
      services/<slug>.log# log of each started service
      artifacts/<slug>/… # files collected via the tests' `artifacts` globs

Log and artifact file names are derived from the test path (lower-cased, non-alphanumerics collapsed to -, plus a short hash against collisions), but consumers must not parse them: the authoritative names are the relative paths recorded in run.yaml (log, artifacts, junit).

Run ids and ordering

A run id starts with the run’s UTC timestamp at second resolution plus a random suffix, e.g. 20260802-193731-4bdc. Ids are unique per history but only sort chronologically at second granularity — consumers order runs by the record’s startedAt (millisecond ISO 8601 timestamp), using the id as tie-breaker.

run.yaml

A YAML document with these fields. A JSON schema of this format ships as schema/testrun.schema.json; runners write a # yaml-language-server: $schema=... modeline as the first line of every record, so editors validate and complete it.

Consumers must ignore unknown fields (producers may add fields in later format versions) and must skip run folders whose run.yaml is missing or unreadable.

tests[]

Every path in a record — log, artifacts, service logs — is relative to the run folder and /-separated on every platform, including Windows. A run travels (as an archive, a CI artifact, an S3 object) and is read by viewers on other machines, so a producer must not write the local separator.

suite

The shape of the Testfile the run came from, so a run.yaml explains itself without the Testfile next to it: the tree, what kind each node is, which tags it carries and which matrix combination an expanded instance belongs to. It describes the whole file, including tests that filters excluded from this run — tests[] says what actually executed.

suite is a single root node; every node has these fields:

Producers should record suite; consumers must tolerate its absence (records written by earlier runners have none) and fall back to deriving nesting from the path values.

fromEnvironment

A run is only reproducible from its record if the record says what the environment added. This block carries that, and nothing secret:

Each overrides[] entry:

Absent when the environment contributed nothing, so an ordinary run records nothing extra.

services[]

analysis

A run says what happened. It cannot say what it meant — that the three failures share one cause, that the flake was a port collision, that the red was infrastructure and not the change. Someone works that out afterwards, and today they write it in a pull request comment that nobody reading the run will ever see.

analysis:
  text: |
    All three failures come from the port 5432 collision in the parallel
    group: ci/migrations and ci/e2e both start a database service.
    Not caused by this change.
  author: claude-code
  at: 2026-08-01T12:10:00.000Z

Rules that keep it honest:

• A runner never writes it. The field is added after the run, by whoever did the reading; a runner has nothing to say here.
• It is an annotation, not a result. Consumers must show it as somebody’s opinion — never let it change a status, a count or a verdict. A run whose analysis says “this is fine” is still a failed run.
• It is optional and replaceable. A later reading may overwrite it. A producer writing it must preserve every other field of the record.

Merged runs

Two things produce several run folders for what is conceptually one test run: sharding (each shard runs a disjoint part of the suite) and a matrix of jobs (every job runs the same suite somewhere else). A merged run combines them into a single record — one verdict, one duration, the union of the tests — so consumers show it like any other run.

A merged run is an ordinary run: id, startedAt, status and the rest mean what they always mean, and a consumer that ignores merged still reads it correctly. What merging adds is recorded rather than hidden:

How the top-level fields are derived:

A test path may only appear once per variant combination. Shards merge without variants because no leaf appears twice; a matrix of jobs runs the same test everywhere, so those runs must carry distinct variants — that is what keeps path + variants unique and the merged run readable. A producer that finds a duplicate must fail rather than silently drop or overwrite a result.

Group nodes are the exception. A group is scaffolding around the tests below it, and every shard records the groups its own leaves sit in, so the same group path legitimately appears in several runs. Merging folds those into one entry per path + variants: the worst status of the contributions and the sum of their durations, with no single origin.

Copied logs and artifacts are namespaced by the run they came from — tests/<origin-id>/<slug>.log — so the same test’s logs from different legs stay apart. As everywhere else, the authoritative paths are the ones recorded in run.yaml.

In this repository testfile-viewer merge <run…> writes such a run; it takes run folders (an unpacked CI artifact) or ids from a local history.

Log files

Logs are plain UTF-8 text with \n line endings. Lines produced by the runner itself (not the test’s process) are prefixed with # — e.g. retry notices, timeout messages, cached: markers. Everything else is the test’s stdout and stderr, merged in arrival order.

There is no persisted whole-run log: viewers assemble the merged view on demand by concatenating, in record order, a === <path> (<status>[, <duration>ms]) === header per test followed by its log, then === service <name> (<status>) === sections. (Runs recorded by early runners may contain a pre-merged output.log; consumers should serve it as-is when present.)

junit.xml

Every run folder a runner records contains the run as JUnit XML: one <testcase> per leaf test (name = last path segment, classname = parent path), <failure> elements carrying the test’s log, <skipped/> markers, and counts/timestamps on the suite elements. Merged runs (see above) carry no junit.xml and no junit field — which is why the field is optional.

Secrets

Values loaded from envFile files, the variables a Testfile names in secrets:, and anything handed in with TESTFILE_SECRET_ (see the format specification), are secrets: producers must mask them (***) in every recorded log and must not write them into run.yaml. (Values shorter than 4 characters are exempt — masking them would mark ubiquitous substrings as secret without hiding anything.)

Retention and concurrency

Producers may prune old runs (the reference runner keeps the most recent 50 folders). A run folder is written completely before it is discoverable by its run.yaml; consumers watching .testfile/runs/ should tolerate folders appearing and disappearing at any time.

Legacy

Very early runners wrote a single runs.yaml index next to runs/ instead of per-run records. Consumers may read it (entries have the same shape as run.yaml), but must not require it; producers must not write it.