Skip to main content

Terminology

This section defines key terms used throughout the CTRF specification.

Core Concepts​

Report​

One emitted CTRF document or payload. A report may contain a complete logical run, one shard or partition of a run, or merged results. Related documents can share a runId, but each materially different document should have its own reportId when identifiers are used.

Emitted Report​

A CTRF document that has been written, published, uploaded, or otherwise made available as an artifact. It is treated as an immutable snapshot.

Derived Report​

A new CTRF document produced by post-processing one or more emitted reports, such as by merging, filtering, enriching, or computing insights. It does not replace or mutate its source artifacts.

Run​

A single logical execution of a test suite or collection of tests. A run may consist of multiple physical executions (e.g., shards or parallel workers) that are part of the same coordinated execution context.

Test Case​

An individual logical test represented in the tests array. testId can identify the same test case across runs.

Execution​

The complete reported lifecycle of one test case within a run. executionId identifies this lifecycle, including previous attempts and the final outcome represented by the test object.

Status​

The final outcome of a test case. Must be one of: passed, failed, skipped, pending, or other.

Execution Concepts​

Attempt​

A single try within a test execution. Attempt number 1 is the initial execution. When retries occur, retryAttempts records the initial attempt and any earlier retries, while the test object represents the final attempt.

Retry / Retry Attempt​

Any attempt with an attempt number greater than 1. Retries occur when a test is re-executed after a failure or based on a retry policy.

Flaky Test​

A test is considered flaky only if its final status is passed and it experienced one or more failed attempts before passing.

Context Concepts​

Environment​

The system, platform, or context in which tests were executed, including build, commit, branch, OS, and other metadata.

Build​

The CI/CD pipeline execution or build context in which the tests were run. Build metadata appears in the environment object.

Analysis Concepts​

Baseline​

A previously generated CTRF report used as a reference for comparison with the current report when evaluating trends or regressions.

Insights​

Derived or aggregated metrics calculated across one or more test runs. Insights may be applied at the run-level or at the individual test case level.

Metric​

A quantitative value describing some property of a set of tests or runs (e.g., pass rate, average duration).

Metric Delta​

A structure describing a metric's current value, the corresponding value from a baseline run, and the computed change between them.

Actor Concepts​

Producer​

A tool, system, or process that generates CTRF reports by capturing and structuring test execution data.

Consumer​

A tool, system, or process that reads and interprets CTRF reports for purposes such as analysis, visualization, alerting, or trend detection.

Extension​

Custom metadata stored under a namespaced key inside an extra object. Extension keys and values that a consumer does not recognize are treated as opaque.

Attachment​

An external file or resource referenced by a test case or previous attempt, such as a screenshot, log file, or other artifact. attachmentId identifies one attachment reference instance.