Skip to main content

Metrics Reference

Metrics are quantitative values that describe properties of tests or runs. They are used within the Insights Object to enable trend analysis and cross-run comparisons.

Metric Levels​

Run-Level Metrics​

Run-level metrics are calculated using all tests from all runs being analyzed. These provide a holistic view of test health across multiple runs.

Test-Level Metrics​

Test-level metrics are calculated using all executions of a specific test case across runs. These help evaluate individual test stability and performance over time.

Metric Types​

Percentage Metrics​

Percentage metrics are represented as decimal values between 0 and 1:

ValueMeaning
00%
0.2525%
0.550%
1.0100%

Absolute Metrics​

Absolute metrics are raw values with specific units (e.g., milliseconds for duration, counts for totals).

Standard Metrics​

passRate​

The proportion of tests with status passed.

  • Type: Percentage metric
  • Level: Run-level and test-level
  • Calculation: passed tests / total tests

failRate​

The proportion of tests with status failed.

  • Type: Percentage metric
  • Level: Run-level and test-level
  • Calculation: failed tests / total tests

flakyRate​

The proportion of tests identified as flaky.

A test is considered flaky only if its final status is passed and it experienced one or more failed attempts before succeeding. Tests that retry but never pass (final status failed) are not considered flaky.

  • Type: Percentage metric
  • Level: Run-level and test-level
  • Calculation: flaky tests / total tests

averageRunDuration​

The mean duration of all test runs.

  • Type: Absolute metric (milliseconds)
  • Level: Run-level only
  • Calculation: sum of run durations / number of runs

averageTestDuration​

The mean duration of all test executions.

  • Type: Absolute metric (milliseconds)
  • Level: Run-level and test-level
  • Calculation (run-level): sum of test durations / number of tests
  • Calculation (test-level): sum of all attempt durations for this test / number of attempts across analyzed runs

p95RunDuration​

The 95th percentile of test run durations. 95% of runs completed within this duration.

  • Type: Absolute metric (milliseconds)
  • Level: Run-level only

p95TestDuration​

The 95th percentile of test execution durations. 95% of tests completed within this duration.

  • Type: Absolute metric (milliseconds)
  • Level: Test-level only