[nextest-runner] add the ability to fail tests if flaky

[sunshowers]

There are some use cases where we would like to retry failing tests, but then mark them as failures (and the test as a failure) in the end. Add support for this via both configuration and as --flaky-result=pass/fail.

TODO:

• Get a quick-junit release out (this currently uses a local patch)

[codecov]

Codecov Report

❌ Patch coverage is 94.07745% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.57%. Comparing base (e9e3722) to head (b0a293a).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
nextest-runner/src/reporter/structured/libtest.rs	21.42%	22 Missing ⚠️
nextest-runner/src/reporter/events.rs	98.99%	2 Missing ⚠️
cargo-nextest/src/dispatch/core/run.rs	83.33%	1 Missing ⚠️
nextest-runner/src/reporter/displayer/imp.rs	95.45%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3148      +/-   ##
==========================================
+ Coverage   84.45%   84.57%   +0.12%     
==========================================
  Files         156      156              
  Lines       43314    43683     +369     
==========================================
+ Hits        36580    36945     +365     
- Misses       6734     6738       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds support for treating flaky tests (tests that pass on retry) as either passing or failing, configurable via both nextest config (flaky-result = "pass"|"fail") and a new CLI flag (--flaky-result pass|fail). This integrates through execution, reporting (terminal/libtest JSON/JUnit), rerun logic, and integration/fixture expectations.

Changes:

• Introduces FlakyResult::{Pass, Fail} behavior in config parsing and independent override resolution (separate from retry policy).
• Adds --flaky-result CLI override and threads it through runner/executor contexts.
• Updates reporters (terminal status lines, libtest JSON, JUnit) plus rerun/outcome logic and integration fixtures/tests to recognize “flaky but failing”.

Reviewed changes

Copilot reviewed 19 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
nextest-runner/src/runner/imp.rs	Adds CLI-forced flaky-result plumbing into the runner builder/inner.
nextest-runner/src/runner/executor.rs	Applies CLI override for flaky-result when constructing per-test execution state.
nextest-runner/src/reporter/structured/libtest.rs	Treats flaky-fail as failed in libtest JSON + adds unit test coverage.
nextest-runner/src/reporter/events.rs	Adjusts run stats accounting so flaky-fail contributes to failures and status levels.
nextest-runner/src/reporter/displayer/unit_output.rs	Adds resolve_for_describe to decide output display using overall execution description.
nextest-runner/src/reporter/displayer/status_level.rs	Updates tests to use the new output-display resolver.
nextest-runner/src/reporter/displayer/snapshots/*.snap	Updates status line snapshot to include FLKY-FL variants.
nextest-runner/src/reporter/displayer/imp.rs	Displays FLKY-FL in final status lines and prints an explanatory intermediate line.
nextest-runner/src/reporter/aggregator/junit.rs	Emits JUnit “flaky failure” as a failure with <flakyFailure> rerun semantics.
nextest-runner/src/record/rerun.rs	Uses describe() so rerun outcome treats flaky-fail as failed + adds tests.
nextest-runner/src/config/elements/retry_policy.rs	Extends retries TOML schema with flaky-result and adds parsing/override tests.
integration-tests/tests/integration/main.rs	Adds end-to-end integration tests for config and CLI flaky-result behavior.
integration-tests/tests/integration/fixtures.rs	Teaches output/JUnit verification about FLKY-FL and flaky-fail JUnit semantics.
fixtures/nextest-tests/.config/nextest.toml	Adds a fixture profile demonstrating flaky-result = "fail" overrides.
fixture-data/src/nextest_tests.rs	Marks a fixture test as flaky-result-fail for expected outcomes.
fixture-data/src/models.rs	Adds CheckResult::FlakyFail plus run property flags and expected rerun logic updates.
cargo-nextest/src/dispatch/core/value_enums.rs	Adds FlakyResultOpt value enum for clap.
cargo-nextest/src/dispatch/core/run.rs	Adds --flaky-result flag and wires it into TestRunnerBuilder.
Cargo.toml	Bumps quick-junit to 0.6.0.
Cargo.lock	Updates lockfile for quick-junit (and transitive quick-xml).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[trunk-io]

     

View Full Report ↗︎ ⋅ Docs