From 0781292538bd983616b59e528ab6b8dd2d2303c5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Wed, 10 Jun 2026 16:33:55 +0200 Subject: [PATCH] Phase 1: Test runner consolidation on Microsoft.Testing.Platform Documents the staged plan to retire Arcade's parallel test-runner code paths in favour of Microsoft.Testing.Platform (MTP) and dotnet test. This first PR makes no behavior changes for repos that do not opt into the deprecated paths. It adds: * Documentation/TestRunnerConsolidation.md - full 5-phase plan, rationale, and impact estimate. * MSBuild deprecation warnings (suppressible) for: - ARCADE0001: explicit UsingToolXUnit (already documented as deprecated in DefaultVersions.props but never warned). - ARCADE0002: UseVSTestRunner=true. - ARCADE0010: XUnitProject items in Helix projects (use MTPProject instead). * Deletes the orphaned tools/xunit-reporter/xunit-reporter.py Helix script which has self-marked itself deprecated since 2021 and is no longer invoked from any Arcade target. Defaults flips (Phase 2) and code deletions (Phases 3-5) will follow in separate PRs after arcade-validation runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- Documentation/TestRunnerConsolidation.md | 142 ++++++++++++++++++ .../tools/DefaultVersions.props | 3 + .../tools/Tests.targets | 21 +++ .../tools/xunit-reporter/xunit-reporter.py | 10 -- .../tools/xunit-runner/XUnitRunner.targets | 17 +++ 5 files changed, 183 insertions(+), 10 deletions(-) create mode 100644 Documentation/TestRunnerConsolidation.md delete mode 100644 src/Microsoft.DotNet.Helix/Sdk/tools/xunit-reporter/xunit-reporter.py diff --git a/Documentation/TestRunnerConsolidation.md b/Documentation/TestRunnerConsolidation.md new file mode 100644 index 00000000000..6785ae853e4 --- /dev/null +++ b/Documentation/TestRunnerConsolidation.md @@ -0,0 +1,142 @@ +# Test runner consolidation on Microsoft.Testing.Platform + +## Why + +Arcade currently maintains five parallel test-runner code paths in +`src/Microsoft.DotNet.Arcade.Sdk/tools/` (`XUnit`, `XUnitV3`, `MSTest`, +`VSTest`, `Microsoft.Testing.Platform`) plus two parallel Helix work-item +generators in `src/Microsoft.DotNet.Helix/Sdk/` (`xunit-runner` / +`CreateXUnitWorkItems` and `mtp-runner` / `CreateMTPWorkItems`). + +Most of that surface area predates [Microsoft.Testing.Platform][mtp] (MTP): + +* xUnit v3 [defaults to MTP][xunitv3-mtp] and produces a self-hosting test + executable when paired with `Microsoft.Testing.Platform.MSBuild`. +* MSTest 4.x [defaults to MTP][mstest-runner] via `MSTest.Sdk` / + `MSTest` metapackage. +* NUnit, TUnit, and other frameworks ship MTP runners. +* `dotnet test` is now MTP-aware (`TestingPlatformDotnetTestSupport`) and in + .NET 10 the [`dotnet test --use-testing-platform`][dotnet-test-mtp] + experience is the official orchestrator for MTP-based projects. +* VSTest is in maintenance mode; the [testfx team has stated][vstest-status] + that future investment is on MTP. + +The legacy paths therefore exist purely as compatibility surface. Each one +adds property names that have to be documented and supported, complicates +the `Test` target dispatch, and ships duplicated logger / TRX wiring that +MTP gives us for free. + +## Goal + +Reduce the Arcade test-execution surface to: + +* **One** runner targets file in the Arcade SDK that invokes MTP exes + directly (already present: + [`tools/Microsoft.Testing.Platform/Microsoft.Testing.Platform.targets`][mtp-targets]). +* **One** thin `dotnet test` wrapper for projects that have not yet + migrated to MTP — used only when the project is not a Testing Platform + application. +* **One** Helix work-item generator + ([`CreateMTPWorkItems`][create-mtp]) replacing both the legacy + `CreateXUnitWorkItems` flow and any framework-specific work-item + generators. + +Helix orchestration itself (`SendHelixJob`, queue selection, correlation +payloads, `EnableHelixJobMonitor`, XHarness for iOS/Android/WASM, test +retry, AzDO test-run lifecycle) is **out of scope** — those are Arcade's +genuine value-add and are independent of which runner runs the test. + +## Plan + +This is staged so each phase is independently revertable. + +### Phase 1 — Signal direction (this PR) + +* Add this document. +* Emit MSBuild warnings when a project opts into a path that we intend to + remove: + * Setting `UsingToolXUnit` (any value) — property is already documented + as deprecated in [`DefaultVersions.props`][defaultversions] but emits + no warning. + * Setting `UseVSTestRunner=true`. + * Including `XUnitProject` items in a Helix project (use `MTPProject` + instead — see the + [Helix Sdk readme](../src/Microsoft.DotNet.Helix/Sdk/Readme.md)). +* Delete the orphaned `xunit-reporter.py` Helix Python script which is + already self-marked as deprecated and is no longer invoked from any + Arcade target. + +No behavior changes for repos that do not set the deprecated properties. + +### Phase 2 — Flip defaults + +* Default `UseMicrosoftTestingPlatformRunner=true` is already in place for + xUnit v3 ([`XUnitV3.targets`][xunitv3-targets]). Audit downstream repos + via arcade-validation to confirm there are no remaining consumers + relying on the VSTest path. +* Default `EnableMSTestRunner=true` for MSTest projects, with an opt-out + for repos that still depend on VSTest-specific loggers / runsettings + features. +* Remove `Microsoft.NET.Test.Sdk` from the implicit reference set for MTP + projects (already conditional on `IsTestingPlatformApplication != true` + in [`Tests.props`][tests-props]; ensure that flag is reliably set for + every MTP path). + +### Phase 3 — Drop xUnit v2 + +* Delete `tools/XUnit/` (`XUnit.targets` and `XUnit.Runner.targets`). + xUnit v2 is in maintenance upstream; v3 has been the Arcade + recommendation since [#15671][pr-15671]. +* Repos still on v2 can pin to an older Arcade SDK or set + `TestRunnerName=XUnit` and import their own targets. + +### Phase 4 — Collapse runner targets + +* Replace `XUnitV3.Runner.targets`, `VSTest.targets`, and the runner + portion of `MSTest.targets` with a single `RunTests` target that: + * for MTP projects (`IsTestingPlatformApplication=true`), execs the + self-hosting test app with MTP reporter flags; and + * for non-MTP projects, invokes `dotnet test` with standard + `--logger trx;...` arguments. +* Framework-specific `.targets` shrink to package-reference lists only. + +### Phase 5 — Helix-side cleanup + +* Delete `tools/xunit-runner/` (`XUnitRunner.props`, `XUnitRunner.targets`, + `XUnitPublish.targets` — the publish helper has already been generalized + to `_MTPPublishTargetsPath`) and `CreateXUnitWorkItems.cs`. `MTPProject` + already covers xUnit v3, MSTest 4, NUnit, and TUnit. +* Drop the related `XUnitProject`, `XUnitPublishTargetFramework`, + `XUnitRuntimeTargetFramework`, `XUnitRunnerVersion`, and + `XUnitArguments` properties from the SDK and the + [SendingJobsToHelix doc][send-helix-doc]. + +### Phase 6 — `TestArchitectures` loop + +* `Tests.targets`' inner/outer loops over `TestArchitectures` × + `TargetFrameworks` exist to feed runner CLIs. Once `RunTests` is just + `dotnet test`, the .NET SDK's per-TFM dispatch covers the TFM axis. The + architecture axis (x86/x64 multiplexing of a single binary on .NET + Framework) is Arcade-specific and stays — but the wrapper shrinks to + roughly 20 lines. + +## Estimated impact + +| Area | Files removed | LoC removed | Public properties / items removed | +|---|---|---|---| +| Arcade SDK runner targets | XUnit v2 dir, `VSTest.targets`, two `*.Runner.targets` | ~400 | `UseVSTestRunner`, `UsingToolXUnit`, `TestRunnerName=XUnit\|VSTest` | +| Helix Sdk legacy xUnit | `xunit-runner/`, `CreateXUnitWorkItems.cs`, `xunit-reporter/` | ~600 | `XUnitProject`, `XUnitPublishTargetFramework`, `XUnitRuntimeTargetFramework`, `XUnitRunnerVersion`, `XUnitArguments`, `EnableXUnitReporter` | +| **Total** | **~10 files** | **~1000 LoC** | **8 properties + 1 item group** | + +[mtp]: https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-intro +[mtp-targets]: ../src/Microsoft.DotNet.Arcade.Sdk/tools/Microsoft.Testing.Platform/Microsoft.Testing.Platform.targets +[create-mtp]: ../src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs +[xunitv3-targets]: ../src/Microsoft.DotNet.Arcade.Sdk/tools/XUnitV3/XUnitV3.targets +[defaultversions]: ../src/Microsoft.DotNet.Arcade.Sdk/tools/DefaultVersions.props +[tests-props]: ../src/Microsoft.DotNet.Arcade.Sdk/tools/Tests.props +[send-helix-doc]: ./AzureDevOps/SendingJobsToHelix.md +[xunitv3-mtp]: https://xunit.net/docs/getting-started/v3/microsoft-testing-platform +[mstest-runner]: https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-intro +[dotnet-test-mtp]: https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test +[vstest-status]: https://github.com/microsoft/vstest/blob/main/README.md +[pr-15671]: https://github.com/dotnet/arcade/pull/15671 diff --git a/src/Microsoft.DotNet.Arcade.Sdk/tools/DefaultVersions.props b/src/Microsoft.DotNet.Arcade.Sdk/tools/DefaultVersions.props index c053822a4ff..c228f120fa3 100644 --- a/src/Microsoft.DotNet.Arcade.Sdk/tools/DefaultVersions.props +++ b/src/Microsoft.DotNet.Arcade.Sdk/tools/DefaultVersions.props @@ -38,7 +38,10 @@ + <_UsingToolXUnitExplicitlySet Condition="'$(UsingToolXUnit)' != ''">true true + + + + + + $(PlatformTarget) x64 diff --git a/src/Microsoft.DotNet.Helix/Sdk/tools/xunit-reporter/xunit-reporter.py b/src/Microsoft.DotNet.Helix/Sdk/tools/xunit-reporter/xunit-reporter.py deleted file mode 100644 index 750909a815a..00000000000 --- a/src/Microsoft.DotNet.Helix/Sdk/tools/xunit-reporter/xunit-reporter.py +++ /dev/null @@ -1,10 +0,0 @@ -#!/usr/bin/env python - -import helix.event -import helix.settings - -if __name__ == '__main__': - settings = helix.settings.settings_from_env() - event_client = helix.event.create_from_uri(settings.event_uri) - event_client.warning(settings, "Obsolete", "xunit-reporter.py is deprecated, please remove 'EnableXUnitReporter' from your build and use 'EnableAzurePipelinesReporter' instead.") - print("This reporter is deprecated") diff --git a/src/Microsoft.DotNet.Helix/Sdk/tools/xunit-runner/XUnitRunner.targets b/src/Microsoft.DotNet.Helix/Sdk/tools/xunit-runner/XUnitRunner.targets index 5c2f250360e..3b4baf4a509 100644 --- a/src/Microsoft.DotNet.Helix/Sdk/tools/xunit-runner/XUnitRunner.targets +++ b/src/Microsoft.DotNet.Helix/Sdk/tools/xunit-runner/XUnitRunner.targets @@ -19,6 +19,23 @@ + + + + +