Skip to content

Rename XUnitV3Project to MTPProject in Helix Sdk (breaking)#16986

Merged
Evangelink merged 8 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/helix-mtp-support
Jun 12, 2026
Merged

Rename XUnitV3Project to MTPProject in Helix Sdk (breaking)#16986
Evangelink merged 8 commits into
dotnet:mainfrom
Evangelink:dev/amauryleve/helix-mtp-support

Conversation

@Evangelink

@Evangelink Evangelink commented Jun 8, 2026

Copy link
Copy Markdown
Member

Summary

Breaking change. Renames the existing XUnitV3Project Helix item type to MTPProject and changes the generated work item command to use only Microsoft.Testing.Platform's built-in flags (no forced reporter args).

xUnit v3 with the Microsoft.Testing.Platform (MTP) runner uses the exact same self-hosting dotnet exec ... <assembly>.dll execution model as MSTest 4.x, NUnit-MTP, TUnit, and any other MTP-based framework. The previous XUnitV3Project and the originally-proposed new MTPProject were 95% the same code; only the reporter args differed. This PR collapses that duplication.

<ItemGroup>
  <MTPProject Include=""..\test\MyApp.Tests\MyApp.Tests.csproj"" />
</ItemGroup>

MTPProject covers any MTP-based test project: MSTest 4.x, xUnit v3 with MTP, NUnit with the MTP runner, TUnit, custom MTP.

What this contains

  • Renames CreateXUnitV3WorkItems -> CreateMTPWorkItems, tools/xunitv3-runner/ -> tools/mtp-runner/, XUnitV3Runner.{props,targets} -> MTPRunner.{props,targets}.
  • Drops the UseMicrosoftTestingPlatformRunner switch and the legacy -xml/-noAutoReporters xunit-v3 code path. MTP is now the only execution mode for this Helix item type; users still on legacy xunit-v3 native or VSTest hand-author a <HelixWorkItem>.
  • Generated command is framework-agnostic. The only MTP flag injected is --results-directory . (built into the MTP core, accepted by every MTP framework), so reporter output lands in the work item working directory. Reporter selection (--report-trx, --report-trx-filename ...) and framework-specific flags (xUnit v3's --auto-reporters off) are NOT forced - they are extension/framework-specific and would error on frameworks that have not opted in to that extension.
  • Adds MTPAdditionalArguments property / task parameter as the escape hatch for any extra MTP flags users want applied across every emitted work item.
  • Updates Microsoft.DotNet.Helix.Sdk.props (UsingTask), the Helix SDK Readme section, Documentation/AzureDevOps/SendingJobsToHelix.md, and tests/UnitTests.proj (arcade's own self-test).

Generated work item command

dotnet exec --roll-forward Major \
    --runtimeconfig MyApp.Tests.runtimeconfig.json \
    --depsfile MyApp.Tests.deps.json \
    MyApp.Tests.dll \
    --results-directory .

With MTP's --auto-reporters on by default and Microsoft.Testing.Extensions.TrxReport referenced (transitive for MSTest.Sdk and for arcade XUnitV3.targets), the TRX reporter activates automatically and writes a default-named *.trx file into the work item directory; arcade's existing TRXFormat Python reporter globs *.trx so the auto-generated filename is fine.

Why a breaking rename instead of additive

The previous draft of this PR (which left XUnitV3Project in place and added a parallel MTPProject) was reviewed offline. Conclusion: the two paths are identical except for the reporter flag, so duplicating the task + targets + props infrastructure is purely accidental complexity. A rename is cheaper to maintain and clearer to users.

Downstream impact

A code search across dotnet/* (and public GitHub) shows dotnet/efcore/eng/helix.proj is the only in-tree consumer of XUnitV3Project (5 occurrences: 1 Include + 4 Remove). efcore already sets UseMicrosoftTestingPlatformRunner=true in test/Directory.Build.props, so they're on the MTP code path today.

Migration for efcore is mechanical:

Today After this PR
Item type <XUnitV3Project> <MTPProject>
Report format xUnit XML (--report-xunit-xml-filename testResults.xml) TRX produced automatically by Microsoft.Testing.Extensions.TrxReport
Reporter package Microsoft.Testing.Extensions.XunitXmlReport (transitive via arcade XUnitV3.targets) Microsoft.Testing.Extensions.TrxReport (also transitive via arcade XUnitV3.targets)
--auto-reporters off injected automatically opt in via <MTPAdditionalArguments>--auto-reporters off</MTPAdditionalArguments>

Concretely: 5 occurrences of XUnitV3Project in eng/helix.proj get renamed to MTPProject. If they want to preserve the current --auto-reporters off behaviour they add <MTPAdditionalArguments>--auto-reporters off --report-trx --report-trx-filename testResults.trx</MTPAdditionalArguments> (the explicit --report-trx is needed when --auto-reporters is off). Otherwise the TRX auto-reporter picks up automatically. A follow-up PR against efcore tracking this migration will be opened before this PR is merged (see #17001 for the broader rollout-coordination tracking).

Prerequisites for users

Test projects must reference Microsoft.Testing.Extensions.TrxReport:

  • MSTest.Sdk projects: included transitively, nothing to do.
  • xUnit v3 projects built with Microsoft.DotNet.Arcade.Sdk's XUnitV3 targets: included implicitly, nothing to do.
  • NUnit-MTP / TUnit / custom MTP: add <PackageReference Include=""Microsoft.Testing.Extensions.TrxReport"" /> to the test project.

AzureDevOps reporter caveat

The MTP Microsoft.Testing.Extensions.AzureDevOpsReport only activates when --report-azdo is explicitly passed AND TF_BUILD=true. We never pass --report-azdo, so there is no risk of a duplicate AzDO test run on top of the one the Helix SDK already opens. If a user has wired --report-azdo into their test project they should drop it for Helix runs, or pass --auto-reporters off via MTPAdditionalArguments.

Out of scope

Tracking

Copilot AI review requested due to automatic review settings June 8, 2026 12:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends Microsoft.DotNet.Helix.Sdk with first-class support for Microsoft Testing Platform (MTP) test projects by introducing new item types (MSTestProject, MTPProject) that auto-publish and generate Helix work items, analogous to the existing XUnitV3Project experience.

Changes:

  • Add CreateMTPWorkItems MSBuild task to generate Helix work items for published MTP test assemblies.
  • Add mtp-runner props/targets to restore/publish MTP projects and emit HelixWorkItem items automatically.
  • Document usage, prerequisites, and patterns for MSTest/NUnit/TUnit and other MTP-based test projects in the Helix SDK readme.
Show a summary per file
File Description
src/Microsoft.DotNet.Helix/Sdk/tools/mtp-runner/MTPRunner.targets Adds restore/publish pipeline for @(MTPProject) and folds @(MSTestProject) into it before creating work items.
src/Microsoft.DotNet.Helix/Sdk/tools/mtp-runner/MTPRunner.props Hooks the new MTP runner targets into the Helix SDK’s mono-queue target import chain.
src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.props Registers the new CreateMTPWorkItems task via UsingTask.
src/Microsoft.DotNet.Helix/Sdk/Readme.md Documents MSTestProject / MTPProject usage, prerequisites, and guidance around TRX + AzDO reporting.
src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Implements the new task to generate HelixWorkItem command/payload/timeout for MTP projects, including dedup logic.

Copilot's findings

  • Files reviewed: 5/5 changed files
  • Comments generated: 1

Comment thread src/Microsoft.DotNet.Helix/Sdk/tools/mtp-runner/MTPRunner.targets Outdated
ViktorHofer
ViktorHofer previously approved these changes Jun 9, 2026
xUnit v3 with the Microsoft.Testing.Platform runner (default for v3) uses the exact same self-hosting 'dotnet exec ... <assembly>.dll' execution model as MSTest 4.x, NUnit-MTP, TUnit, and any other MTP-based test framework. The only difference between the previous XUnitV3Project Helix item type and the proposed MTPProject was the reporter args. This commit drops the duplication by:

* Renaming the item type XUnitV3Project to MTPProject, the task CreateXUnitV3WorkItems to CreateMTPWorkItems, and the folder/files xunitv3-runner -> mtp-runner / XUnitV3Runner -> MTPRunner.
* Dropping the UseMicrosoftTestingPlatformRunner switch and the legacy '-xml/-noAutoReporters' code path. MTP is now the only supported execution mode for this Helix item type; users on legacy non-MTP runners hand-author <HelixWorkItem>.
* Switching the generated reporter args to TRX ('--report-trx --report-trx-filename testResults.trx'), which is cross-framework and parsed natively by arcade's TRXFormat reporter. Microsoft.Testing.Extensions.TrxReport is required; MSTest.Sdk references it transitively and arcade's XUnitV3 SDK targets reference it implicitly under MTP mode.
* Adding MSTestProject as a discoverability shim that folds into MTPProject before any other target runs.

Migration: the only external in-tree consumer is dotnet/efcore's eng/helix.proj, which can rename '<XUnitV3Project ... />' to '<MTPProject ... />' as part of its next arcade SDK bump. Arcade's own tests/UnitTests.proj is updated in this commit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Evangelink Evangelink force-pushed the dev/amauryleve/helix-mtp-support branch from 733957e to 9537ab2 Compare June 9, 2026 14:43
@Evangelink Evangelink changed the title Add MSTestProject and MTPProject items to Helix SDK Rename XUnitV3Project to MTPProject in Helix Sdk (breaking) and add MSTestProject alias Jun 9, 2026
On reflection the alias is asymmetric: we do not ship XUnitV3Project, NUnitProject, or TUnitProject aliases either. Singling out MSTest invites the false impression that MSTestProject does something MSTest-specific. The MSTest case is still called out explicitly in the supported-frameworks list of the Readme and SendingJobsToHelix doc, so discoverability via doc search is preserved.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 10, 2026 07:41
@Evangelink Evangelink changed the title Rename XUnitV3Project to MTPProject in Helix Sdk (breaking) and add MSTestProject alias Rename XUnitV3Project to MTPProject in Helix Sdk (breaking) Jun 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 9/9 changed files
  • Comments generated: 2

Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Outdated
Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs
- Fix MSB4096 in BuildMTPProjects: drop the speculative AdditionalProperties match from the inner MTPProject metadata-update ItemGroup. Bare %(AdditionalProperties) in batched item-transform context triggers MSB4096 when items don't define it; original XUnitV3Runner.targets matched only on Identity. The forwarded AdditionalProperties on the inner MSBuild calls (for multi-TFM build) is unchanged.

- Quote and validate TrxReportFilename in CreateMTPWorkItems: wrap in double quotes so shell-significant chars (spaces, etc.) don't tokenize, and reject path separators or embedded quotes early with a clear error.

- Fix latent bug uncovered by the new tests: TimeSpan.TryParse resets the out parameter to default(TimeSpan) on failure, which would have silently produced a 00:00:00 timeout when a malformed MTPWorkItemTimeout was supplied. Use a temp variable and only overwrite the 5-minute default on success.

- Add CreateMTPWorkItemsTests with 15 cases covering command shape, reporter args, custom TRX filename quoting, invalid filename rejection (path separators, embedded quote, empty), missing required metadata, valid + invalid timeout, dedup by (Identity, AdditionalProperties), multi-TFM via differing AdditionalProperties, custom PathToDotnet, and Arguments append order.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Evangelink Evangelink marked this pull request as ready for review June 10, 2026 15:00
Copilot AI review requested due to automatic review settings June 10, 2026 15:00
@Evangelink Evangelink enabled auto-merge June 10, 2026 15:01
ViktorHofer
ViktorHofer previously approved these changes Jun 10, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 10/10 changed files
  • Comments generated: 0 new

Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Outdated
Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Outdated
Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Outdated
Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Outdated
Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Outdated
Comment thread src/Microsoft.DotNet.Helix/Sdk/tools/mtp-runner/MTPRunner.targets Outdated
@Youssef1313

Copy link
Copy Markdown
Member

(MTP) runner (which is the default for v3)

I don't think this is correct. The default for xunit.v3 is xunit's runner AFAIK.

Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs
Evangelink added a commit to dotnet/efcore that referenced this pull request Jun 11, 2026
Prepare a PR to fix helix rename from PR dotnet/arcade#16986

This will only need to be merged when the new arcade will flow (I would expect the arcade flow PR to fail).
- Remove de-duplication of MTPProjects: the old CreateXUnitV3WorkItems
  task didn't deduplicate either; defer to MSBuild item semantics and
  keep the task minimal.
- Drop the TrxReportFilename defensive validation: MTP itself rejects
  values containing path separators, so the redundant pre-check just
  added complexity. Filename is still quoted in the emitted command so
  spaces are safe.
- Simplify the timeout-parsing block to use 'out timeout' directly with
  a default reset on failure (TimeSpan.TryParse zeroes the out var when
  it fails).
- Shrink the MTPTrxReportFilename property documentation in the
  targets file to a one-liner.
- Update tests: replace InvalidTrxReportFilenameIsRejected with
  TrxReportFilenameIsPassedThroughVerbatim and replace
  DuplicatesByIdentityAndAdditionalPropertiesAreCollapsed with
  DuplicateInputsArePassedThroughAsSeparateWorkItems.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous CreateXUnitV3WorkItems MTP path passed --auto-reporters off
explicitly; the rename to CreateMTPWorkItems dropped it. Restoring it
locks the behaviour down so any other MTP reporter extension a test
project happens to reference cannot auto-activate inside the Helix work
item and produce extra reports the arcade Helix pipeline does not
expect. The intent in Helix runs is for the TRX reporter to be the only
active reporter.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 11, 2026 09:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 10/10 changed files
  • Comments generated: 4

Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs
Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs
Evangelink and others added 2 commits June 11, 2026 11:59
…onalArguments

`--auto-reporters` is registered by xUnit v3's MTP integration only
(see xunit/xunit src/common/MicrosoftTestingPlatform/CommandLineOptionsProvider.cs);
MSTest, NUnit, TUnit and other MTP frameworks reject it as an unknown
option, so the previous commit would break every non-xUnit MTP project.

Instead of trying to special-case per framework, expose a new
MTPAdditionalArguments task parameter (surfaced via the MTPRunner.targets
property of the same name) so users can append any framework-specific
switches once for all their MTP work items - xUnit users can set
`<MTPAdditionalArguments>--auto-reporters off</MTPAdditionalArguments>`,
MSTest users can leave it blank, and TUnit users can pass whatever else
they need without arcade having to know about it.

The extra args land between the forced reporter args and any per-project
`Arguments` metadata, so per-project values can still override the
global ones if needed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The --report-trx and --report-trx-filename switches require the
Microsoft.Testing.Extensions.TrxReport extension to be referenced by the
test project. For projects that have not referenced it, passing those
flags fails at runtime with an unknown-option error. Following the same
principle that removed --auto-reporters off (xUnit-v3-specific), drop
these from the forced command and let users opt in via the
MTPAdditionalArguments property.

With MTP's --auto-reporters on by default, projects that DO reference the
TrxReport extension produce a TRX file automatically. arcade's TRXFormat
reporter globs *.trx in the work item directory so the auto-generated
filename is fine; --report-trx-filename is only needed when a caller
wants a deterministic name (then they pass it via MTPAdditionalArguments).

The generated command now passes only the built-in --results-directory .
(an MTP-core option that works on every framework) so reporter output
lands in the work item working directory where Helix collects results.

- Drop TrxReportFilename task parameter and MTPTrxReportFilename property
- Update CreateMTPWorkItems comment and reporterArgs
- Drop CustomTrxReportFilenameIsQuoted and TrxReportFilenameIsPassedThroughVerbatim tests
- Update remaining tests to assert the new --results-directory-only shape
- Update Readme.md MTPAdditionalArguments example to include --report-trx
  for callers who want filename control

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 11, 2026 13:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 10/10 changed files
  • Comments generated: 1

Comment thread src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs Outdated
The targets file's introductory comment contained literal MTP option
names with double-dash prefixes inside an XML <!-- ... --> block, which
XML does not allow ('-- cannot appear inside a comment'). MSBuild
refused to load the file with MSB4024 and every Helix-using stage
failed. Rephrase the comment to defer reporter/auto-reporter wording
to the Helix Sdk Readme so the targets file stays parseable.

Also update the CreateMTPWorkItems.MTPProjects XML doc to describe the
new command shape - it no longer says 'after the auto-injected reporter
flags' (no flags are injected anymore) and now points to the actual
ordering: built-in --results-directory, then MTPAdditionalArguments,
then per-project Arguments metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Evangelink Evangelink merged commit a6bacc0 into dotnet:main Jun 12, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants