Rename XUnitV3Project to MTPProject in Helix Sdk (breaking)#16986
Merged
Evangelink merged 8 commits intoJun 12, 2026
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends Microsoft.DotNet.Helix.Sdk with first-class support for Microsoft Testing Platform (MTP) test projects by introducing new item types (MSTestProject, MTPProject) that auto-publish and generate Helix work items, analogous to the existing XUnitV3Project experience.
Changes:
- Add
CreateMTPWorkItemsMSBuild task to generate Helix work items for published MTP test assemblies. - Add
mtp-runnerprops/targets to restore/publish MTP projects and emitHelixWorkItemitems automatically. - Document usage, prerequisites, and patterns for MSTest/NUnit/TUnit and other MTP-based test projects in the Helix SDK readme.
Show a summary per file
| File | Description |
|---|---|
src/Microsoft.DotNet.Helix/Sdk/tools/mtp-runner/MTPRunner.targets |
Adds restore/publish pipeline for @(MTPProject) and folds @(MSTestProject) into it before creating work items. |
src/Microsoft.DotNet.Helix/Sdk/tools/mtp-runner/MTPRunner.props |
Hooks the new MTP runner targets into the Helix SDK’s mono-queue target import chain. |
src/Microsoft.DotNet.Helix/Sdk/tools/Microsoft.DotNet.Helix.Sdk.props |
Registers the new CreateMTPWorkItems task via UsingTask. |
src/Microsoft.DotNet.Helix/Sdk/Readme.md |
Documents MSTestProject / MTPProject usage, prerequisites, and guidance around TRX + AzDO reporting. |
src/Microsoft.DotNet.Helix/Sdk/CreateMTPWorkItems.cs |
Implements the new task to generate HelixWorkItem command/payload/timeout for MTP projects, including dedup logic. |
Copilot's findings
- Files reviewed: 5/5 changed files
- Comments generated: 1
ViktorHofer
previously approved these changes
Jun 9, 2026
xUnit v3 with the Microsoft.Testing.Platform runner (default for v3) uses the exact same self-hosting 'dotnet exec ... <assembly>.dll' execution model as MSTest 4.x, NUnit-MTP, TUnit, and any other MTP-based test framework. The only difference between the previous XUnitV3Project Helix item type and the proposed MTPProject was the reporter args. This commit drops the duplication by:
* Renaming the item type XUnitV3Project to MTPProject, the task CreateXUnitV3WorkItems to CreateMTPWorkItems, and the folder/files xunitv3-runner -> mtp-runner / XUnitV3Runner -> MTPRunner.
* Dropping the UseMicrosoftTestingPlatformRunner switch and the legacy '-xml/-noAutoReporters' code path. MTP is now the only supported execution mode for this Helix item type; users on legacy non-MTP runners hand-author <HelixWorkItem>.
* Switching the generated reporter args to TRX ('--report-trx --report-trx-filename testResults.trx'), which is cross-framework and parsed natively by arcade's TRXFormat reporter. Microsoft.Testing.Extensions.TrxReport is required; MSTest.Sdk references it transitively and arcade's XUnitV3 SDK targets reference it implicitly under MTP mode.
* Adding MSTestProject as a discoverability shim that folds into MTPProject before any other target runs.
Migration: the only external in-tree consumer is dotnet/efcore's eng/helix.proj, which can rename '<XUnitV3Project ... />' to '<MTPProject ... />' as part of its next arcade SDK bump. Arcade's own tests/UnitTests.proj is updated in this commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
733957e to
9537ab2
Compare
On reflection the alias is asymmetric: we do not ship XUnitV3Project, NUnitProject, or TUnitProject aliases either. Singling out MSTest invites the false impression that MSTestProject does something MSTest-specific. The MSTest case is still called out explicitly in the supported-frameworks list of the Readme and SendingJobsToHelix doc, so discoverability via doc search is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix MSB4096 in BuildMTPProjects: drop the speculative AdditionalProperties match from the inner MTPProject metadata-update ItemGroup. Bare %(AdditionalProperties) in batched item-transform context triggers MSB4096 when items don't define it; original XUnitV3Runner.targets matched only on Identity. The forwarded AdditionalProperties on the inner MSBuild calls (for multi-TFM build) is unchanged. - Quote and validate TrxReportFilename in CreateMTPWorkItems: wrap in double quotes so shell-significant chars (spaces, etc.) don't tokenize, and reject path separators or embedded quotes early with a clear error. - Fix latent bug uncovered by the new tests: TimeSpan.TryParse resets the out parameter to default(TimeSpan) on failure, which would have silently produced a 00:00:00 timeout when a malformed MTPWorkItemTimeout was supplied. Use a temp variable and only overwrite the 5-minute default on success. - Add CreateMTPWorkItemsTests with 15 cases covering command shape, reporter args, custom TRX filename quoting, invalid filename rejection (path separators, embedded quote, empty), missing required metadata, valid + invalid timeout, dedup by (Identity, AdditionalProperties), multi-TFM via differing AdditionalProperties, custom PathToDotnet, and Arguments append order. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ViktorHofer
previously approved these changes
Jun 10, 2026
Youssef1313
reviewed
Jun 10, 2026
Youssef1313
reviewed
Jun 10, 2026
Youssef1313
reviewed
Jun 10, 2026
Youssef1313
reviewed
Jun 10, 2026
Youssef1313
reviewed
Jun 10, 2026
Youssef1313
reviewed
Jun 10, 2026
Member
I don't think this is correct. The default for xunit.v3 is xunit's runner AFAIK. |
Youssef1313
reviewed
Jun 10, 2026
Evangelink
added a commit
to dotnet/efcore
that referenced
this pull request
Jun 11, 2026
Prepare a PR to fix helix rename from PR dotnet/arcade#16986 This will only need to be merged when the new arcade will flow (I would expect the arcade flow PR to fail).
- Remove de-duplication of MTPProjects: the old CreateXUnitV3WorkItems task didn't deduplicate either; defer to MSBuild item semantics and keep the task minimal. - Drop the TrxReportFilename defensive validation: MTP itself rejects values containing path separators, so the redundant pre-check just added complexity. Filename is still quoted in the emitted command so spaces are safe. - Simplify the timeout-parsing block to use 'out timeout' directly with a default reset on failure (TimeSpan.TryParse zeroes the out var when it fails). - Shrink the MTPTrxReportFilename property documentation in the targets file to a one-liner. - Update tests: replace InvalidTrxReportFilenameIsRejected with TrxReportFilenameIsPassedThroughVerbatim and replace DuplicatesByIdentityAndAdditionalPropertiesAreCollapsed with DuplicateInputsArePassedThroughAsSeparateWorkItems. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous CreateXUnitV3WorkItems MTP path passed --auto-reporters off explicitly; the rename to CreateMTPWorkItems dropped it. Restoring it locks the behaviour down so any other MTP reporter extension a test project happens to reference cannot auto-activate inside the Helix work item and produce extra reports the arcade Helix pipeline does not expect. The intent in Helix runs is for the TRX reporter to be the only active reporter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…onalArguments `--auto-reporters` is registered by xUnit v3's MTP integration only (see xunit/xunit src/common/MicrosoftTestingPlatform/CommandLineOptionsProvider.cs); MSTest, NUnit, TUnit and other MTP frameworks reject it as an unknown option, so the previous commit would break every non-xUnit MTP project. Instead of trying to special-case per framework, expose a new MTPAdditionalArguments task parameter (surfaced via the MTPRunner.targets property of the same name) so users can append any framework-specific switches once for all their MTP work items - xUnit users can set `<MTPAdditionalArguments>--auto-reporters off</MTPAdditionalArguments>`, MSTest users can leave it blank, and TUnit users can pass whatever else they need without arcade having to know about it. The extra args land between the forced reporter args and any per-project `Arguments` metadata, so per-project values can still override the global ones if needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The --report-trx and --report-trx-filename switches require the Microsoft.Testing.Extensions.TrxReport extension to be referenced by the test project. For projects that have not referenced it, passing those flags fails at runtime with an unknown-option error. Following the same principle that removed --auto-reporters off (xUnit-v3-specific), drop these from the forced command and let users opt in via the MTPAdditionalArguments property. With MTP's --auto-reporters on by default, projects that DO reference the TrxReport extension produce a TRX file automatically. arcade's TRXFormat reporter globs *.trx in the work item directory so the auto-generated filename is fine; --report-trx-filename is only needed when a caller wants a deterministic name (then they pass it via MTPAdditionalArguments). The generated command now passes only the built-in --results-directory . (an MTP-core option that works on every framework) so reporter output lands in the work item working directory where Helix collects results. - Drop TrxReportFilename task parameter and MTPTrxReportFilename property - Update CreateMTPWorkItems comment and reporterArgs - Drop CustomTrxReportFilenameIsQuoted and TrxReportFilenameIsPassedThroughVerbatim tests - Update remaining tests to assert the new --results-directory-only shape - Update Readme.md MTPAdditionalArguments example to include --report-trx for callers who want filename control Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The targets file's introductory comment contained literal MTP option
names with double-dash prefixes inside an XML <!-- ... --> block, which
XML does not allow ('-- cannot appear inside a comment'). MSBuild
refused to load the file with MSB4024 and every Helix-using stage
failed. Rephrase the comment to defer reporter/auto-reporter wording
to the Helix Sdk Readme so the targets file stays parseable.
Also update the CreateMTPWorkItems.MTPProjects XML doc to describe the
new command shape - it no longer says 'after the auto-injected reporter
flags' (no flags are injected anymore) and now points to the actual
ordering: built-in --results-directory, then MTPAdditionalArguments,
then per-project Arguments metadata.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ViktorHofer
approved these changes
Jun 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Breaking change. Renames the existing
XUnitV3ProjectHelix item type toMTPProjectand changes the generated work item command to use only Microsoft.Testing.Platform's built-in flags (no forced reporter args).xUnit v3 with the Microsoft.Testing.Platform (MTP) runner uses the exact same self-hosting
dotnet exec ... <assembly>.dllexecution model as MSTest 4.x, NUnit-MTP, TUnit, and any other MTP-based framework. The previousXUnitV3Projectand the originally-proposed newMTPProjectwere 95% the same code; only the reporter args differed. This PR collapses that duplication.MTPProjectcovers any MTP-based test project: MSTest 4.x, xUnit v3 with MTP, NUnit with the MTP runner, TUnit, custom MTP.What this contains
CreateXUnitV3WorkItems->CreateMTPWorkItems,tools/xunitv3-runner/->tools/mtp-runner/,XUnitV3Runner.{props,targets}->MTPRunner.{props,targets}.UseMicrosoftTestingPlatformRunnerswitch and the legacy-xml/-noAutoReportersxunit-v3 code path. MTP is now the only execution mode for this Helix item type; users still on legacy xunit-v3 native or VSTest hand-author a<HelixWorkItem>.--results-directory .(built into the MTP core, accepted by every MTP framework), so reporter output lands in the work item working directory. Reporter selection (--report-trx,--report-trx-filename ...) and framework-specific flags (xUnit v3's--auto-reporters off) are NOT forced - they are extension/framework-specific and would error on frameworks that have not opted in to that extension.MTPAdditionalArgumentsproperty / task parameter as the escape hatch for any extra MTP flags users want applied across every emitted work item.Microsoft.DotNet.Helix.Sdk.props(UsingTask), the Helix SDK Readme section,Documentation/AzureDevOps/SendingJobsToHelix.md, andtests/UnitTests.proj(arcade's own self-test).Generated work item command
With MTP's
--auto-reporterson by default andMicrosoft.Testing.Extensions.TrxReportreferenced (transitive forMSTest.Sdkand for arcadeXUnitV3.targets), the TRX reporter activates automatically and writes a default-named*.trxfile into the work item directory; arcade's existing TRXFormat Python reporter globs*.trxso the auto-generated filename is fine.Why a breaking rename instead of additive
The previous draft of this PR (which left
XUnitV3Projectin place and added a parallelMTPProject) was reviewed offline. Conclusion: the two paths are identical except for the reporter flag, so duplicating the task + targets + props infrastructure is purely accidental complexity. A rename is cheaper to maintain and clearer to users.Downstream impact
A code search across
dotnet/*(and public GitHub) showsdotnet/efcore/eng/helix.projis the only in-tree consumer ofXUnitV3Project(5 occurrences: 1Include+ 4Remove). efcore already setsUseMicrosoftTestingPlatformRunner=trueintest/Directory.Build.props, so they're on the MTP code path today.Migration for efcore is mechanical:
<XUnitV3Project><MTPProject>--report-xunit-xml-filename testResults.xml)Microsoft.Testing.Extensions.TrxReportMicrosoft.Testing.Extensions.XunitXmlReport(transitive via arcadeXUnitV3.targets)Microsoft.Testing.Extensions.TrxReport(also transitive via arcadeXUnitV3.targets)--auto-reporters off<MTPAdditionalArguments>--auto-reporters off</MTPAdditionalArguments>Concretely: 5 occurrences of
XUnitV3Projectineng/helix.projget renamed toMTPProject. If they want to preserve the current--auto-reporters offbehaviour they add<MTPAdditionalArguments>--auto-reporters off --report-trx --report-trx-filename testResults.trx</MTPAdditionalArguments>(the explicit--report-trxis needed when--auto-reportersis off). Otherwise the TRX auto-reporter picks up automatically. A follow-up PR against efcore tracking this migration will be opened before this PR is merged (see #17001 for the broader rollout-coordination tracking).Prerequisites for users
Test projects must reference
Microsoft.Testing.Extensions.TrxReport:MSTest.Sdkprojects: included transitively, nothing to do.Microsoft.DotNet.Arcade.Sdk'sXUnitV3targets: included implicitly, nothing to do.<PackageReference Include=""Microsoft.Testing.Extensions.TrxReport"" />to the test project.AzureDevOps reporter caveat
The MTP
Microsoft.Testing.Extensions.AzureDevOpsReportonly activates when--report-azdois explicitly passed ANDTF_BUILD=true. We never pass--report-azdo, so there is no risk of a duplicate AzDO test run on top of the one the Helix SDK already opens. If a user has wired--report-azdointo their test project they should drop it for Helix runs, or pass--auto-reporters offviaMTPAdditionalArguments.Out of scope
--list-tests+--treenode-filter.Microsoft.Testing.Extensions.HelixReportMTP extension (would live in microsoft/testfx).HELIX_DUMP_FOLDERawareness inCrashDump/HangDumpMTP extensions (microsoft/testfx).--roll-forward Majoris pre-existing and unchanged fromCreateXUnitV3WorkItems; tracking removal/narrowing in Helix Sdk MTP/xUnit work items unconditionally pass --roll-forward Major which can silently mask missing runtimes #17000.Tracking
--roll-forward Majorfollow-up, pre-existing)