Skip to content

Helix Sdk MTP/xUnit work items unconditionally pass --roll-forward Major which can silently mask missing runtimes #17000

@Evangelink

Description

@Evangelink

Spun off from review feedback on #16986 (comment) by @Youssef1313.

Problem

In CreateMTPWorkItems (and previously in CreateXUnitV3WorkItems) the emitted Helix work item command unconditionally passes --roll-forward Major:

string command = `$"{PathToDotnet} exec --roll-forward Major " +`
    `$"--runtimeconfig {assemblyBaseName}.runtimeconfig.json " +`
    `$"--depsfile {assemblyBaseName}.deps.json " +`
    `$"{assemblyName} {reporterArgs}"` + ...

Quoting @Youssef1313:

This [is] extremely extremely concerning to me. Why do we roll forward to Major?

I'm reading this as ""if I have a test project that targets net8.0 and net9.0, and .NET 8 runtime wasn't found, then both TFMs will run with .NET 9, causing silent loss of coverage.""

This behavior is identical to what was in the previous CreateXUnitV3WorkItems task, so #16986 is not the cause - it's a pre-existing issue that the rename inherited.

Proposed direction

  • Investigate whether --roll-forward Major is still required (it was likely added because Helix queues did not always have an exact runtime match available; we should confirm whether queue images now ship a matching runtime per TFM).
  • If unconditional Major roll-forward is no longer needed, drop it (or make it opt-in via metadata on MTPProject / a property).
  • If it is still needed for some queues, narrow the policy (e.g. Minor or LatestMinor) to avoid silently running net8.0 work items on a net9.0 runtime.

Impact

Silent loss of multi-TFM coverage is the worst-case outcome - failures look green because the assembly happily runs on a newer runtime instead of failing fast.

cc @Youssef1313 @ilyas1974

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions