Behavior changes between MTP and Legacy (msbuild)

[james-newell-forge]

Hi Team,

I first want to say thank you for making this tool and continuing to upgrade it to the MTP test runner.

I was exploring moving our project to MTP while upgrading to dotnet 10.0 and one thing I noticed was that I was getting different coverage results between using legacy and MTP. I (really Claude AI) managed to track it down to a few behavior changes between the two and was wondering if we could have a discussion about giving the option to restore the original behavior.

SkipAutoProps is true by default

So, in legacy SkipAutoProps is false by default. This wouldn't be a big deal but you actually can't set this option to false using the option --coverlet-skip-auto-props. The following code fragment shows the bug: if you don't set it, the value is true by default but if you do set it, it is also true.

CoverageConfiguration.cs

public bool SkipAutoProps =>
    GetBoolOptionWithDefault(CoverletOptionNames.SkipAutoProps, defaultValue: true);

private bool GetBoolOptionWithDefault(string optionName, bool defaultValue)
{
    bool isSet = _commandLineOptions.IsOptionSet(optionName);
    if (isSet)
    {
        _logger?.LogDebug($"[Explicitly set] {optionName}: true");
        return true;
    }

    _logger?.LogDebug($"[Default] {optionName}: {defaultValue}");
    return defaultValue;
}

Additional default ExcludeByAttributes

In legacy, only the ExcludeFromCodeCoverageAttribute attribute is included by default, but in MTP it includes GeneratedCodeAttribute and CompilerGeneratedAttribute attributes which seems to be added to compiled code automatically by the SDK. This caused coverage discrepencies during testing. There seems to be no way to change the defaults. Maybe we could add an option to exclude defaults as an escape hatch (maybe --coverlet-no-exclude-by-attribute-defaults), or whatever ideas you have.

CoverageConfiguration.cs

private static readonly string[] s_defaultExcludeByAttributes =
[
    "ExcludeFromCodeCoverage",
    "ExcludeFromCodeCoverageAttribute",
    "GeneratedCodeAttribute",
    "CompilerGeneratedAttribute"
];

Timestamp added to coverage file

This one i can definetly see the benefit of having, but can we make it opt-in or some way to disable it via options? For example, we could have an option --coverlet-postfix-timestamp that when set includes the timestamp in the file, otherwise use the default legacy behavior

[james-newell-forge]

One more thing, I could submit a PR if you wanted some dev help with these changes. I already made them on my local, so it wouldn't take much work.

[pinkfloydx33]

I noticed the same as the above and would also benefit from the same escape hatches

Also missing is a way to specify the output directory. I realize it uses the results-directory flag but either an option to completely bypass that or a way to set a sub-folder within said directory would be appreciated. Our CI is expecting coverage to be in a distinct location from test result files. I can manually post-process but its a gap from previous behavior which I could control with msbuild flags

[Bertk]

Thank you for the helpful information. Could you please try the preview version from the Coverlet nightly build? While it may not resolve all issues, it includes improvements such as “Add timestamp to coverage report filenames for uniqueness” (#1870) and “Add --coverlet-file-prefix option for unique report files” (#1869). The preview version also supports the .NET 10 SDK target framework.

The default values can be avoided using a configuration file.

Configuration File Defaults (via coverlet.mtp.appsettings.json):

When using the configuration file, only [coverlet.*]* is automatically prepended to exclude filters.