test-case-filter doesn’t accept multiple classes with OR in stryker-config.json

[mehmetpekdemir]

Hello,
I have a problem and I’d like to explain it to you in detail.
I’m using dotnet-stryker version 3.13.2.
My project is targeting .NET 8.0 and I'm using xUnit.

My stryker-config.json file looks like this:

{
  "stryker-config": {
    "reporters": [
      "json"
    ],
    "test-case-filter": "FullyQualifiedName=Test.RedisChannelManagerTests | FullyQualifiedName=Test.ForwarderTests"
  }
}

I run it with the following command:

dotnet stryker --config-file stryker-config.json

When I try to run this, I get an error.
I provided the test names using their namespace and class names.
What I want is for only the tests listed above to be executed, and the rest to be filtered out.

Can you help me?
Why isn’t this working?

Also,

I’ve tried different syntaxes for test-case-filter.
So far, the only one that worked was:

{
  "stryker-config": {
    "reporters": [
      "json"
    ],
    "test-case-filter": "FullyQualifiedName!~Test.RedisChannelManagerTests"
  }
}

However, I want to filter using multiple tests with an OR condition.
Can you help me with this?

[rouke-broersma]

We pass the test case filter into vstest directly, if it doesn't work I'm not sure it's a valid test case filter: https://github.com/microsoft/vstest/blob/main/src/Microsoft.TestPlatform.ObjectModel/Client/TestPlatformOptions.cs#L22

I looked it up and it seems like your syntax is supposed to work. What is the error you get?
Also would it not be easier to create a category for test suites that should be included or excluded from mutation testing instead of using fully qualified name?

[mehmetpekdemir]

Version: 3.13.2

[14:35:50 INF] Analysis starting.
[14:35:50 INF] Identifying projects to mutate in /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/InstantNotification.sln. This can take a while.
A new version of Stryker.NET (4.5.1) is available. Please consider upgrading using dotnet tool update -g dotnet-stryker

[14:35:53 INF] Found 4 source projects
[14:35:53 INF] Found 1 test projects
[14:35:53 INF] Found project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Infrastructure/Infrastructure.csproj to mutate.
[14:35:53 INF] Found project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Api/Api.csproj to mutate.
[14:35:53 INF] Found project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Domain/Domain.csproj to mutate.
[14:35:53 INF] Found project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Application/Application.csproj to mutate.
[14:35:53 INF] Analysis complete.
[14:35:53 INF] Building solution InstantNotification.sln
[14:36:02 WRN] Project '/Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/test/Test/Test.csproj' did not report any test. This may be because the test adapter package, xunit.runner.visualstudio, failed to deploy. Check if any dependency is missing or there is a version conflict.
[14:36:02 INF] Number of tests found: 0 for project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Infrastructure/Infrastructure.csproj. Initial test run started.
[14:36:02 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:02 WRN] Project '/Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/test/Test/Test.csproj' did not report any test. This may be because the test adapter package, xunit.runner.visualstudio, failed to deploy. Check if any dependency is missing or there is a version conflict.
[14:36:02 INF] Number of tests found: 0 for project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Api/Api.csproj. Initial test run started.
[14:36:02 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:02 WRN] Project '/Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/test/Test/Test.csproj' did not report any test. This may be because the test adapter package, xunit.runner.visualstudio, failed to deploy. Check if any dependency is missing or there is a version conflict.
[14:36:02 INF] Number of tests found: 0 for project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Domain/Domain.csproj. Initial test run started.
[14:36:02 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:02 WRN] Project '/Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/test/Test/Test.csproj' did not report any test. This may be because the test adapter package, xunit.runner.visualstudio, failed to deploy. Check if any dependency is missing or there is a version conflict.
[14:36:02 INF] Number of tests found: 0 for project /Users/mehmet.pekdemir/Desktop/gatek/dotnet-mutation-test-runner/script/mutationtestproject/src/Application/Application.csproj. Initial test run started.
[14:36:02 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:03 WRN] Failed to load analyzer 'Microsoft.CodeAnalysis.Razor.Compiler': (error : ReferencesNewerCompiler, analyzer: All).
[14:36:03 WRN] The analyzer 'Microsoft.CodeAnalysis.Razor.Compiler' references a newer version (4.9.0.0) of the compiler than the one used by Stryker.NET.
[14:36:03 WRN] Failed to load analyzer 'Microsoft.CodeAnalysis.Razor.Compiler': (error : ReferencesNewerCompiler, analyzer: All).
[14:36:03 WRN] The analyzer 'Microsoft.CodeAnalysis.Razor.Compiler' references a newer version (4.9.0.0) of the compiler than the one used by Stryker.NET.
[14:36:04 INF] 347 mutants created
[14:36:04 INF] Capture mutant coverage using 'CoverageBasedTest' mode.
[14:36:04 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:04 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:04 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:04 WRN] Runner 0: Test assembly does not contain any test, skipping.
[14:36:04 INF] 5 mutants got status CompileError. Reason: Mutant caused compile errors
[14:36:04 INF] 92 mutants got status NoCoverage. Reason: Not covered by any test.
[14:36:04 INF] 26 mutants got status Ignored. Reason: Removed by block already covered filter
[14:36:04 INF] 224 mutants got status Ignored. Reason: Removed by exclude from code coverage filter
[14:36:04 INF] 347 total mutants are skipped for the above mentioned reasons
[14:36:04 INF] 0 total mutants will be tested
[14:36:04 WRN] It looks like all mutants with tests were ignored. Try a re-run with less ignoring!
[14:36:04 WRN] It looks like all non-ignored mutants are not covered by a test. Go add some tests!
[14:36:04 WRN] It looks like all mutants resulted in compile errors. Mutants sure are strange!

my logs are the above.
The test classes I want to ignore don’t have a consistent naming or namespace pattern, so I prefer not to group them and instead list them individually.

[mehmetpekdemir]

Can you help me ?

[rouke-broersma]

I don't see any error from stryker, does the mutation testrun find unit tests if you run without test case filter?

[mehmetpekdemir]

Yes, it’s working successfully.
in fact, even the following filter works perfectly fine:

{
  "stryker-config": {
    "reporters": [
      "json"
    ],
    "test-case-filter": "FullyQualifiedName!~Test.RedisChannelManagerTests"
  }
}

I can apply this filter without any issues.
However,
if I try to use OR to include multiple classes — regardless of whether I use and or or — I get errors.
The error logs are the same as the ones I shared above.

[mehmetpekdemir]

?