TestsFilter arguments not working on NUnit Tests

[kamilchodola]

Hello,

In my tests i wanted to use argument --testsFilter to pick tests from 2 categories eg.:
--testsFilter="test.Categories.Contains(\"Cat1\") AND test.Categories.Contains(\"Cat2\")".

When I use an argument like this, my testSuite is not filtered to tests with these two categories. It starts to run all tests from an assembly. I must use Test Selection Language from NUnit passed in --nativeRunnerArguments argument:
--where cat==Cat1&&cat==Cat2
But here is a small problem too, because I need to write it like above, without whitespaces and quotation marks. In the example, on the NUnit page, there is an example like below:
nunit3-console mytest.dll --where "cat == Urgent || Priority == High"

If I use whitespaces without quotation marks it starts filtering to bad tests.

[angelovstanton]

Mm, you need to use the filter for dotnet vstest since underneath we use it to run the tests.

https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-vstest?tabs=netcore21

[angelovstanton]

Can you try the format described in their official documentation and tell me whether it worked?

[kamilchodola]

Format from VStest properly filtered tests in my case. But I have NUnit framework, so it not started. I can only verify by sum od tests printed in the console.

[angelovstanton]

If your projects are on .NET Core or Standard and you install needed nuget packages, you will be able to run not only MSTest tests with dotnet vstest but NUnit as well.

[kamilchodola]

OK, I probably know why it doesn't work. I have a category attached to the whole fixture, not to each test. In that case, Meissa returns 0 categories on filtering. Is that correct? I know in the code it's made for a reason, but I think when we give a category for the whole fixture, each test in this fixture is in this category. When we are using Test Selection Language in NUnit using --where cat=Cat1 we choose all tests from fixtures with category Cat1 and all tests with category Cat1.

[angelovstanton]

If you have 10 minutes for an experiment I can guide you how to debug the plugin and figure out what exactly is the problem. You will just need to create a console app, reference the nunit plugin and create instance of NativeTestsRunnerTestCasesPluginService class and call the method ExtractAllTestCasesFromTestLibrary supplying the path to your dll. Run it in debug and see what is happening.

[kamilchodola]

I am debugging it already from the source code of Meissa ;) I attach to a dotnet.exe process and make few breakpoints. I made an experiment just 5 minutes ago. I have a class like a below

[TestFixture]
[Category("Cat1")]
public class TestClass
{
  [Test]
  public void Test1{}
  [Test]
  public void Test2{}
}

When i start Meissa Runner with --testsFilter="test.Categories.Contains("Cat1"), method FilterCases return 0 tests.

After that, I change the implementation of a fixture to:

[TestFixture]
public class TestClass
{
  [Test][Category("Cat1")]
  public void Test1{}
  [Test][Category("Cat1")]
  public void Test2{}
}

this time it works good, retrive both tests and split them into agents correctly.

[angelovstanton]

If I remember correctly we get the attributes only of the test methods, this is why

[kamilchodola]

Yes, I see it on the code. But is it correct? I made a workaround by passing category for each test case (luckily it was only 10 tests cases in a situation I needed).

[angelovstanton]

To be honest not sure. Maybe it should take the types as well.