dotnet test <dll> --filter parameter is ignored when there are a large number of tests. All tests in all classes are run regardless of filter.

[fenjas]

• NUnit and NUnit3TestAdapter versions: 3.13.3 and 4.3.1
• Visual Studio edition and full version number (see Help About): VS 2022 17.4.2 (v 4.8.04084)
• What .net platform and version is being targeted: .NET 6.0

I have a VS C# solution with 3 test classes as per pic attached. I'm using a PowerShell script to execute these tests on a schedule using the following statements:

• dotnet test biditests.dll --filter TestCategory~Daily
• dotnet test biditests.dll --filter TestCategory~Monthly
• dotnet test biditests.dll --filter TestCategory~GroupBrandNames

The --filter parameter seems to be ignored, so when I run the dotnet command targeting ~Daily, all the tests in all 3 classes are executed instead of the ones targeted by the filter.

If I execute the dotnet command targeting ~GroupBrandNames or ~Monthly, the filter works fine presumably because there are only 2 and 60 tests respectively in each class. If I keep the no. of tests in the Daily class under 2000, everything works as expected. As of now it contains in excess of 3000 tests.

I also tried other ways to run the tests using the project instead of the the dll, and changing the filter to using the FQDN, etc. However, the issue is the number of tests.

I found an article which mentioned creating a .runsettings file and upping the AssemblySelectLimit from 2000 to something greater than the number of tests. I did this, however it does not seem to work (assuming I did it correctly, file attached) and from what I understood applies only when you're running tests from the IDE which is not my case.

As a fix of sorts, I ended up using a Environment.Exit() call in the OneTimeTearDown method after the last test fixture is executed so the runner exits before getting a chance to run tests from another class. It's ugly but it works for now.

Logging this here since I found a similar issue and thought mine would be related!

Thanks.

[OsirisTerje]

1. Add a runsettings file with Verbosity=5 and see what the filter expression internally is.
2. Try to use the NUnit Where syntax and see what that gives and compare the filter.
3. See if you can provide a simple repro solution for this

Information about the settings is here: https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html

[fenjas]

1. ok, right now verbosity is set to 0. Setting it to 5.
2. ok
3. Will try to

Thanks!

[OsirisTerje]

Just in case: To get the output from dotnet test, remember to set the consolelogger.
E.g.:

[fenjas]

Turns out that adding -s .runsettings to the dotnet command with AssemblySelectLimit set to a value greater than the total number of tests does seem to address the issue. Regardless, I created a C# solution demonstrating this behaviour and 'fix' -> https://github.com/fenjas/Nunit2000TestsLimit

At the risk of being pedantic though, I noticed that the total number of Passed tests is the same in both cases i.e when the filter is applied and when it is ignored. See the output in the no_runsettings.txt and with_runSettings.txt included.

I still have to try this out on the actual tests I run live so will follow up once I do.

Test Run Successful.
Total tests: 2280
Passed: 2280
Total time: 2.6461 Seconds

Test Run Successful.
Total tests: 2280
Passed: 2280
Total time: 3.1467 Seconds

[torers-dnv]

Yes, seems as filter behavior changes when exceeding 2000 tests. Filter is applied on results after running tests rather than excluding tests from running in the first place, which is weird and unexpected.
Thank you for the AssemblySelectLimit workaround @fenjas , it reduced the build time by 12 minutes on our TeamCity build.

[OsirisTerje]

If you're using dotnet test you can also set the limit directly without creating a .runsettings file.

dotnet test <somefilterexpression> -- NUnit.AssemblySelectLimit=99999

Also see https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#assemblyselectlimit