Filtering tests doesn't work when using EnableNUnitRunner

[aelij]

Running an individual test doesn't work once the new Testing Platform is enabled. The runner always runs all tests. Using dotnet test --filter doesn't work either.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <EnableNUnitRunner>true</EnableNUnitRunner>
    <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
    <OutputType>Exe</OutputType>

    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
    <PackageReference Include="NUnit" Version="4.3.2" />
    <PackageReference Include="NUnit.Analyzers" Version="4.6.0">
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
    <PackageReference Include="NUnit3TestAdapter" Version="5.0.0-beta.5" />
  </ItemGroup>

</Project>

[mikkelbu]

Hi @aelij
I don't know much about the Testing Platform nor the NUnit3TestAdapter, but can you enable the Dump files, see Tips&Tricks https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html, using a runsettings file, and then upload the results.

[OsirisTerje]

I can confirm the reported bug:

The repro code is in https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue1152

The comparison is shown below, in both cases I am trying to run -1- test, but with the EnableNUnitRunner the Test Explorer doesn't filter. Dotnet test does filter correctly however:

dotnet test --filter Name=Test1 --logger "Console;verbosity=normal"

The results from Test Explorer is:

@Evangelink Do you see what happens here ?

[Evangelink]

Hi @OsirisTerje, @aelij,

I am sorry that I didn't have a good look at the issue from the start. This is actually expected behavior because dotnet test is directly tighted to VSTest so any argument that is meant to be used with Microsoft.Testing.Platform needs to be passed down. The easiet is to do dotnet test -- --filter Name=Test1. You have also different syntaxes described at https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#dotnet-test---microsofttestingplatform-mode.

I'll try to see how to improve the documentation there.

[aelij]

@Evangelink Must say that's not very intuitive.... is there a way to transfer the filter param to the test platform?
Also what about VS Test Explorer?

[Evangelink]

Must say that's not very intuitive

I agree but we are fighting we limitations of past design decisions...

is there a way to transfer the filter param to the test platform?

Yes as linked in the doc, you can either do -- like you would do for dotnet run (e.g. dotnet test -- --filter Test1) or you can use -p:TestingPlatformCommandLineArguments (e.g. dotnet test -p:TestingPlatformCommandLineArguments="--minimum-expected-tests 10").

Also what about VS Test Explorer?

By enabling the support for Microsoft Testing Platform in VS:

Everything should be working fine. Note that a similar option exists with Rider.

[aelij]

@Evangelink That preview option is enabled on my machine, and it doesn't work - all tests are executed (only in NUnit, MS and X work fine).

[aelij]

@Evangelink "is there a way to transfer the filter param to the test platform?" - I meant that the platform should do it, not the user :)