TestFilter does not support test cases with spaces and special characters

[plainionist]

Running "dotnet test" with a test filter for a test case with spaces the following exception occurs:

dotnet test --settings ../../../build/.runsettings --no-build --logger "trx;LogFileName=C:\Temp\0.trx" --filter "FullyQualifiedName="BackLook.Apps.WorkInProgress.Specs.Computing work in progress Project.No Workpackages exist"" --no-build

Test run for C:\ws\BackLook\system\App.WorkInProgress\BackLook.Apps.WorkInProgress.Specs\bin\Debug\net48\BackLook.Apps.WorkInProgress.Specs.dll (.NETFramework,Version=v4.8)
Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
An exception occurred while invoking executor 'executor://nunit3testexecutor/': Unexpected Word 'work' at position 64 in selection expression.
Stack trace:
at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Expect(Token[] valid) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 252
at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Parse(String input) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 66
at NUnit.VisualStudio.TestAdapter.NUnitTestFilterBuilder.ConvertVsTestFilterToNUnitFilter(IVsTestFilter vsFilter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestFilterBuilder.cs:line 72
at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 116 at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension2 executor, Tuple2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass48_0.<RunTestInternalWithExecutors>b__0() at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.<Run>b__0() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInSTAThread(Action action, Boolean waitForCompletion) at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable1 executorUriExtensionMap, Int64 totalTests)

according to https://docs.microsoft.com/de-de/dotnet/core/testing/selective-unit-tests?pivots=mstest spaces and special characters can be encoded using "url style encoding" but that causes the same exception.

example: --filter "FullyQualifiedName=BackLook.Apps.WorkInProgress.Specs.Computing%20work%20in%20progress%20Project.No%20Workpackages%20exist"

i am using

<PackageReference Include="nunit" Version="3.12.0" />
<PackageReference Include="NUnit3TestAdapter" Version="4.0.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.3.0" />

and "dotnet cli" version 5.0.301

[OsirisTerje]

This is a known issue, part of the FQN issues.

[plainionist]

any plans for adding support for such characters?

[OsirisTerje]

@plainionist
This issue seems now to be fixed in the latest version of Visual Studio 2019, 16.11.3.

See repro here: https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue711

Note that the '.' in the name causes Test Explorer to add an extra hierarchy node. The Test Explorer uses the '.' in the testname as a node separator.

Works both with and without Real-time Test Discovery

[plainionist]

@OsirisTerje
I just updated VS to 16.11.5 and the problem still exists. I also doubt that the issue is related to VS but to unit3-vs-adapter.
as described above my use case is NOT running tests with spaces in VS itself but on the console using dotnet CLI.
The error message clearly indicates that the filter parser is not supporting my use case:

An exception occurred while invoking executor 'executor://nunit3testexecutor/': Unexpected Word 'work' at position 64 in selection expression.
Stack trace:
at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Expect(Token[] valid) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 252
at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Parse(String input) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 66
at NUnit.VisualStudio.TestAdapter.NUnitTestFilterBuilder.ConvertVsTestFilterToNUnitFilter(IVsTestFilter vsFilter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestFilterBuilder.cs:line 72

==> pls reopen the issue

thx

[OsirisTerje]

Reopened.

@plainionist Can you confirm the repro at https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue711 reproduces your issue? If not, can you update it so that it does?

[plainionist]

as far as i can see in the code the repro would probably sufficient to reproduce the issue, at least i see spaces in the test cases.
the question is: how do u execute it. if we call it from "dotnet" CLI with a filter like

"--filter "FullyQualifiedName=""

then we would probably get the same issue.

dotnet test --filter "FullyQualifiedName=Unit To Test Input A equals itself"

results in

An exception occurred while invoking executor 'executor://nunit3testexecutor/': Unexpected Word 'To' at position 24 in selection expression.
Stack trace:
   at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Expect(Token[] valid) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 255
   at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Parse(String input) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 66
   at NUnit.VisualStudio.TestAdapter.NUnitTestFilterBuilder.ConvertVsTestFilterToNUnitFilter(IVsTestFilter vsFilter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestFilterBuilder.cs:line 73
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 116
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

putting the test name in quotes gives similar issues

dotnet test --filter "FullyQualifiedName=\"Unit To Test Input A equals itself\""
dotnet test --filter "FullyQualifiedName='Unit To Test Input A equals itself'"

[ThisFunctionalTom]

Hi,

we have the same problem with version 4.1.0.

[nojaf]

Hello, I'm facing a similar issue where my test name is having spaces.
I noticed the tokenizer is skipping these spaces here:

nunit3-vs-adapter/src/NUnitTestAdapter/TestFilterConverter/Tokenizer.cs

Lines 297 to 301 in 0b118ec

	private void SkipBlanks()
	{
	while (char.IsWhiteSpace(NextChar))
	index++;
	}

Is there any reason for this? Perhaps, we can make it more context aware and not always skip the spaces.

[OsirisTerje]

The problem with spaces in names is that the test names are very close to FQN for classes and namespaces, and as they can't have spaces, the test names can't either. The real time discovery in Visual Studio would not be able to handle them. For categories using string parameters, it can in principle have spaces, but once they are made like typed categories, they can't.
One could imagine a mode where this is allowed, but it would not work properly in Visual Studio, and it would require an increase in complexity to handle both scenarios.
An alternative is to make a difference between name and displayname (there already is, but one could use this more expressively), perhaps inventing a new kind of displayname, being a mixture of the name and the displayname. Then, it should work all the way down to the framework.
If anyone wants to contribute something here, we would be happy for that. Bear in mind that any changes have to be compatible and work with the current "way of working".

[nojaf]

Well, I'm running this from the command line, so I wonder if there could be a later point in time where the spaces could be stripped for Visual Studio.

[OsirisTerje]

The adapter tries to deduct from where it is being called, there are properties being set to that effect. But you then have to bring that information all the way down through the code, or it could swap in different tokenizers based on that, so surely, it could be doable. Would you like to give it a try?

[nojaf]

I might want to give this a try, could you point me in the right direction where the call site deduction is done?
I would then try to pass this information to the tokenizer. Any advice on how to write a test for this scenario would also be appreciated.

[OsirisTerje]

You should not pass the information to the tokenizer, but create a new (derived) class for it, so that you change the method there. That way the existing code is not changed.
The property to check for is in the TestAdapter base class, and is IsRunningUnderIde.

[NastassiaDzianisava]

Hello. I have a similar issue

##[error]An exception occurred while invoking executor 'executor://nunit3testexecutor/': Unexpected Word 'test' at position 15 in selection expression.
##[debug]Processed: ##vso[task.logissue type=error;]An exception occurred while invoking executor 'executor://nunit3testexecutor/': Unexpected Word 'test' at position 15 in selection expression.
##[error]Stack trace:
##[debug]Processed: ##vso[task.logissue type=error;]Stack trace:
##[debug]Processed: ##vso[task.logissue type=error;]Stack trace:
##[error] at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Expect(Token[] valid) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 252
##[debug]Processed: ##vso[task.logissue type=error;] at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Expect(Token[] valid) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 252
##[error] at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Parse(String input) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 65
##[debug]Processed: ##vso[task.logissue type=error;] at NUnit.VisualStudio.TestAdapter.TestFilterConverter.TestFilterParser.Parse(String input) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\TestFilterConverter\TestFilterParser.cs:line 65
##[error] at NUnit.VisualStudio.TestAdapter.NUnitTestFilterBuilder.ConvertVsTestFilterToNUnitFilter(IVsTestFilter vsFilter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestFilterBuilder.cs:line 72
##[debug]Processed: ##vso[task.logissue type=error;] at NUnit.VisualStudio.TestAdapter.NUnitTestFilterBuilder.ConvertVsTestFilterToNUnitFilter(IVsTestFilter vsFilter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitTestFilterBuilder.cs:line 72
##[error] at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 116 ##[debug]Processed: ##vso[task.logissue type=error;] at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 116
##[error] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension2 executor, Tuple2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
##[debug]Processed: ##vso[task.logissue type=error;] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension2 executor, Tuple2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
##[error] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass48_0.b__0()
##[debug]Processed: ##vso[task.logissue type=error;] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.<>c__DisplayClass48_0.b__0()
##[error] at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.b__0()
##[debug]Processed: ##vso[task.logissue type=error;] at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.<>c__DisplayClass0_0.b__0()
##[error]--- End of stack trace from previous location where exception was thrown ---
##[debug]Processed: ##vso[task.logissue type=error;]--- End of stack trace from previous location where exception was thrown ---
##[error] at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion)
##[debug]Processed: ##vso[task.logissue type=error;] at Microsoft.VisualStudio.TestPlatform.PlatformAbstractions.PlatformThread.Run(Action action, PlatformApartmentState apartmentState, Boolean waitForCompletion)
##[error] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInSTAThread(Action action, Boolean waitForCompletion)
##[debug]Processed: ##vso[task.logissue type=error;] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.TryToRunInSTAThread(Action action, Boolean waitForCompletion)
##[error] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable1 executorUriExtensionMap, Int64 totalTests) ##[debug]Processed: ##vso[task.logissue type=error;] at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable1 executorUriExtensionMap, Int64 totalTests)

[OsirisTerje]

@AnastasiaDen16 And what is your test expression ?

[NastassiaDzianisava]

@OsirisTerje
Visual studio 16.10.4
NUnit version="3.12.0"
NUnit3TestAdapter version="4.0.0"

[OsirisTerje]

@AnastasiaDen16 OK, but your test expression ? What is it ? There are known FQN issues, is yours different and what is it?

[NastassiaDzianisava]

@OsirisTerje I don't understand, what do you mean. I have this issue the first time.

[OsirisTerje]

@AnastasiaDen16 This issue is about testfilters with spaces and special characters. Does your situation have the same? The error message says you have an unexpected word "test" in the selection expression. How does your selection expression look like? You say you run from Visual Studio, then I expect you have something in the Search part of the Test Explorer window, right? What is that? Next, Is there something special about the test code you have, can you shorten down that code and show how the method definition there look like, or even better, provide a very small repro solution. Does this help?

[NastassiaDzianisava]

@OsirisTerje Thank you, I understand what the problem is. Yes, I have a space in the category. As I understand, this way of describing the category is unacceptable, right?

[OsirisTerje]

Yes, it will not work for now. We'll see if we can fix this over time, but the workaround is to make it into a single word.

[NastassiaDzianisava]

@OsirisTerje Ok, thank you for answer.

[OsirisTerje]

@NastassiaDzianisava A small tip: When I create categories, I normally make them as typesafe classes, as shown in the figure below:

That way I avoid misspelling when they're being used too.

[jake8787]

Tried to use new libraries again, but it still hasn't been fixed
So I'm back to good old
NUnit 3.12.0
NUnit3TestAdapter 3.17.0
Whitespaces work there

[plainionist]

@OsirisTerje

The problem with spaces in names is that the test names are very close to FQN for classes and namespaces, and as they can't have spaces, the test names can't either.

In F# i can use spaces in class and method names like this:

[<TestFixture>]
module ``Calculating team availabilities (Fruits)`` =

if i then use dnSpy to look into the compiled code it looks like this

[TestFixture]
[CompilationMapping(SourceConstructFlags.Module)]
public static class Calculating\u0020team\u0020availabilities\u0020(Fruits)

[OsirisTerje]

@plainionist Good point! Now where that is failing, is then the question. ........

[plainionist]

@OsirisTerje i think the missing support is in the parser of the filter at least as far as i understood the callstack of the error i posted at the beginning of this thread

[OsirisTerje]

@plainionist I am a bit out of available time at the moment, but if you feel you see where it is, a and how it could be fixed, a PR would be very appreciated.

[ankostis]

The problem seems to be in the NUnitTestAdapter/TestFilterConverter/Tokenizer.cs:Tokenizer.GetNextToken() method, which does not support any quoting or escape characters (eg. backslash).

Till then, a workaround is to AND together all the valid parts of the rhs value, like this:

--filter "testcase&with&spaces"

(reported also in this SO)

[Smaug123]

I hit this myself yesterday. Reading the thread I'm not sure what is causing the symptoms. If I'm understanding correctly, there are a couple of distinct problems:

• There's no way to specify certain valid filters on the command line to NUnitTestAdapter.
• VS can't handle valid test names with spaces and full stops in.

The former is pretty sad if true - it's an instance of the classic problem "I have a tree-like data structure which the shell forces me to provide as a string". The usual solution is to allow you to specify it in a tree serialisation format such as JSON (e.g. --filter-json '{"FullyQualifiedName": "blah"}' or --filter-json '{"op": "And", "operand": [{"FullyQualifiedName": "blah"}, {"op": "Not", "operand": {"FullyQualifiedName": "blah"}}]}').

The latter is just a bug in VS, right? Can they not fix it?

[OsirisTerje]

1. There is probably a bug in the token parser that changed at some time, see @jake8787 comment above. Introducing Json format is interesting, but it will have to be handled all the way down to the framework. Otherwise it would just be another way to create a string based filter.
2. They can not "just fix it", because the Test Explorer (and thus also dotnet test) assumes that tests are following C# FQN. That is "by design". NUnit has the concept and it might work using the NUnit.Console or NUnitLite runners. Please try one of them.

Also, there are many comments on this, and many people interested - we would be very happy to receive PRs for this. First thing would be to confirm (1) above, based on the repro published, to see if that is actually a regression bug. Once it is confirmed, it should be possible to check what changed between those two versions and then provide a fix - through a PR. If you have time @Smaug123 it would be awesome if you could do something with this.

[Smaug123]

Ah, I see that the filters are ultimately already expressed in XML. There would seem to be an easy targeted fix, then: in TestFilterParser.Parse, check if the input is valid XML with an outer node of <filter></filter>, and just use it unchanged if it is. That would seem to provide maximum flexibility without changing any existing use cases?

[OsirisTerje]

@Smaug123 That is correct. That is a possible fix by adding that option. But, I am not sure if too many would use that, so also going for fixing the probable bug in the filter test parser would be awesome.

If the current filter statements results in a n invalid xml , for some inputs, we're in trouble. Setting up a set of testcases to verify if that is the case or not, is also a way of checking the current test filter parser.

Also note there is a similar issue related to the syntax of Explicit that causes filter problems. That one could also be handled at the same time.