SpecFlow tests fail on retry in Azure Pipeline

[cmacgowan]

Hi SpecFlow Community,

Andreas Willich recommended that we post a bug here ...

"Please create an issue at github.com/SpecFlowOSS/SpecFlow I think our FullyQualifiedName Generation has a bug. – Andreas Willich" -- https://stackoverflow.com/questions/64851698/specflow-tests-fail-on-retry-in-azure-pipeline

We are running SpecFlow tests ina CI/CD Azure Pipeline. The SpecFlow Scenario Outline tests are failing during the Azure Pipeline Retry Process.

We are using the following environment.

SpecFlow (3.4.14)
SpecFlow.NUnit (3.4.14)
Nunit.ConsoleRunner (3.11.1)
Azure Pipeline (using VSTest)

SpecFlow Version:

• 3.5
• [X ] 3.4
• 3.3
• 3.1
• 3.0
• 2.4
• 2.3
• 2.2
• 2.1
• 2.0
• 1.9

Used Test Runner

• SpecFlow+Runner
• MSTest
• [X ] NUnit
• Xunit

Version number: Nunit.ConsoleRunner (3.11.1)

Project Format of the SpecFlow project

• [X ] Classic project format using packages.config
• Classic project format using <PackageReference> tags
• Sdk-style project format

.feature.cs files are generated using

• [X ] SpecFlow.Tools.MsBuild.Generation NuGet package
• SpecFlowSingleFileGenerator custom tool

Visual Studio Version

• [X ] VS 2019
• VS 2017
• VS 2015

Enable SpecFlowSingleFileGenerator Custom Tool option in Visual Studio extension settings

• Enabled
• [X ] Disabled

Are the latest Visual Studio updates installed?

• [X ] Yes
• No, I use Visual Studio version <Major>.<Minor>.<Patch>

.NET Framework:

• >= .NET 4.5
• before .NET 4.5
• .NET Core 2.0
• .NET Core 2.1
• .NET Core 2.2
• .NET Core 3.0
• [X ] .NET Core 3.1
• .NET 5.0

Test Execution Method:

• Visual Studio Test Explorer
• [X ] TFS/VSTS/Azure DevOps – Task – PLEASE SPECIFY THE NAME OF THE TASK
  vstest.console.exe /TestCaseFilter:"FullyQualifiedName ... (See attached log for more info)
• Command line – PLEASE SPECIFY THE FULL COMMAND LINE

<SpecFlow> Section in app.config or content of specflow.json

I can't seem to find this - I may continue to look and edit this post :-) 
https://docs.specflow.org/projects/specflow/en/latest/Installation/Configuration.html

Issue Description

Hi SpecFlow Community,

We are running SpecFlow tests ina CI/CD Azure Pipeline. The SpecFlow Scenario Outline tests are failing during the Azure Pipeline Retry Process.

We are using the following environment.

SpecFlow (3.4.14)
SpecFlow.NUnit (3.4.14)
Nunit.ConsoleRunner (3.11.1)
Azure Pipeline (using VSTest)

The Azure Pipeline has a built in feature to retry failed tests. When a test fails the test name is created and passed to VSTest. When a SpecFlow Scenario Outline is used - the test name is build from the Scenario with the various data from the Example table in the Scenario Outline - to build a unique test name.

VSTest will throw an exception at this point (Unhandled Exception: System.ArgumentException: Illegal characters in path) - that will cause the retry process to fail.

Any solutions to creating a better formed test name ???

Below are some log files samples:

**************  VSTest Console  **************

2020-11-14T02:24:33.1789135Z vstest.console.exe /TestCaseFilter:"FullyQualifiedName=LegalHomeTests.Features.LatestNewsFeature.ValidatePracticalLawNewsUpdatedLinkForDifferentRegion(\"UK\"\,\"uk.practicallaw.thomsonreuters.com\"\,\"30\"\,null)|FullyQualifiedName=LegalHomeTests.Features.LatestNewsFeature.ValidatePracticalLawNewsUpdatedLinkForDifferentRegion(\"US\"\,\"content.next.westlaw.com\"\,\"13\"\,null)|FullyQualifiedName=LegalHomeTests.Features.LatestNewsFeature.ValidateToSetNewsPreferencesToViewContentOnLatestNewsTileOnHomePage(\"Commercial Transactions\"\,\"US\"\,null)|FullyQualifiedName=LegalHomeTests.Features.LatestNewsFeature.ValidateToSetNewsPreferencesToViewContentOnLatestNewsTileOnHomePage(\"Agriculture & Rural Land\"\,\"UK\"\,null)|FullyQualifiedName=LegalHomeTests.Features.SmokeTestsFeature.LatestNewsTileLoadsUSContent"
2020-11-14T02:24:33.1791527Z "C:\Users\HQAutomationUser\agent\_work\7\s\Zurich.Test\Zurich.Test.Automation.Framework\AutomationFramework\LegalHomeTests\bin\Debug\netcoreapp3.1\LegalHomeTests.dll"

**************  Exception  **************

2020-11-14T02:24:33.1791527Z "C:\Users\HQAutomationUser\agent\_work\7\s\Zurich.Test\Zurich.Test.Automation.Framework\AutomationFramework\LegalHomeTests\bin\Debug\netcoreapp3.1\LegalHomeTests.dll"
2020-11-14T02:24:33.1792436Z /Settings:"C:\Users\HQAutomationUser\agent\_work\_temp\ve5nmahpehf.tmp.runsettings"
2020-11-14T02:24:33.1795485Z /Logger:"trx"
2020-11-14T02:24:33.1796049Z /TestAdapterPath:"C:\Users\HQAutomationUser\agent\_work\7\s"
2020-11-14T02:24:33.4321502Z ##[error]Unhandled Exception: System.ArgumentException: Illegal characters in path.
2020-11-14T02:24:33.4322624Z ##[error] at System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional)
2020-11-14T02:24:33.4323996Z ##[error] at System.IO.Path.IsPathRooted(String path)
2020-11-14T02:24:33.4324892Z ##[error] at vstest.console.Internal.FilePatternParser.GetMatchingFiles(String filePattern)
2020-11-14T02:24:33.4326608Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.CommandLineOptions.AddSource(String source)
2020-11-14T02:24:33.4327690Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.TestSourceArgumentExecutor.Initialize(String argument)
2020-11-14T02:24:33.4328926Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
2020-11-14T02:24:33.4329826Z ##[error] at System.Lazy`1.CreateValue()
2020-11-14T02:24:33.4330510Z ##[error] at System.Lazy`1.LazyInitValue()
2020-11-14T02:24:33.4331183Z ##[error] at System.Lazy`1.get_Value()
2020-11-14T02:24:33.4332295Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
2020-11-14T02:24:33.4333195Z ##[error] at System.Lazy`1.CreateValue()
2020-11-14T02:24:33.4333880Z ##[error] at System.Lazy`1.LazyInitValue()
2020-11-14T02:24:33.4334546Z ##[error] at System.Lazy`1.get_Value()
2020-11-14T02:24:33.4335637Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
2020-11-14T02:24:33.4336662Z ##[error] at System.Lazy`1.CreateValue()
2020-11-14T02:24:33.4337342Z ##[error] at System.Lazy`1.LazyInitValue()
2020-11-14T02:24:33.4338009Z ##[error] at System.Lazy`1.get_Value()
2020-11-14T02:24:33.4339094Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
2020-11-14T02:24:33.4339994Z ##[error] at System.Lazy`1.CreateValue()
2020-11-14T02:24:33.4340670Z ##[error] at System.Lazy`1.LazyInitValue()
2020-11-14T02:24:33.4341338Z ##[error] at System.Lazy`1.get_Value()
2020-11-14T02:24:33.4342442Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
2020-11-14T02:24:33.4343340Z ##[error] at System.Lazy`1.CreateValue()
2020-11-14T02:24:33.4344101Z ##[error] at System.Lazy`1.LazyInitValue()
2020-11-14T02:24:33.4344772Z ##[error] at System.Lazy`1.get_Value()
2020-11-14T02:24:33.4345732Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.GetArgumentProcessors(String[] args, List`1& processors)
2020-11-14T02:24:33.4346701Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)
2020-11-14T02:24:33.4347635Z ##[error] at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)
2020-11-14T02:24:33.4417125Z Vstest.console.exe exited with code -532462766.

Any help is appreciated
Thanks - Chris ([email protected])

Steps to Reproduce

We are running in the Azure Pipeline. Happy to give you more information - or do a call if you wish.

Repro Project

I can probably arrange this if it would be helpful

[SabotageAndi]

Ah, you are using NUnit. Interesting, as we use only features that are normally available.
We use the normal Test and TestCase attributes.

Probably the Agriculture & Rural is the problem in the command line. That & is not escaped. Could you change that to something else?

It would only be a bug in the FullQualifiedName that we could fix, if you would have used the SpecFlow+ Runner.

[cmacgowan]

Hi Andreas, Thanks - I'll take a look ...

…

On 2020-11-18 04:54, Andreas Willich wrote: Ah, you are using NUnit. Interesting, as we use only features that are normally available. We use the normal Test and TestCase attributes. Probably the Agriculture & Rural is the problem in the command line. That & is not escaped. Could you change that to something else? It would only be a bug in the FullQualifiedName that we could fix, if you would have used the SpecFlow+ Runner. -- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. Links: ------ [1] #2213 (comment) [2] https://github.com/notifications/unsubscribe-auth/ADC7VEUXG2EEUTUJBJX7VCTSQORWHANCNFSM4TZQPOYA

[ryandorendorf]

Hello, I'm on Chris's team in developing these tests.
I have a question and some additional information.

Question

What do you mean by normally available?

Ah, you are using NUnit. Interesting, as we use only features that are normally available.

When I look at the nuget packages i see this is the most common runner used for specflow.

Additional test cases

Also I have run tests without the & symbol in our parameters. I think the issue is because the scenario outline adds parenthesis to the test name and those don't get escaped.
Full log file: SpecFlow_FailedRetryLog.txt

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=LegalHomeTests.Features.SitesFeature.ValidateNoResultFoundOnSitesSubPageNoSpace(\"Favorites\"\,\"FORCEFAIL\"\,null)"
...
NUnit Adapter 3.17.0.0: Test execution started
##[error]An exception occurred while invoking executor 'executor://nunit3testexecutor/': Filter string '"Favorites"\,"FORCEFAIL"\,null' includes unrecognized escape sequence.

Screenshot of Test in Azure Devops

There is a different error if we have a space in our parameter.
Full Log File: SpecFlow_FailedRetryLog_WithSpace.txt

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=LegalHomeTests.Features.SitesFeature.ValidateNoResultFoundOnSitesSubPage(\"Favorites\"\,\"FORCE FAIL\"\,null)"
...
##[error]Unhandled Exception: System.ArgumentException: Illegal characters in path.

Resolution I'm seeking

From all my analysis I think the issue boils down to when we try to retry the tests in Azure devops it puts the FullyQualifiedName of all failed tests into the TestCaseFilter parameter, and when a test from a scenario outline fails we get an error. I believe it is because the test name generated from specflow places parenthesis in the test name and those don't get escaped.

Essentially, we want to know if there is some way to configure or change the naming schema for scenario outlines to not include these parenthesis and use '_' or something instead. I have already met extensively with Microsoft to see if they could resolve this in the VSTest task but found no solution.

I think we would have the same problem with specflow+Runner unless it results in a different naming convention for the tests.

[SabotageAndi]

@ryandorendorf SpecFlow supports xUnit, NUnit, SpecFlow+ Runner and MSTest. That NUnit is the most downloaded NuGet package, but that changes nothing. We are supporting all of them.

What SpecFlow does is at compile time to generate out of the feature files a code-behind file (the feature.cs- files). And there we are using dependent on the test runner, different attributes, that are you also use when you write unit tests.
And here we are using the standard NUnit attributes (TestAttribute and TestCaseAttribute). We don't have any custom extensions for NUnit to connect it with SpecFlow.
We "just" generate coded tests.
And that was what I meant with ... as we use only features that are normally available

I tried to reproduce the issue and I was successful.
The repro is here: https://github.com/SabotageAndi/NUnitRetry2213

When you look at the generated code for this scenario

Scenario Outline: Add two numbers
	Given the first number is 50
	And the second number is 70
	When the two numbers are added
	Then the result should be 120

Examples: 
	| Column            |
	| Spa ces           |
	| Amp&ersam         |
	| Spaces & Ampersam |

we generate this code (https://github.com/SabotageAndi/NUnitRetry2213/blob/master/NUnitRetry2213/Features/Calculator.feature.cs#L82)

[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Add two numbers")]
[NUnit.Framework.CategoryAttribute("mytag")]
[NUnit.Framework.TestCaseAttribute("Spa ces", null)]
[NUnit.Framework.TestCaseAttribute("Amp&ersam", null)]
[NUnit.Framework.TestCaseAttribute("Spaces & Ampersam", null)]
public virtual void AddTwoNumbers(string column, string[] exampleTags)

I think this correct generated. The first parameter is a string. If we (as SpecFlow) would start escaping this, you will get it also so as a parameter in your bindings.

So I created a GitHub issue for the NUnit Testadapter (nunit/nunit3-vs-adapter#807). Let see what the NUnit people say about this.

[akshaygit09]

@ryandorendorf
Facing similar issue.. any solution yet ?
also why it always passing null with each scenario we are using Examples.