Reduce the impact of flaky tests to be able to make them more actionable

[joperezr]

Objective: Improve the infrastructure to reduce the impact of flaky tests and make them more actionable.

Tasks:

• Consider Isolating flaky tests into individual jobs to be able to re-run just those when they fail.
• Implement mechanisms to quickly identify and disable failing tests.
• Ensure that test failures are actionable with proper logs and diagnostics. (e.g. Once a test fails, we have enough information to investigate and fix them)

[sebastienros]

Candidate for quarantining: AzureServiceBusExtensionsTests.VerifyWaitForOnServiceBusEmulatorBlocksDependentResources

Examples:

• https://dev.azure.com/dnceng-public/public/_build/results?buildId=912694
• https://dev.azure.com/dnceng-public/public/_build/results?buildId=912738

[eerhardt]

I've disabled the following in Disable Azure ServiceBus emulator functional tests (dotnet/aspire#7067):

• AzureServiceBusExtensionsTests.VerifyWaitForOnServiceBusEmulatorBlocksDependentResources
• AzureServiceBusExtensionsTests.VerifyAzureServiceBusEmulatorResource
• The ServiceBus portion of the Azure Functions Playground test

[davidfowl]

Why do we have this #7056

[danmoseley]

Why do we have this #7056

The assumption was there was test specific work to fix flaky tests, and infra work to reduce the impact when tests were actually flaky. But I guess everyone's using just this issue

[davidfowl]

I think we're done with this for 9.1.

• We moved to GitHub actions for PRs
• We disabled flaky tests https://github.com/dotnet/aspire/issues?q=is%3Aissue%20state%3Aopen%20label%3Aflaky-test, they should mostly be in the right area labels. We should mark these high priority and assign them like normal bugs. (@joperezr @danmoseley looking for your help here)
• We fixed lots of flaky tests. There was a mixture of product issues and test issues.
• We added LOTS of logs to the CI pipelines and product. We should be capturing everything required to isolate test problems.

We don't have a way to quarantine test run as yet. That would be the last thing I think we could do. Run all tests marked with ActiveIssue in a separate workflow that could be used to investigate issues. I was thinking about a manually triggerable workflow to run a specific test project.

[davidfowl]

Closing this issue out.