CMS 5.2.0-beta1. Regression test: Insufficiently clear test steps

[sabina-talipova]

Description

We need to review our regression tests with our testing team.
Here are some problems that need to be solved:

• Many scenarios do not clearly describe the preliminary steps that must be taken to obtain a clear result, so the the result may vary.
• Many test steps in one script have different meanings in the name of the same object.
• Some steps must be combined into one, since performing them separately will lead to incorrect interpretation of the steps.
• Insufficient coverage of all module functionality.

Note:

This work should be done after we have decided on the list of supported modules.
Mainly concentrate on core modules.

Possible steps to take:

• Create a task for each module to improve test scenarios
• Find out which cases Behat testing covers.
• Prepare preliminary test steps that must be completed before starting the test scenarios to achieve optimal results.
• Rewrite the steps using the Call->Action->Result pattern to make the test cases more consistent.

[GuySartorelli]

Insufficient coverage of all module functionality.

This part at least is intentional. Maybe if the testing team takes over all manual regression testing they may want the steps to be more comprehensive, but there was a very intentional action to remove a bunch of manual test cases and convert them to behat tests instead to reduce the workload required to perform manual regression tests.

[sabina-talipova]

I absolutely agree with this, as it is more profitable. But we need to make sure that all possible cases are truly covered by Behat testing.

[maxime-rainville]

Let's have a catch up with the testing team first. I'll try to arrange a retro and we can go from there.