Update test descriptions of functionality test suite 4 #6013

fluiddot · 2023-08-01T09:21:40Z

This PR is a follow up on this comment regarding updating the test descriptions.

To test:

Observe that all CI jobs (including the optional e2e tests) pass.

PR submission checklist:

I have considered adding unit tests where possible.
I have considered if this change warrants user-facing release notes more info and have added them to RELEASE-NOTES.txt if necessary.

peril-wordpress-mobile · 2023-08-01T10:14:06Z

Wanna run full suite of Android and iOS UI tests? Click here and 'Approve' CI job!

peril-wordpress-mobile · 2023-08-01T10:14:06Z

Wanna run full suite of Android and iOS UI tests? Click here and 'Approve' CI job!

peril-wordpress-mobile · 2023-08-01T10:14:06Z

Wanna run full suite of Android and iOS UI tests? Click here and 'Approve' CI job!

geriux · 2023-08-01T10:24:03Z

...test-js-gutenberg-editor-test-suite-4-buttons-block-should-render-dark-mode-colors-1-ios.png

I see a red underline below the text, I guess because of the check spelling being enabled on your simulator, I wonder if this would cause issues on CI since I believe the test devices we use don't have autocorrect enabled. Have you encountered any flakiness about it?

Ah, good point. I checked the failures and doesn't seem to be related to visual checks. In any case, I'll review the ones I generated and disable the spelling just in case. Thanks for the feedback 🙇 !

This was solved in 1680eb6.

As I shared in this comment, this error was caused by using a non-English keyboard, so it's unrelated to having the spell-checking option enabled. In fact, CI's simulator has spell-checking enabled when running the tests.

In relation to this, I created the #6038 PR to incorporate a new tool that will be used here to solve the issue. @geriux let me know if you could take a look, thanks 🙇 !

fluiddot · 2023-08-01T17:59:14Z

@dcalhoun @geriux the PR is ready for review, thanks 🙇 !

dcalhoun

🚀

...te-4-buttons-block-should-have-its-selection-caret-with-the-same-color-as-the-font-1-ios.png

On iOS, the context menu displayed when selecting the text wasn't displaying the expected options. To solve this the text is clicked before being selected. Additionally, previous actions for selecting the Button block have been removed as aren't necessary.

fluiddot · 2023-08-02T12:04:50Z

I'm afraid the visual test Buttons block - should have its selection / caret with the same color as the font still fails despite the recent fixes.

I'll keep investigating and use a separate PR for testing with CI, as seems it's likely to have discrepancies in the context menu.

dcalhoun · 2023-08-02T12:28:43Z

I'm afraid the visual test Buttons block - should have its selection / caret with the same color as the font still fails despite the recent fixes.

My guess is that the skinnier, translucent context menu is a result of the reveal animation having not yet finished. Is this issue resolved if a timeout is added to await the animation?

fluiddot · 2023-08-02T13:16:33Z

I'm afraid the visual test Buttons block - should have its selection / caret with the same color as the font still fails despite the recent fixes.

My guess is that the skinnier, translucent context menu is a result of the reveal animation having not yet finished. Is this issue resolved if a timeout is added to await the animation?

Yep, I thought the same, in fact, I tried to solve it by adding an extra wait in a70d9c7. But still, it didn't work. Maybe it can be fixed just by increasing the timeout, I'll take a look 🤔.

geriux · 2023-08-04T08:16:46Z

Yep, I thought the same, in fact, I tried to solve it by adding an extra wait in a70d9c7. But still, it didn't work. Maybe it can be fixed just by increasing the timeout, I'll take a look 🤔.

So sorry you're experiencing this flaky test 😓 what if we crop the screenshot? We just need to check that the caret selection has the right color for this test so maybe if we do something like:

index 56397746f..2cf3eef29 100644
--- a/__device-tests__/gutenberg-editor-sanity-test-4-visual.test.js
+++ b/__device-tests__/gutenberg-editor-sanity-test-4-visual.test.js
@@ -149,8 +149,10 @@ describe( 'Gutenberg Editor - Test Suite 4', () => {
 			);
 
 			// Visual test check
+			const location = await buttonBlock.getLocation();
 			const screenshot = await takeScreenshot( {
 				withoutKeyboard: true,
+				topOffset: location.y,
 			} );
 			expect( screenshot ).toMatchImageSnapshot();

index 111f824b3..9ccb5f804 100644
--- a/__device-tests__/utils.js
+++ b/__device-tests__/utils.js
@@ -11,12 +11,14 @@ const { isAndroid } = e2eUtils;
  * @param {Object}  options                    Options
  * @param {boolean} [options.withoutKeyboard]  Prevents showing the keyboard in the screenshot.
  * @param {number}  [options.heightPercentage] Specify a custom height in percentage.
+ * @param {number}  [options.topOffset]        Specify a custom top offset to crop.
  * @return {Buffer} Sreenshot image.
  */
-export async function takeScreenshot(
-	options = { withoutKeyboard: false, heightPercentage: undefined }
-) {
-	const { withoutKeyboard, heightPercentage } = options;
+export async function takeScreenshot( {
+	withoutKeyboard,
+	heightPercentage,
+	topOffset,
+} = {} ) {
 	const iPadDevice = process.env.IPAD;
 	const orientation = await editorPage.driver.getOrientation();
 	const isPortrait = orientation === 'PORTRAIT';
@@ -27,6 +29,7 @@ export async function takeScreenshot(
 	const portraitWidthSize = iPadDevice ? 768 : 320;
 	const landscapeWidthSize = iPadDevice ? 1024 : 640;
 	const widthSize = isPortrait ? portraitWidthSize : landscapeWidthSize;
+	const pixelRatio = isAndroid() ? 3.5 : 3;
 
 	const base64Image = Buffer.from( screenshot, 'base64' );
 	const image = await jimp.read( base64Image );
@@ -50,6 +53,20 @@ export async function takeScreenshot(
 		}
 	}
 
+	// Custom top offset
+	if ( topOffset ) {
+		const topOffsetWithRatio = isAndroid()
+			? topOffset
+			: topOffset * pixelRatio;
+		const finalTopOffset = topOffsetWithRatio - statusBarHeight;
+		image.crop(
+			0,
+			finalTopOffset,
+			image.getWidth(),
+			image.getHeight() - finalTopOffset
+		);
+	}
+
 	// Custom height in percentage
 	if ( heightPercentage ) {
 		const height = ( heightPercentage * image.getHeight() ) / 100;

It would generate:

iOS	Android

This would solve the issue of not having the same options in the contextual menu or language. If not, we could disable the test and re-think its approach.

fluiddot · 2023-08-04T08:31:38Z

So sorry you're experiencing this flaky test 😓 what if we crop the screenshot? We just need to check that the caret selection has the right color for this test so maybe if we do something like:

That's a good idea @geriux ! I'd like to explore first why we are getting a different screenshot, but I agree that the approach of cropping the screenshot would definitely solve the problem. Besides, it would be helpful in other cases where we only want to visual test a specific area of the editor. This approach probably would help make the visual testing less flaky.

I haven't being able to focus on moving this forward, but I'll try to spare time today to wrap it up 👍.

fluiddot · 2023-08-07T09:36:42Z

That's a good idea @geriux ! I'd like to explore first why we are getting a different screenshot, but I agree that the approach of cropping the screenshot would definitely solve the problem. Besides, it would be helpful in other cases where we only want to visual test a specific area of the editor. This approach probably would help make the visual testing less flaky.

I haven't being able to focus on moving this forward, but I'll try to spare time today to wrap it up 👍.

I've checked the video capture in SauceLabs and noticed that the contextual menu is translucid for some reason, which diverges from the local results. That said, I'll take the approach you shared Gerardo, and crop the screenshot to avoid conflicts with that menu between CI and the local environment.

…suite-4

fluiddot · 2023-08-09T16:04:34Z

Following @geriux idea, I've updated the test that was failing using the new helper takeScreenshotByElement. You can see how the screen looks now in f562601. I hope this workaround finally addresses the failures 🤞 .

fluiddot · 2023-08-09T17:14:43Z

Following @geriux idea, I've updated the test that was failing using the new helper takeScreenshotByElement. You can see how the screen looks now in f562601. I hope this workaround finally addresses the failures 🤞 .

It's still failing due to just one pixel 🤦. I'll crop the screenshot to avoid including the contextual menu.

The screenshot is now taken using the Button block instead of the Buttons block.

fluiddot · 2023-08-10T11:06:07Z

Finally, all E2E tests passed with the latest fix (861fa45) 🎊. I'll proceed now to merge the PR, sorry @geriux and @dcalhoun for the extended delay to finish and merge the PR 🙇 .

fluiddot added the Testing Anything related to automated tests label Aug 1, 2023

fluiddot self-assigned this Aug 1, 2023

Update test descriptions of functionality test suite 4

5f22327

fluiddot force-pushed the update/test-description-functionality-test-suite-4 branch from 65b9b87 to 5f22327 Compare August 1, 2023 10:14

fluiddot marked this pull request as ready for review August 1, 2023 10:14

fluiddot requested a review from dcalhoun August 1, 2023 10:14

fluiddot mentioned this pull request Aug 1, 2023

Expand functionality test suite 4 #5977

Merged

2 tasks

geriux reviewed Aug 1, 2023

View reviewed changes

Remove spell-check visual hints from e2e screenshots

1680eb6

dcalhoun approved these changes Aug 1, 2023

View reviewed changes

fluiddot commented Aug 2, 2023

View reviewed changes

...te-4-buttons-block-should-have-its-selection-caret-with-the-same-color-as-the-font-1-ios.png Show resolved Hide resolved

fluiddot added 3 commits August 2, 2023 10:30

Fix Buttons block e2e test

1989eca

On iOS, the context menu displayed when selecting the text wasn't displaying the expected options. To solve this the text is clicked before being selected. Additionally, previous actions for selecting the Button block have been removed as aren't necessary.

Update Buttons block e2e test screenshot

b255282

Add extra wait in Buttons block e2e test

a70d9c7

fluiddot added 2 commits August 9, 2023 17:54

Merge branch 'trunk' into update/test-description-functionality-test-…

de98802

…suite-4

Update screenshot in Buttons block e2e test

f562601

Update screenshot in Buttons block e2e test

861fa45

The screenshot is now taken using the Button block instead of the Buttons block.

fluiddot mentioned this pull request Aug 10, 2023

Initialize E2E tests with initial HTML #6044

Merged

2 tasks

fluiddot added this to the 1.102.0 (23.1) milestone Aug 10, 2023

fluiddot merged commit 3b6042e into trunk Aug 10, 2023

fluiddot deleted the update/test-description-functionality-test-suite-4 branch August 10, 2023 11:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update test descriptions of functionality test suite 4 #6013

Update test descriptions of functionality test suite 4 #6013

fluiddot commented Aug 1, 2023 •

edited

Loading

peril-wordpress-mobile bot commented Aug 1, 2023 •

edited

Loading

peril-wordpress-mobile bot commented Aug 1, 2023

peril-wordpress-mobile bot commented Aug 1, 2023

geriux Aug 1, 2023

fluiddot Aug 1, 2023

fluiddot Aug 1, 2023

fluiddot Aug 2, 2023

fluiddot Aug 8, 2023

fluiddot commented Aug 1, 2023

dcalhoun left a comment

fluiddot commented Aug 2, 2023 •

edited

Loading

dcalhoun commented Aug 2, 2023

fluiddot commented Aug 2, 2023

geriux commented Aug 4, 2023

fluiddot commented Aug 4, 2023

fluiddot commented Aug 7, 2023

fluiddot commented Aug 9, 2023

fluiddot commented Aug 9, 2023

fluiddot commented Aug 10, 2023

Update test descriptions of functionality test suite 4 #6013

Update test descriptions of functionality test suite 4 #6013

Conversation

fluiddot commented Aug 1, 2023 • edited Loading

peril-wordpress-mobile bot commented Aug 1, 2023 • edited Loading

peril-wordpress-mobile bot commented Aug 1, 2023

peril-wordpress-mobile bot commented Aug 1, 2023

geriux Aug 1, 2023

Choose a reason for hiding this comment

fluiddot Aug 1, 2023

Choose a reason for hiding this comment

fluiddot Aug 1, 2023

Choose a reason for hiding this comment

fluiddot Aug 2, 2023

Choose a reason for hiding this comment

fluiddot Aug 8, 2023

Choose a reason for hiding this comment

fluiddot commented Aug 1, 2023

dcalhoun left a comment

Choose a reason for hiding this comment

fluiddot commented Aug 2, 2023 • edited Loading

dcalhoun commented Aug 2, 2023

fluiddot commented Aug 2, 2023

geriux commented Aug 4, 2023

fluiddot commented Aug 4, 2023

fluiddot commented Aug 7, 2023

fluiddot commented Aug 9, 2023

fluiddot commented Aug 9, 2023

fluiddot commented Aug 10, 2023

fluiddot commented Aug 1, 2023 •

edited

Loading

peril-wordpress-mobile bot commented Aug 1, 2023 •

edited

Loading

fluiddot commented Aug 2, 2023 •

edited

Loading