Screencopy.grab_screenshot uses logical size for the resulting image

[Saviq]

grim does things right:

$ WAYLAND_DISPLAY=wayland-1 miral-shell --display-scale=2 --add-wayland-extension all --x11-output 800x600 &
$ WAYLAND_DISPLAY=wayland-1 grim foo.png
$ file foo.png
foo.png: PNG image data, 800 x 600, 8-bit/color RGBA, non-interlaced

But Screencopy.grab_screenshot() does not scale:

In [1]: from mir_ci.tests.robot.platforms.wayland.Screencopy import Screencopy
2024-08-06 11:01:59,666 - RPA.core.certificates - INFO - Truststore not in use, HTTPS traffic validated against `certifi` package. (requires Python 3.10.12 and 'pip' 23.2.1 at minimum)

In [2]: sc = Screencopy()

In [3]: sc._screenshots_dir="/tmp"

In [4]: await sc.grab_screenshot()
Out[4]: (1, <PIL.Image.Image image mode=RGBA size=400x300>)

[Saviq]

Apparently this doesn't affect the Nvidia platform, so the different eglstream screenshotting code path yields the correct result.

Except for the fact that grim behaves right on Intel, so it might be a coincidence that Nvidia yields correctly sized screenshots here.

[mattkae]

What does grim do?

• It initializes an array of outputs using the wl_output interface
• It then listens on wl_output for the scale & geometry (in this case, the geometry is 800x600 and the scale is 2)
• We also listen on the xdg_output_v1 interface for the logical size (in this case 400x300)
• From this, we calculate the logical_scale which is just a fraction of geometry.width / logical_geometry.width (in this case 2.0)

So then how do we determine the size of the final image?

• If -g is passed, then we just use that
• Else if -o is passed, then we determine it from the selected output. In this case, we copy logical_geometry as the outputted size
• Otherwise, we find the total width and height of all of the outputs based on their logical size (in this case that would be 400x300)
• For scale, we always select the largest possible logical_scale (in this case that would be 2)

When grim renders, it simply multiplies the geometry size by the scale, hence we get an image of 800x600.

[Saviq]

How does it then end up with the correct resolution on Nvidia, when we seem to be giving screenshots of output, not logical, geometry? 🤷

[mattkae]

What does mir-ci do

• ScreencopyTracker used WlrScreencopyManagerV1 to take the screenshot
• This protocol returns a buffer with buffer_width and buffer_height that we use to set the size of the final image
• This buffer size is the width/height of the output in its current mode divided by the scale (in this case, it will be 400x300)

[mattkae]

Just brain dumping FYI. Time to track down where Nvidia goes different

[mattkae]

There is no reason why the size should be any different on Nvidia as far as I can tell. The code path gets the resolution divided by the scale in both instances, and that result is exactly the same regardless of the platform. Maybe I'm missing something but there aren't two different code paths there. Of course configuration is two code paths, but the result is still the same if they are configured to be the same value

[mattkae]

At the end of the day, grim is simply scaling up the resulting image. In this case, we give them a 400x300 buffer and they scale it up to 800x600. They use PIXMAN_FILTER_BILINEAR for this.

Do we want to do a similar thing in mir-ci @Saviq? Technically our result isn't wrong, since that's the actual size of the buffer 🤷

[Saviq]

This is just… why would you want a scaled down shot and then scale it up… I've not seen anything in the screencopy proto that says what scale should the image be… but why would you want it scaled down…

[Saviq]

Maybe grim is compensating for some compositors giving logical size, others real? Because the protocol does not talk about that?

What if we give it real size shots? Then, there's other users of that protocol…

And then then… how does @tarek-y-ismail get real size screenshots on his HW‽

[mattkae]

wlr screencopy wants to send down an unscaled buffer (so it seems - this is not explained outright). Therefore, it is up to the client to present that buffer in some way. That way technically doesn't have to be defined by the protocol since the interpretation of that buffer happens outside of the protocol.

With that being said, I think that the correct course of action on our end is to accept the fact that we have non-real sized shots, and then scale things up on the client side if we want to display them scaled.

[Saviq]

Unscaled is physical resolution in my dictionary…

You mean screen copy wants logical geometry?…

[mattkae]

Okay I was very confused. I still don't know why it's scaled in CI, because the size that's being reported is indeed the physical size (gotten from the current mode). I was mistaken in that regard

[Saviq]

Hum. This gives you native size on both Nvidia and Intel?

In [1]: from mir_ci.tests.robot.platforms.wayland.Screencopy import Screencopy
2024-08-06 11:01:59,666 - RPA.core.certificates - INFO - Truststore not in use, HTTPS traffic validated against `certifi` package. (requires Python 3.10.12 and 'pip' 23.2.1 at minimum)

In [2]: sc = Screencopy()

In [3]: sc._screenshots_dir="/tmp"

In [4]: await sc.grab_screenshot()
Out[4]: (1, <PIL.Image.Image image mode=RGBA size=400x300>)

[mattkae]

@tarek-y-ismail Just found out that in CI we really do get the logical size sent down as the image size instead of the physical size. I still get the physical size 🤷 I am going to do some more investigations right now, but it's looking strange

[mattkae]

I have reproed on the Github Runner! Time to find out why :)

[mattkae]

Wait a gosh darn minute... I rebuilt it locally, and it works properly in both grim and via python. Have we just been using the wrong demo server the whole time 🤔

[Saviq]

Have we just been using the wrong demo server the whole time 🤔

Well, not wrong, but stable… and canonical/mir#3474 is what fixed this, not on stable yet.

#145 adds testing on the latest version of Mir.