Support capture and streaming of both regular and generated frames

[SaltyBet]

Details:

• GPU-based frame generation has arrived, in many forms:
  • Up to 1 frame generated between 2 real frames:
    • API-Based:
      • DLLS 3 Frame Generation
      • FSR 3 Frame Generation
    • Driver-Based:
      • AMD Fluid Motion Frames (AFMF)
  • Up to 2 frames generated between 2 real frames:
    • Capture-Based:
      • Lossless Scaling Frame Generation (LSFG):
        • Introducing Frame Generation - LSFG
        • New frame capture API
        • LSFG 1.1 – New UI detection model
        • Introducing LSFG 2.0
        • X3 Frame Generation
• Currently, Sunshine seems to capture normally-rendered frames only.
• This is a request for Sunshine developers to provide Sunshine the capability to capture/stream not only GPU-rendered frames, but also GPU-generated frames, for all currently known methods of frame generation (DLLS 3 FG, FSR 3 FG, AFMF, LSFG).

[gschintgen]

Since no OS mention is made I'll add that in Linux the situation is different: The various capture loops just capture at a fixed frequency. IOW they should be able to capture the generated frames. (In particular if they're captured via kmsgrab straight from the GPU's output framebuffer.)

I've also seen reports that some games have working FSR3 frame generation in Linux. It'd be interesting to have confirmation that this is indeed working when used in conjunction with Sunshine streaming.

[SaltyBet]

@gschintgen, thanks for the input.

I was referring to Windows when I started the discussion, not mentioned explicitly, but you could (somewhat) infer it (neither AFMF nor LSFG exist on Linux, sadly).

If the behaviour on Linux is as you describe it, that is, screen capture on Linux captures the actual displayed frame, no matter the source (rendered/generated), then that's great.

This would be a Windows-only issue in that case.

Maybe one of the Sunshine devs can chime in on this subject.

[gschintgen]

I don't know much about Sunshine on Windows, but this might be of interest to you: LizardByte/Sunshine#2580
Possibly this new capture backend supports this already. (Given the merge date you'll have to try out the latest pre-releases if you want to try it out.)

[ReenigneArcher]

To use that capture method you need to disable the service and run Sunshine manually.

[radugrecu97]

I don't know much about Sunshine on Windows, but this might be of interest to you: LizardByte/Sunshine#2580
Possibly this new capture backend supports this already. (Given the merge date you'll have to try out the latest pre-releases if you want to try it out.)

I tried and it doesn't work with AMD Fluid Motion Frames.

I tested on my monitor, which was awake.

The overlay says that the game display mode is not correct ONLY when streaming. I did get it to "activate" when I disabled Fullscreen optimizations and enabled "override high dpi scaling". But the game was barely responsive in general. I tried with Plate Up.

The overlay is still invisible on the client while present on the host.