[Regression] Performance regression since v0.0.33-17061-af052b06

[tincore]

Quick summary

Hi,

Thanks for the awesome work.

I noticed a lot of stuttering on one device. This was not happening before and tried previous versions until I could find the exact release where it started to happen.

v0.0.33-17061-af052b06

Default configuration before and after. From perfect performance to big frame drops (15 fps and below) and stuttering.

Version v0.0.33-17055-60ae4c11 is the last performing properly for me. Other previous versions also work fine. Tested with 'Catherine'

I available to try settings or release that you might suggest.

Details

I noticed a lot of stuttering on one device that was not happening before. Ryzen 7 5800H with 64Gb ram and Nvidia 3070.

Same behavior with OpenGL, Vulkan NVidia 3070 or AMD Renoir

Build with regression

v0.0.33-17061-af052b06

Attach two log files

RPCS3.log.gz
RPCS3.log.gz

Attach capture files for visual issues

No response

System configuration

AMD Ryzen 7 5800H with Radeon Graphics | 16 Threads | 58.71 GiB RAM | TSC: 3.194GHz | AVX+ | FMA3
Operating system: POSIX, Name: Linux, Release: 6.11.0-9-generic, Version: #9-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 14 13:19:59 UTC 2024
·A 0:00:00.165736 {Vulkan Device Enumeration Thread} RSX: Found Vulkan-compatible GPU: 'AMD Radeon Graphics (RADV RENOIR)' running on driver 24.2.3
·A 0:00:00.166359 {Vulkan Device Enumeration Thread} RSX: Found Vulkan-compatible GPU: 'NVIDIA GeForce RTX 3070 Laptop GPU' running on driver 560.35.3.0

Other details

No response

[Roikkudev]

This regression also affects Demon's Souls BLUS30443.

FPS drops to under 3 and audio is breaking, game is unplayable, barely starts.
However, on a few later versions the game is more playable, yet still stutters every 1-3 seconds, rendering it unplayable.

On v0.0.33-17055-60ae4c11 the game still works great.

I am on Linux Mint Cinnamon 22 and running the linux appimage.
AMD Ryzen 5 5600H
16 GB RAM
GeForce RTX 3070

Shall I make a separate issue out of it with logs, or elad335 do you think this one might fix Demon's Souls as well?

[AnubyteCode]

Side note, one of the recent update made my Ridge Racer 7 stop crashing in repeatable menu locations. Maybe 2 weeks ago, sorry I can't add more info.

[tincore]

You might download the two revisions and compare behavior. Your issue could be not related to this. El mié, 11 dic 2024, 1:31, AnubyteCode ***@***.***> escribió:

…

Side note, one of the recent update made my Ridge Racer 7 stop crashing in repeatable menu locations. Maybe 2 weeks ago, sorry I can't add more info. — Reply to this email directly, view it on GitHub <#16382 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4HKFCYNYDUBMU5WYBDSZD2E6BVXAVCNFSM6AAAAABTHECDSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZTGM2DMOJZG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[AnubyteCode]

@tincore I've got some things to take care of thru the weekend but I'll try sometime soon.

I had also used "file > all titles > remove all caches" feature around the same time as the version line specified, perhaps that will help here? Simply using RR7's "remove all caches" had never fixed it before.

[tincore]

I've tried that. It made no difference. Also different hardware does not seem to show the performance regresión. El mié, 11 dic 2024, 22:31, AnubyteCode ***@***.***> escribió:

…

@tincore <https://github.com/tincore> I've got some things to take care of thru the weekend but I'll try sometime soon. I had also used "file > all titles > remove all caches" feature around the same time as the version line specified, perhaps that will help here? Simply using RR7's "remove all caches" had never fixed it before. — Reply to this email directly, view it on GitHub <#16382 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4HKFDS324ZQF6JDO6UTRD2FCVKBAVCNFSM6AAAAABTHECDSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZXGIZTINBQGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Roikkudev]

This regression also affects Lord of the Rings Conquest BLUS30226. What was changed that has such a big impact on several games?

The main symptoms are, FPS is reduced to under 10, choppy unplayable gameplay with crackling/stuttering audio as well.

Everything works in v0.0.33-17055-60ae4c11, so after that all updates are unusable for me on Linux/Vulkan.

If you can fix OP's game issue, perhaps these several other games get some help as well.
I'm not sure if I feel like posting 10 different issues for this same regression, as I keep finding more and more games affected by this update.

These comments allow for attachments, so maybe I will post logs in these comments? I'm pretty sure it's related to the same regression.

[schm1dtmac]

Seems to be linked to PR #16240 based on the given build numbers, bit strange that changes related to getting system time are causing these perf drops. The one thing I noticed you all have in common is that you’re running Linux though, any idea if the perf regression is seen on Windows too or not?

[pchome]

Fix:

#!/bin/sh

# Revert "utils/sysinfo.cpp: Yield CPU time when measuring TSC freq"
git revert edfe940543054ee194db7ebd781e4d9f96fe0ccf
# Revert "Fix get_system_time()"
git revert 97f2b6b701b66b72de82ed7e7ae12c28ed43162b
# Revert "Replace rsx::uclock with get_system_time()"
git revert 92bf6ed0a70da35152033965de524024003d3385
# Revert "Compilation hotfix"
git revert a325eb52bd3861c31ffdf6b950e4186c8b5c08c0
# Revert "utils/sysinfo.cpp: New TSC calibration technique"
git revert 84217917d5cba0c8f789b05eb6cd6be05b6c673e
# Revert "utils: Make get_tsc_freq() inlined and non-blocking"
git revert c70c08bb07f14e848e62d225d4c09ab55ea6067c
# Revert "get_system_time(): Add fast TSC-based path"
git revert 2e8029a45b3deeaf31156b093541e0961520c29d
# Revert "asm: Fix utils::rational_mul optimization"
git revert a9a454faf73172cc5c9dd2ab03a54f4437132a9c
# Revert "Optimize get_system_time using 128 bit math"
git revert 3378b03c1fb848da9aa189ce4fd633e05ea5a52c
# Revert "Implement u64_x_u64_=_u128 optimization"
git revert 8d9911e38335df0eea1b77fceca20caed5ee8e33

# Revert "Savestates: Fix main_ppu_module definition"
git revert 7a4e88c146b8dca7b9c036cee2431874a4305b45
# Revert "shared_ptr.hpp: Rewrite shared_ptr to single_ptr conversion"
git revert cfeb0223400a0d6ec96b0f913762575dd365c8d9
# Revert "serialzation.hpp: Fix add_padding"
git revert 0cc655074d0b5389c7b7cc9723b2b94522e4bcbe
# Revert "Fixup lv2_socket"
git revert a5ba96e991982c30c64eca62b32fe5003d6ab5fc

# Revert "Fix Emulator::Pause() segfault"
git revert 623f5822b3f729be62dd4ff5a592a247f82f418c
# Revert "Add some FXO init checks"
git revert c6dadc537bba8180407db61fc9bced110ba81d55
# Revert "StrFmt.cpp: Make function printing shorter"
git revert 98a0c76a0869526f62fb60578c57a49e499367b0
# Revert "Fixup GDB"
git revert a2d5b5a0e2c4c7e099327d4b2985776c61c5efec
# Revert "Remove lv2_socket_native destructor"
git revert d376ba59949a20c7b5a2fd8d192058fc34688e94
# Revert "Thread.h: Add a few noexcept"
git revert 6a4b9430c0500f8ace419929f4fef842bb9a6415
# Revert "sys_config: Move cleanup to IDM abort"
git revert 3bf735161f0594a78c36918670f10788f987bd7b

# Revert "Fix atomic_ptr value constructing overloads (#16473)"
git revert 15f29eedee456a17357b8f3feb3c74072de81b16
# Revert "util/shared_ptr.hpp: STX pointers library fixes"
git revert 4d0c835df3d2cc784e0e81045631d1d3678608f5
# Revert "Fixup sys_process"
git revert 6eb41385952a7580399751b857e42dacb6e93d50

# Revert "fix some warning"
git revert 33c3e3fb0fbd9d1ae54bf45d10f3081fc2c05212
# Revert "Fix fs::file log formatting"
git revert 53817dcc902053e17caf71888fdd1d5ec01bf08d
# Revert "IDM: Implement lock-free smart pointers (#16403)"
git revert 575a245f8d4bcd42039627533d1748c3af0f36a7

To apply everything at once (non-interactive) you can do something like this:
$ for i in `grep -o -E '[a-f0-9]{16,}' revert-tsc-and-pointers.sh`; do git revert --no-commit $i || break; done
$ git revert --continue

Basically reverting elad335`s new TSC stuff (fixes cracking audio and performance) and new smart pointers stuff (fixes broken savestates). Everything else just a glue for this reverts.
Surely reverted some real fixes.

[pchome]

@Megamouse , @kd-11
Maybe not everyone are able to build from sources, so can those PR's be reverted?
Then reapply real fixes, and carefully review and test TSC and Shared Pointers improvements before merging back.
You better understand this code, I can only safely reapply some formatting changes (everything else carefully hidden in @elad335's commits like eastern eggs).

[kd-11]

If #16499 makes the problem go away we need to collect the affected CPUs here. They may have a buggy TSC, which means a blacklist is required to fall back to QPC.

I see Ryzen 5800H and 5600H, so maybe Ryzen 5000H series?

[pchome]

Yup, last commit sims fixed performance regression.
But segfaults on savestate

RPCS3: Emu State Load Thread: '': SYS: 
Segfault writing location 00000000000031c3 at 0000000000a6ceb5.
Thread: Main Thread.

run.sh: line 5: 25856 Segmentation fault      build/bin/rpcs3

Ryzen cpu zen+ Lenovo thinkpad
AMD Ryzen 7 3700U

$ dmesg | grep -i tsc

[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2295.780 MHz processor
[    0.184797] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x2117a2b78e4, max_idle_ns: 440795263183 ns
[    0.298804] TSC synchronization [CPU#0 -> CPU#2]:
[    0.298804] Measured 7309500619 cycles TSC warp between CPUs, turning off TSC clock.
[    0.298820] tsc: Marking TSC unstable due to check_tsc_sync_source failed
[    6.202692] vboxdrv: TSC mode is Invariant, tentative frequency 2295681955 Hz
[    6.380617] kvm_amd: TSC scaling supported

some warnings about tsc in dmesg,
but I read in some linux kernel thread something like "Lenovo is known to fuck with tsc, so there is nothing we can do"

[    0.297831] smpboot: x86: Booting SMP configuration:
[    0.297834] .... node  #0, CPUs:      #2 #4 #6
[    0.298804] TSC synchronization [CPU#0 -> CPU#2]:
[    0.298804] Measured 7309500619 cycles TSC warp between CPUs, turning off TSC clock.
[    0.298820] tsc: Marking TSC unstable due to check_tsc_sync_source failed
[    0.299892]  #1 #3 #5 #7
[    0.300858] smp: Brought up 1 node, 8 CPUs
[    0.300858] smpboot: Total of 8 processors activated (36732.48 BogoMIPS)
...
[    0.674583] Unstable clock detected, switching default tracing clock to "global"
               If you want to keep using the local clock, then add:
                 "trace_clock=local"
               on the kernel command line
[    0.674794] clk: Disabling unused clocks

On my other AMD cpu I can do something like
hpet=disable tsc=reliable clocksource=tsc nmi_watchdog=0 nowatchdog
in kernel params, but still not sure about laptop.

EDIT: Thanks, you were lightning fast to figure out the issue.

[kd-11]

Ok, I think I see what's going on here. The kernel is flagging the tsc as unreliable. I have to wonder if reading tsc in such a system triggers a kernel fallback, probably faulting and the kernel fills in the registers from some other source. We shouldn't use it in those cases for high performance timing I guess.
Does this issue occur on other platforms or only linux? RSX timestamps are currently fetched from TSC. If a slow fallback is in use, there may be performance lost there as well.

[pchome]

I guess from this issue you can get feedback only from linux users, but I saw other issues in the "Most recent" filer list which refer to tsc changes.
Some of them even in "Arm64 megaissue".
This should fix many issues, just need to ask an connect all of them.

[pchome]

For example:
#16465 (comment)

Users only refer to build numbers.
so 17229 vs 17230
clicking the link in above commen giving me rpcs3-v0.0.34-17230-575a245f
575a245 is https://github.com/RPCS3/rpcs3/commit/575a245f8d4bcd42039627533d1748c3af0f36a7

[Trevonn]

If #16499 makes the problem go away we need to collect the affected CPUs here. They may have a buggy TSC, which means a blacklist is required to fall back to QPC.

I see Ryzen 5800H and 5600H, so maybe Ryzen 5000H series?

The CPUs themselves are okay it's the firmware/BIOS that's at fault, particularly on Lenovo Laptops. you can work around the issue by patching your kernel. That's what I do on my Legion 5. Otherwise it will always default to HPET.

I posted about this a few years ago on reddit
https://www.reddit.com/r/linux_gaming/comments/rsvjqb/psa_if_your_clocksource_is_hpet_rather_than_tsc/

Edit: In my experience testing PC games Windows was unaffected by the firmware bugs