Potential Issue with PHP-FPM 8.3.16: "WARNING: scoreboard: writer detected a potential crashed reader, decrementing reader count."

[havranro]

Description

Description:
After upgrading to PHP-FPM 8.3.16, we have encountered an issue that appears to occur only on certain services. Despite our efforts to adjust configurations and increase resources, the problem persists. We have not been able to identify any specific pattern or correlation during the PHP process that could be causing this issue. It seems to happen randomly. Although the number of errors is not large, they do appear consistently.

Error Message:

WARNING: scoreboard: writer detected a potential crashed reader, decrementing reader count.

Environment:

PHP Version: 8.3.16
Operating System (Docker image): php:8.3.16-fpm-alpine
Server Type:
Configuration:

pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35

memory_limit=512M
max_execution_time=90
post_max_size=20M
upload_max_filesize=20M
max_file_uploads=20

Additional Information:
We have tried the following without success:

• Adjusting various configuration settings.
• Increasing system resources.

Any insights or suggestions on how to resolve this issue would be greatly appreciated. Thank you!

PHP Version

PHP-FPM 8.3.16

Operating System

php:8.3.16-fpm-alpine

[iluuu1994]

/cc @bukka

[bukka]

This should not be a huge issue because some of the writers was killed when holding the lock or maybe contention is just too big. If it's because of too big contention (too many workers try to write to scoreboard and some of them might be stalling), then it might potentially result in race when writing to scoreboard (it means potentially messed up value in status page). We could maybe consider making the wait time before releasing the lock configurable and also add that trace log level which should give more info about the problem (currently it's just compile option which is more for dev debugging). I will change it to feature request as that's the only way how we could address it better.

Please note that previously this might led to locking the child (if the lock holder was killed) so this was a bug fix to just add warning but it couldn't come with extra config option because that's not allowed for bug fix. I thought about it already though.