Use SPDK dynamic scheduler

[judahrand]

Is your feature request related to a problem? Please describe.
When running Mayastor on clusters which have variable workloads CPU usage and power consumption increases significantly.

Describe the solution you'd like
SPDK has a dynamic scheduler which, I believe, should reduce the power consumption without significantly impacting performance. Crucially, when threads are idle they are switched into interrupt mode and stop polling.

Additional context
I've tried deploying Mayastor at home but the increased power consumption was enough for me to abandon it. I don't have much understanding of SPDK but thought I'd put this up as from my reading it seems like this might help. I'd welcome being further educated though!

[judahrand]

It would also be fantastic if Mayastor could run in interrupt mode for cases where low latency is not important! This was added for NVMEoF TCP in the newest version of SPDK, I believe.

https://spdk.io/release/2024/09/27/v24.09_release/

[tiagolobocastro]

Indeed seems SPDK now has interrupt mode for nvme-tcp. We have to evaluate this mode itself and then figure out how we can integrate with it.
We have our own reactors which would need adapting to support this.
I think this would be a very good enhancement for Mayastor.

[skwair]

Just adding some measurements I did to this issue for reference: I'm running a Kubernetes cluster with Talos 1.9.0 on 3 NUCs (Intel Ultra 5 125H, 48GB RAM), idling at around 11-13W (measured at the plug) each.

After deploying Mayastor, each machine increases its power consumption by ~17W, totaling to ~30W per unit, which is pretty significant.

Looking forward to see how switching to the interrupt mode could improve this!