Add per-cpu fanout mode

[pikrzysztof]

see #33

By assigning packets on per-cpu basis we ensure kernel's skb structure (incl locks) gets shared across fewer cores than other fanout strategies. This should result in higher throughput/lower resource usage.

As per man 8 ethtool, all the valid -N -U --config-nfc --config-ntuple options make sure flows get always scheduled to the same core so by using PACKET_FANOUT_CPU we ensure that packets from the same flow get assigned to the same socket.

Let's give it a try!

[pikrzysztof]

This helped quite a lot, see https://github.com/kentik/operations/issues/8982#issuecomment-2020149647 - can we please merge this?

[ghost]

Thanks @pikrzysztof!

[pikrzysztof]

For the interested

https://grafana.yyz1.kentik.com/explore?orgId=1&left=%7B%22datasource%22:%22PABDA7AB1AD2A1489%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22prometheus%22,%22uid%22:%22PABDA7AB1AD2A1489%22%7D,%22editorMode%22:%22code%22,%22expr%22:%221e10%20%2A%20%28%28%5Cn%20%20%20%20sum%20by%28host%29%20%28irate%28prometheus_node_cpu_seconds_total%7Bhost%3D~%5C%22c11%5B8%5D.%2A%5C%22,mode%3D~%5C%22.%2Airq%5C%22%7D%5B$__rate_interval%5D%29%29%20%5Cn%20%20%20%20%20%2F%20on%28host%29%20group_left%20sum%20by%20%28host%29%28%28irate%28prometheus_node_cpu_seconds_total%7Bhost%3D~%5C%22c11%5B8%5D.%2A%5C%22%7D%5B$__rate_interval%5D%29%29%29%5Cn%20%20%20%20%20%5Cn%29%5Cn%20%20%20%20%20%5Cn%20%20%20%20%20%5Cn%20%20%20%20%20%20%2F%20on%20%28host%29%20%28irate%28prometheus_node_network_receive_bytes_total%7Bhost%3D~%5C%22c11%5B8%5D.%2A%5C%22,%20device%3D%5C%22bond1%5C%22%7D%5B$__rate_interval%5D%29%2A8%29%29%22,%22legendFormat%22:%22__auto%22,%22range%22:true,%22instant%22:true%7D%5D,%22range%22:%7B%22from%22:%221711035108198%22,%22to%22:%221711616223256%22%7D%7D

shows cputime spent in interrupts divided by rx data. Flattening in the last two days is the effect of per-cpu fanout mode. We're observing:

• much flatter curve which means the socket scales better when the rx rate increases
• less cost per unit of rx data overall so it's cheaper to run