Broker CPU metric measured per node and not container

[Milana-Gelman-PX]

Hi,
We tested cruise control with our kafka clusters and we see a diffrent cpu value for each kafka broker in our metrics vs. the values in cruise control UI.
We using container_cpu_usage_seconds_total metric of kubelet that measuring CPU per kafka container,
I can see that cruise control using node_cpu_seconds_total metric of node exporter (

cruise-control/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/monitor/sampling/prometheus/DefaultPrometheusQuerySupplier.java

Line 65 in 97b6d79

String.format("1 - avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[%dm]))",

)

The use of node exporter in cruise control logic causes incorrect rebalancing in our clusters (according to cpu goals) since we running on our nodes many different containers (not only Kafka) that change all the time.

I dont see an option to configure the metric to be per container/use jmx cpu metric instead of node metric. there is a solution to this other than separating Kafka into separate nodes ?

and if separating, will be happy to understand what is behind the idea of using metric at node level ?

(trying to understand if its a bug or a feature)

Thanks.