The result of MAL sum aggregation operation is not as expected

[lidingming2005]

Hi,

I want to aggregate some prometheus metrics from a k8s service, and I define otel colletor like this:

global:
  scrape_interval: 30s
scrape_configs:
  - job_name: 'myjob'
    metrics_path: '/metrics'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_name, __meta_kubernetes_pod_container_port_name]
        action: keep
        regex: target;target-port  
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: host_name
        regex: (.+)
        replacement: $$1 

And My MAL Script is:

expSuffix: service(['service'])
metricPrefix: my
metricsRules:
  - name: meter_test
    exp: server_request_count.sum(['service'])

The sample family is like 'server_request_count{service="my_service", status=, host_name=<pod_name>}', and I expect to get the summation of server_request_count by all status and all host_name. But the result is the average of them, which confused me.

For example, I have 2 pods, the sample family like this:
server_request_count{service="my_service", status="200", host_name="pod1"} a1
server_request_count{service="my_service", status="404", host_name="pod1"} a2
server_request_count{service="my_service", status="200", host_name="pod2"} b1
server_request_count{service="my_service", status="404", host_name="pod2"} b2
The result of my_meter_test which my expected is (a1+a2+b1+b2), but actually I get (a1+a2+b1+b2)/2

Is my understanding of MAL's aggregation operation wrong? Or what should I do to make the result as my excepted?

Thank you very much!

[wu-sheng]

You need sum downsampling. You don't define it, the default is avg.

[wankai123]

MAL can aggregate the metrics whose labels are partly different, but they should belong to the same sample family, have the same metric name, and be sent at the same time from the same data source. Skywalking doesn't persist origin metrics, so MAL can't aggregate them from different sources, only do downsampling for these metrics from MAL result.

[lidingming2005]

First of all, if you want a sum, you need to make sure every time, you are reporting increasement, rather than the total number in the current minute.

(a1 + a2 + b1 + b2) * 2, I am not sure how you to get this, but this is not the sum's expected behavior.

Sum downsampling is only doing sum, https://github.com/apache/skywalking/blob/master/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/meter/function/sum/SumFunction.java

Thank you.

(a1 + a2 + b1 + b2) * 2 is the minute unit result and perhaps because of the scrape_interval is 30s, for which every minute has 2 metrics. When query time range expand to hour level, the result is truly as sum downsampling said, the summation of every minute unit's result.

And when I change scrape_interval to 1 min and report increasement by 1 min, the result is truly as expected.

[lidingming2005]

MAL can aggregate the metrics whose labels are partly different, but they should belong to the same sample family, have the same metric name, and be sent at the same time from the same data source. Skywalking doesn't persist origin metrics, so MAL can't aggregate them from different sources, only do downsampling for these metrics from MAL result.

Thank you. Maybe I should try some other methods.

[wu-sheng]

(a1 + a2 + b1 + b2) * 2 is the minute unit result and perhaps because of the scrape_interval is 30s, for which every minute has 2 metrics. When query time range expand to hour level, the result is truly as sum downsampling said, the summation of every minute unit's result.

Idealy, from the source, where the otel collector read the data, the data should be reset to 0 after every fetch, rather than reset to 0 in 1 minute period.

[lidingming2005]

(a1 + a2 + b1 + b2) * 2 is the minute unit result and perhaps because of the scrape_interval is 30s, for which every minute has 2 metrics. When query time range expand to hour level, the result is truly as sum downsampling said, the summation of every minute unit's result.

Idealy, from the source, where the otel collector read the data, the data should be reset to 0 after every fetch, rather than reset to 0 in 1 minute period.

Thank you very much.