How do I add Kubernetes metadata to Prometheus metrics scraped by Vector agent ?

[yeshwanth1312]

Hi Team,

I am using Vector to scrape Prometheus metrics from my Kubernetes pods. Each pod exposes its metrics via an HTTP endpoint at /metrics. While Vector can enrich logs with Kubernetes metadata using the kubernetes_logs source, I couldn’t find a similar built-in functionality for attaching Kubernetes metadata (such as pod_name, namespace, node."labels", etc.) to Prometheus metrics.

Since this feature does not exist out of the box, I am looking for possible ways to achieve this. Specifically:

• Is there a way to inject Kubernetes metadata into metrics before they are sent to Prometheus remote write?
• Are there any known workarounds or best practices for this use case?

Any insights or recommendations would be greatly appreciated!

Thanks!

[pront]

Hello @yeshwanth1312,

You probably want to do:

1. Convert logs to metrics. You can use log_to_metric for this.
2. Add tags to them. You can use Remap for this. See Vector Metric Model.

Let me know if I understood your use case correctly.

[yeshwanth1312]

No, I am scraping metrics from Kubernetes pods(http://POD_IP:PORT/metrics). Since pod names,ip's are dynamic, how can I set labels like we do for kubernetes_logs, which attaches Kubernetes metadata? I think we need a source similar to kubernetes_metrics, equivalent to kubernetes_logs, but for metrics

[yeshwanth1312]

@pront Do you see a solution for above ?

[pront]

Hmm, I think this is what you want: #5077