Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the vendored code in pkg/export/otel/metric #1515

Open
grcevski opened this issue Jan 10, 2025 · 0 comments
Open

Remove the vendored code in pkg/export/otel/metric #1515

grcevski opened this issue Jan 10, 2025 · 0 comments

Comments

@grcevski
Copy link
Contributor

Since Beyla can monitor multiple processes, it's very typical that it will monitor for a long time where a lot of services might restart, disappear or new will be launched. Naturally, over a period of time the cardinality will keep on increasing unless stale (non-existent) attributes are removed. The Prometheus Go SDK supports deleting of stale metrics, however the OTel metrics SDK doesn't.

To implement the removal of stale metrics with the OTel SDK a spec change is needed for the metrics API.

To avoid excessive cost increases with continuous monitoring of dynamic K8S clusters we were using a temporary branch of the Go SDK where a removal feature was implemented. This approach worked for the short term, but it proved problematic with SDK version upgrades and bundling Beyla with Grafana Alloy.

In the mid term we've found an approach where we have our own implementation of some parts of the OTel metrics SDK which support removal.

This issue is a reminder that we should remove the code in pkg/export/otel/metric once we have a supported mechanism to delete stale metrics from the Go metrics SDK or we move to the collector metrics SDK and we can implement the functionality that way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant