You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since Beyla can monitor multiple processes, it's very typical that it will monitor for a long time where a lot of services might restart, disappear or new will be launched. Naturally, over a period of time the cardinality will keep on increasing unless stale (non-existent) attributes are removed. The Prometheus Go SDK supports deleting of stale metrics, however the OTel metrics SDK doesn't.
To implement the removal of stale metrics with the OTel SDK a spec change is needed for the metrics API.
To avoid excessive cost increases with continuous monitoring of dynamic K8S clusters we were using a temporary branch of the Go SDK where a removal feature was implemented. This approach worked for the short term, but it proved problematic with SDK version upgrades and bundling Beyla with Grafana Alloy.
In the mid term we've found an approach where we have our own implementation of some parts of the OTel metrics SDK which support removal.
This issue is a reminder that we should remove the code in pkg/export/otel/metric once we have a supported mechanism to delete stale metrics from the Go metrics SDK or we move to the collector metrics SDK and we can implement the functionality that way.
The text was updated successfully, but these errors were encountered:
Since Beyla can monitor multiple processes, it's very typical that it will monitor for a long time where a lot of services might restart, disappear or new will be launched. Naturally, over a period of time the cardinality will keep on increasing unless stale (non-existent) attributes are removed. The Prometheus Go SDK supports deleting of stale metrics, however the OTel metrics SDK doesn't.
To implement the removal of stale metrics with the OTel SDK a spec change is needed for the metrics API.
To avoid excessive cost increases with continuous monitoring of dynamic K8S clusters we were using a temporary branch of the Go SDK where a removal feature was implemented. This approach worked for the short term, but it proved problematic with SDK version upgrades and bundling Beyla with Grafana Alloy.
In the mid term we've found an approach where we have our own implementation of some parts of the OTel metrics SDK which support removal.
This issue is a reminder that we should remove the code in pkg/export/otel/metric once we have a supported mechanism to delete stale metrics from the Go metrics SDK or we move to the collector metrics SDK and we can implement the functionality that way.
The text was updated successfully, but these errors were encountered: