Increase global metrics rotation interval to 6 hours

[ebensom]

Description

Currently, istiod enforces a 5 minutes retention time for all stats metrics (i.e. if a static counter does not increase in 5 minutes, it will be dropped):

        - name: METRIC_ROTATION_INTERVAL
          value: 5m0s
        - name: METRIC_GRACEFUL_DELETION_INTERVAL
          value: 5m0s

I would also like to ask you to increase the global rotation interval to 6 hours. This 5 minutes rotation interval is often found to be very aggressive (even drops time series for destinations which are in CrashloopBackoff and waiting for 10 minutes).

Reasons

We identified a negative side-effect of this rotation on our metrics collection and monitoring, specific to VictoriaMetrics streaming aggregation and the increase() MetricsQL function.
With 6 hours rotation, hopefully we would still not hit the OOM and scrapeSizeExeceeded issues we have seen with some clusters, but would make the streaming aggregation more stable.

Contact

@ebensom

ToDos [Developer]

• Set new interval for both istio operator and istio operator light

PRs

• Set global metrics rotation interval to 6h #1278

ACs [PO]

• New interval is set

DoD [Developer & Reviewer]

• Provide documentation.
• Verify if the solution works for both open-source Kyma and SAP BTP, Kyma runtime.
• If the default configuration of Istio Operator has been changed, you performed a manual upgrade test to verify that the change can be rolled out correctly.
• Add release notes.

Attachments

Follow-Up Issues