Support for Istio prometheusMerge

[a-thaler]

Description
Dynatrace metric integration is cumbersome with metrics as there is not the best fitting solution available. Either the user disabled istio mTLS for the metric ports and uses dynatrace annotations, or he uses the telemetry module to push data via OTLP to Dynatrace, but hear so many conversion is needed that it is troublefull as well, also see https://kyma-project.io/#/telemetry-manager/user/integration/dynatrace/README?id=ingest-metrics

It turned out that there is another alternative to the direct Dynatrace integration scenario which does not require an explicit disabling of the TLS mode for the whole port. By enabling the prometheusMerge feature of Istio, the envoys will expose a new port which serves the envoy metrics plus the business metrics (configured via prometheus pod annotation). With that, Dynatrace can scrape all the metrics directly without reducing the TLS setting on the original port. However, the metric data will be transported plain text.

While that solution sounds better then the current integration scenario, it will affect the current telemetry functionality heavily, as suddenly all pods will have prometheus annotations and will be detected by the telemetry prometheus input. And all this pods will mainly return envoy metrics.

Goal

• Understand all the implications of the feature and propose a concept on if the istio feature should be enabled always/optional and if something should be adjusted in the telemetry functionality.
• Perform necessary changes on telemetry side to not conflict with the istio feature
• Enable merge flag in istio

Criterias

• A new document is created in the internal docs folder which:
  • gives a clear proposal on if the istio feature should be enabled by default/optional
  • what kind of adjustments are needed on the telemetry functionality
• Remove the conflicting scrape job for annotated pods running with istio
• Verify documentation for correctness

[a-thaler]

After the concept is sorted out, we will re-use the ticket to realize the feature

[weinanwangsap]

Hi @a-thaler, got two questions about this issue.

1. Can you just enable envoy mTLS for OTLP(maybe by installing OTLP to an isolated namespace)? So that OTLP can collect metrics from all envoy mTLS enabled pod?
2. Can we enable prometheusMerge for kyma istio ourselves? Is there something risk or limitation?

[a-thaler]

Hi @weinanwangsap,

at the moment Istio/Envoy does not support metric exposure via OTLP, you need to pull the metrics. Hereby, the kyma metric agent does support mTLS already, however, Istio/Envoy seem to not support exposing envoy metrics behind a mTLS port. So I don't see a possibility. Let me know if you see a solution to it.

Currently, you cannot enable the prometheusMerge by yourself. We will do a PR to the Istio module to enable it by default.

[a-thaler]

Re-opening to include also the Istio changes in this story

[a-thaler]

Unfortunetaly, the istio change must be treated as a breaking change and cannot be rolled out without configuration option. So the config change got reverted and will be introduced properly with a flag which in the beginning will be disabled.
A dedicated ticket got created at kyma-project/istio#1185