Support for Istio prometheusMerge

[a-thaler]

Description

Very often it is usefull to have the prometheusMerge feature of Istio available, see as motivation mainly this ticket: kyma-project/telemetry-manager#1468. As the enablement of the feature must be treated as a breaking change (see #1184 for motivation), the feature needs to get introduced as a configurable option.
To enable it in mid-term, the option should be disabled in the beginning and generated into the resource spec. At a later time, the default can be switched on, so that existing installations will not be affected by the default switch.

Goal: Introduce an option to enable the prometheusMerge feature

Criterias:

• By default the feature is turned off
• The user has an option to enable the option at runtime, and disable it also again
• The default gets defaulted into the resource itself so that it is persisted per cluster
• When the setting gets switched (both ways), all workloads need to get restarted so that the changes are getting effective (pod spec and envoy port)

API Proposal

apiVersion: operator.kyma-project.io/v1alpha2
kind: Istio
metadata:
    name: default
    namespace: kyma-system
  spec:
     config:
       telemetry:
         metrics:
           prometheusMerge: false

Potential Release Notes:
Collecting custom metrics from a workload which is exposing the metrics endpoint using the same port as the business API requires to scrape the metrics using mTLS. While the Kyma telemetry module supports this scenario, other observability tools are relying on the prometheusMerge feature of Istio which now can be enabled with option XXX.
Having that option enabled, the Istio sidecar running with an application will expose an additional port 15020 exposing the workloads Istio metrics and the custom metrics as annotated by prometheus.io annotations. Istio will automatically rewrite the annotations to point to the new port instead, so that the metric collection will be supported without mTLS.
The feature can be disabled on a workload level using the prometheus.istio.io/merge-metrics: "false" annotation.

Reasons

DoD:

• Provide unit and integration tests.
• Provide documentation.
• Verify if the solution works for both open-source Kyma and SAP BTP, Kyma runtime.
• If you changed the resource limits, explain why it was needed.
• If the default configuration of Istio Operator has been changed, you performed a manual upgrade test to verify that the change can be rolled out correctly.
• Verify that your contributions don't decrease code coverage. If they do, explain why this is the case.
• Add release notes.

Attachments

Relates to #489

[a-thaler]

Relates to #489, so I aligned the API proposals to use the same grouping approach