Enrich cloud-provider to all data passing a gateway

[a-thaler]

Description

Often it is very useful to drill down data via infrastructure elements like region or zones. You can already do that by figuring out the right nodes and the drill down by node. However, that is more complicated and it will be great to have the cloud provider data available on the data itself as resource attributes.

For that you can try to use the ResourceDetectionProcessor, potentially with a custom implementation also reading data from the shoot-info and the node labels, however, it looks like that this processor typically runs per node.
A different approach might be to re-use the existing k8sattributeprocessor and just read the relevant node labels additionally.

Relevant sources for the data are the shoot-info configmap in the kube-system namespace and labels on the nodes itself.

[a-thaler]

When thinking about additional worker pools being supported soon (kyma-project/kyma#18709), it will be relevant to distinguish the node metrics by either type or pool. Checking the semantic conventions of OTEL reveals that there is the host resource attribute family which should be used for machine type and also architecture: https://opentelemetry.io/docs/specs/semconv/attributes-registry/host/

Checking the gardener implementation, a node current has the well-defined kubernetes labels for machine type and architecture:

• node.kubernetes.io/instance-type: m5.xlarge
• kubernetes.io/arch: amd64

Checking the otel-collector it shows that these kind of attributes are set only be resourcedetenctionprocessors as the metadata is provided in a hyperscaler specific fashion.

-> Adding the two attributes host.type and host.arch seems beneficial and falls into the same category like the cloud provider attribtues. Will be great if that can be tackled in the same story