[BUG] Can't filter a metric by a custom tag, when running the agent in a sidecar-container approach.

[lpodolak]

Agent Environment
Running
Datadog agent is run as a sidecar container, along with the app container in Google's GKE. The reason why we don't run the agent on node is the lack of access. The sidecar approach is the only option for our team to run Datadog to do the APM of our apps.

Agent 7.56.0 - Commit: f7e1780 - Serialization version: v5.0.124 - Go version: go1.22.5

Describe what happened:
Applying custom tags (either via the UI option) or via the DD_TAGS env var doesn't work unless the DD_HOSTNAME is specified. The reason I guess is that by default, when the agent is running as a sidecar container on every pod, the DD_HOSTNAME resolves automatically to the NODE name. Applying DD_TAGS brings uncontrolled behavior: when filtering by the custom tag in Metrics Explorer, I get confusing results: probably because there are multiple agents registered under the same node name. To overcome that, I discovered that when I specify the DD_HOSTNAME to be the name of the pod (instead of the default: node's name), the metrics start to match up what i expected (It can be done by binding to "metadata.name" ):

And that is the solution, but unfortunately unacceptable for us because of the pricing model. In this way, the amount of hosts that we pay for would raise to the number of actual pods that we're running, which makes the bill skyrocket to unacceptable levels.

To make the problem even more confusing, the custom tags when applied in our case (so agent running as a sidecar, but without overriding the DD_HOSTNAME):

• CAN be filtered by in Trace Explorer
• CAN NOT be filtered by in Metrics Explorer (thus, we can't use them in Monitors, which is our goal)

Describe what you expected:
I'd expect being able to apply custom tags (either via the UI or by DD_TAGS environment variable) without needing to override the DD_HOSTNAME env var, when datadog runs in a sidcar container mode.

Steps to reproduce the issue:

1. Run the datadog in a sidecar container mode, on multiple pods running on the same GKE node.
2. Apply custom tag via DD_VARS with different values depending on which pod you apply it
3. Try to filter data by the custom metric to produce in Metrics Explorer. Results will be inaccurate.

[lpodolak]

could anybody look at the issue please?