VMSS hosted database agents on Azure are broken

[milos-teleport]

Expected behavior

If my agent is hosted on a VMSS VMs, using Kubernetes or otherwise, I would expect the agent to operate normally.

Current behavior

Agent encounters the following error

2025-01-27T19:09:21.164Z DEBU             Initializing instance metadata client. cloud/clients.go:1000
2025-01-27T19:09:21.251Z ERRO [DB:SERVIC] Failed to handle connection. error:[
ERROR REPORT:
Original Error: *exported.ResponseError GET https://management.azure.com/subscriptions/obfuscated-subscription-uuid/resourceGroups/example-resource-group/providers/Microsoft.Compute/virtualMachines/6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceNotFound",
    "message": "The Resource 'Microsoft.Compute/virtualMachines/6' under resource group 'example-resource-group' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"
  }
}
--------------------------------------------------------------------------------

Stack Trace:
	github.com/gravitational/teleport/lib/cloud/azure/vm.go:141 github.com/gravitational/teleport/lib/cloud/azure.(*vmClient).Get
	github.com/gravitational/teleport/lib/srv/db/common/auth.go:1094 github.com/gravitational/teleport/lib/srv/db/common.(*dbAuth).getCurrentAzureVM
	github.com/gravitational/teleport/lib/utils/fncache.go:241 github.com/gravitational/teleport/lib/utils.FnCacheGetWithTTL[...].func1
	github.com/gravitational/teleport/lib/utils/fncache.go:313 github.com/gravitational/teleport/lib/utils.(*FnCache).get.func1
	runtime/asm_amd64.s:1700 runtime.goexit
User Message: GET https://management.azure.com/subscriptions/obfuscated-subscription-uuid/resourceGroups/example-resource-group/providers/Microsoft.Compute/virtualMachines/6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceNotFound",
    "message": "The Resource 'Microsoft.Compute/virtualMachines/6' under resource group 'example-resource-group' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"
  }
}
--------------------------------------------------------------------------------
] addr:111.111.111.111:56285 db/server.go:1034

The resource target is wrong:

resourceGroups/example-resource-group/providers/Microsoft.Compute/virtualMachines/6

It should be:

resourceGroups/example-resource-group/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss-id/virtualMachines/6

Impacted versions

Verified on 17.x, but this probably impacts all versions at the time of writing this

Recreation steps

Configure a database agent on a VMSS VMs in Azure

Relevant Teleport config example:

      db_service:
        enabled: true
        resources:
        - labels:
            "*": "*"
        azure:
        - types: ["sqlserver"]
          regions: ["*"]
          subscriptions:
            - example-subscription-uuid
          resource_groups: ["*"]
          tags:
            "*": "*

Workaround

Use a regular Azure VM until a fix is available

[zmb3]

Duplicate of #31758?

[milos-teleport]

Confirmed by @hugoShaka that it's not related - correct me if I'm wrong

[gabrielcorado]

@zmb3 This issue is different. The problem is that we're using the wrong resource ID to fetch the VMSS identities (used to connect to the database). This is due to the metadata field we use for regular VMs returning the wrong resource ID when we request from a VMSS.

The other issue is more related to the fact the VMSS is not returned on the list of VMs API used by the VM discovery flow.

[hugoShaka]

Slack context: https://gravitational.slack.com/archives/C01TYKHFVTQ/p1738106542270569?thread_ts=1738015564.986479&cid=C01TYKHFVTQ