[BUG] Response x-ms-client-request-id does not match the original expected request id

[cool-mist]

Library name and version

Azure.Storage.Files.DataLake 12.21.0

Describe the bug

Getting this error while trying to pass x-ms-client-request-id header in the request and using the SDK to get properties

var properties = await client.GetPropertiesAsync();

Unhandled exception. Azure.RequestFailedException: Response x-ms-client-request-id '0b425a6f-a5b4-4a26-b564-5254be724d93, 1d47cce8-30aa-4fa2-b785-6e895cf2b32e' does not match the original expected request id, '0b425a6f-a5b4-4a26-b564-5254be724d93'.
   at Azure.Storage.StorageRequestValidationPipelinePolicy.OnReceivedResponse(HttpMessage message)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.InnerProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline)
   at Azure.Core.HttpPipelineExtensions.ProcessMessageAsync(HttpPipeline pipeline, HttpMessage message, RequestContext requestContext, CancellationToken cancellationToken)
   at Azure.Storage.Blobs.BlobRestClient.GetPropertiesAsync(String snapshot, String versionId, Nullable`1 timeout, String leaseId, String encryptionKey, String encryptionKeySha256, String encryptionAlgorithm, String ifTags, RequestConditions requestConditions, RequestContext context)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesInternal(BlobRequestConditions conditions, Boolean async, RequestContext context, String operationName)
   at Azure.Storage.Blobs.Specialized.BlobBaseClient.GetPropertiesAsync(BlobRequestConditions conditions, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakePathClient.GetPropertiesAsync(DataLakeRequestConditions conditions, CancellationToken cancellationToken)
   at Azure.Storage.Files.DataLake.DataLakeFileClient.GetPropertiesAsync(DataLakeRequestConditions conditions, CancellationToken cancellationToken)
   at Program.<Main>$(String[] args) in Q:\Repos\stt\Program.cs:line 16
   at Program.<Main>(String[] args)

Expected behavior

Request should not error out.

Actual behavior

Request fails

Reproduction Steps

using Azure.Core;
using Azure.Core.Pipeline;
using Azure.Identity;
using Azure.Storage.Files.DataLake;

var uri = "https://snkvsa2.dfs.core.windows.net/test/s1";
var clientUri = new Uri(uri);
var tokenCredential = new AzureCliCredential();
var token = await tokenCredential.GetTokenAsync(new TokenRequestContext(new[] { "https://storage.azure.com/.default" }));
var pipeline = new AuthHandler(token.Token);
var client = new DataLakeFileClient(clientUri, new DataLakeClientOptions()
{
    Transport = new HttpClientTransport(new HttpClient(pipeline))
});

var properties = await client.GetPropertiesAsync();
Console.WriteLine(properties.Value.ETag);

public class AuthHandler : DelegatingHandler
{
    private readonly string shortLivedToken;

    public AuthHandler(string token) : base(new HttpClientHandler())
    {
        this.shortLivedToken = token;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var g = Guid.NewGuid().ToString();
        request.Headers.Add("x-ms-client-request-id", g);
        request.Headers.Add("Authorization", $"Bearer {this.shortLivedToken}");
        return await base.SendAsync(request, cancellationToken);
    }
}

Environment

No response

[cool-mist]

I saw a previous issue marked "closed as not planned". Hence opening this one. Let me know if there is an error in usage pattern. This request doesn't fail when sending directly to ADLS service.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sumantmehtams.

[amnguye]

Maybe an easier way to set the x-ms-client-request id is to use the HttpPipeline.

See
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/samples/Diagnostics.md#setting-x-ms-client-request-id-value-sent-with-requests

[cool-mist]

Maybe an easier way to set the x-ms-client-request id is to use the HttpPipeline.

See https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/core/Azure.Core/samples/Diagnostics.md#setting-x-ms-client-request-id-value-sent-with-requests

I don't see why this is easier than simply adding an http request header to the request. Regardless, this seems to set client request id for all the usage inside its scope which could be problematic in some use-cases.

[amnguye]

@cool-mist Can you clarify on the use cases that are problematic?

@jsquire @tg-msft Is there any issues with setting the client-request-id through the ClientOptions as recommended by Azure.Core?

[jsquire]

@cool-mist: The entire approach used here is not recommended, as you're working around the Azure SDK client and pipeline. There's no reason that you'd need/want to use a custom HttpClient nor handler for these scenarios. While I can understand overlooking the client request guidance, bypassing the client's built-in auth mechanism is an odd choice that I'd really like to understand.

Setting the client request identifier

The recommended approach to setting a custom x-ms-client-request-id value is demonstrated by this sample:

var secretClient = new SecretClient(new Uri("http://example.com"), new DefaultAzureCredential());

using (HttpPipeline.CreateClientRequestIdScope("<custom-client-request-id>"))
{
    // The HTTP request resulting from the client call would have x-ms-client-request-id value set to <custom-client-request-id>
    secretClient.GetSecret("<secret-name>");
}

If that does not work for you, due to some unusual scenario, you'd want to create a synchronous policy and inject the header there, as demonstrated in this sample.

Authorization

All Azure SDK clients work with Azure.Identity credentials natively via constructor overloads that accept the TokenCredential abstraction. This is how you should be performing your authorization. In the case of DataLakeFileClient with options, you'd want to use this overload with your credential.

This is demonstrated in [this sample](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Files.DataLake/samples/Sample02_Auth.cs#L181_:

 // Create a token credential that can use our Azure Active
// Directory application to authenticate with Azure Storage
var credential = new DefaultAzureCredential();

// Create a client that can authenticate using our token credential
DataLakeServiceClient service = new DataLakeServiceClient(ActiveDirectoryBlobUri, credential);

// Make a service request to verify we've successfully authenticated
await service.CreateFileSystemAsync("sample-filesystem-aad");
await service.DeleteFileSystemAsync("sample-filesystem-aad");

[cool-mist]

bypassing the client's built-in auth mechanism is an odd choice that I'd really like to understand.

We call onelake with the same sdks and onelake supports another token scheme which is faster to authorize than "Bearer". The token credential always adds "Bearer" scheme. I wanted to not use the bearer scheme, but from your comments, I understand azure sdk is not intended for such things.

If the azure dotnet sdk team treats this behavior as not-buggy, then please feel free to close this issue.

[jsquire]

bypassing the client's built-in auth mechanism is an odd choice that I'd really like to understand.

We call onelake with the same sdks and onelake supports another token scheme which is faster to authorize than "Bearer". The token credential always adds "Bearer" scheme. I wanted to not use the bearer scheme, but from your comments, I understand azure sdk is not intended for such things.

If the azure dotnet sdk team treats this behavior as not-buggy, then please feel free to close this issue.

@cool-mist: The correct approach for doing this would still be to add a static policy that manages the non-standard header rather than working outside of the Azure SDK pipeline.

[amnguye]

The DataLake SDK does not support OneLake. Which would explain the confusion of the intention of different token schemes, and what you're trying to achieve.