The performance of ServiceClient

[heroes3lover]

Hello experts,
I am trying to find a solution for timeout issues.
After moving web app from .net framework 4.6.2 to .net 6,
Except upgrade a few deprecated packages and replace crmsdk with Power Platform dataverse client, most of code are same.
Now noticed lots of timeout issues with serviceclient, every morning when massive requests come in I saw 2k, 3k 500 errors, and from Azure portal failed request logs are all showing "fail connection to dataverse" some even in 16 ms.
Then I tested in local machine,
I wrote another test console to send same get request to web api which just queries one entity record from D365 and return.
if ServiceClient registered as scope and implemented with clone method it can handle 10 tasks running in parallel.
If ServiceClient registered as singleton it can handle 100 tasks in parallel.
If 1000 tasks run in parallel, all of them failed with timeout error errors, but when I tested with crmsdk package, it can succeed around 270 requests. fail 730 requests.
Is there anyone was facing same issue before?
testing code is simple:

using MultipleApiCall;
using Newtonsoft.Json;
using System.Net.Http.Headers;

var count = 1000;

async Task GetCustomer(string url, HttpClient client)
{
try
{
//GET Method
using (var response = await client.GetAsync(url))
{
return (int)response.StatusCode;
}
}
catch(Exception ex)
{
Console.WriteLine(ex.Message);
return 0;
}
}

using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

var tasks = new List<Task<int>>();
//create the search tasks to be executed
for (int i = 0; i < count; i++)
{
    tasks.Add(GetCustomer("http://localhost:51170/api/Customers?id=61704c13-7b81-ed11-81ad-00224818a814&customerType=contact", client));
}

// Await the completion of all the running tasks. 
var t = Task.WhenAll(tasks); 

try
{
    t.Wait();
    var succeed = 0;
    var failed = 0;
    foreach (var task in tasks)
    {
        var code = await task;
        if ((code == 200) || (code == 204))
            succeed++;
        else
            failed++;

    }
    Console.WriteLine($"Succeed {succeed}");
    Console.WriteLine($"failed {failed}");
}
catch { Console.WriteLine($"application shutdown with exception"); }

}

[heroes3lover]

I compared the performance
crmsdk > serviceclient singleton > scope with serviceclient clone > scope with new serviceclient(connectstring)

[MattB-msft]

Were you running the ServiceClient as .net 4.8 previously? or were you using CrmServiceClient?
also a few things to review:
first,, .net made a number of changes to the network stack in .net 6+ that make it much more 'stingy'.. see : https://devblogs.microsoft.com/dotnet/dotnet-6-networking-improvements/
This seems to be primarily impacting folks with managed networks ( Packet Sniffers / IDS / or TLS offloading logic )

second, if you're working against a single endpoint and are trying to manage horizontal scale out ('n' threads), you need to be sensitive to the host platform which will tell you how many threads it will support.
See : https://learn.microsoft.com/en-us/power-apps/developer/data-platform/api-limits#how-to-maximize-throughput

This is written for the CrmServiceClient and the WebAPI, however the concepts are identical for the ServiceClient (the updated article for ServiceClient is in process right now. )
The major difference between using the ServiceClient and the CrmServiceClient is instead of using .clone you would use the async version of the methods your trying to invoke ( ExecuteAsync, CreateAsync .. and so on ).

[heroes3lover]

Hi Matt,
Thank you for all the information.
We were using CrmServiceClient in .net framework 4.6.2 and registered as scope lifestyle.
Bellow code was the way for initialization.

_logger.LogDebug("Acquired token for {clientId}", authData.ClientId);
CrmServiceClient.AuthOverrideHook = new AuthOverrideHook(authenticationResult.AccessToken);
CrmServiceClient client = new CrmServiceClient(new Uri(authData.Resource), useUniqueInstance: true);

It is a bit hard at this moment to replace all the method to use async version because we got release deadline.
To translate above construction code with ServiceClient, can you please tell me what will be the best way to do ServiceClient initialization? Singleton or Scoped with .clone?

[MattB-msft]

I suspect the issue your seeing is primarily due to the changes in .net 6, though I would need to see the failure messages your seeing to be sure.

What your using there for setup works well for ServiceClient ( though I would make sure your on the latest version of the service client ).
You would use the Auth Pointer as part of the constructor vs the way it was done in CrmServiceClient.

Depending on your hosting Infra, ( functions / API service / WebSite/ etc.. )
I would suggest you keep the way you were doing it for CrmServiceClient,

One thing to make sure of though is that your are managing token cache in your auth behavior so that you do not ask for tokens unless you need to.

[heroes3lover]

Yes, I am using latest version of ServiceClient
PackageReference Include="Microsoft.PowerPlatform.Dataverse.Client" Version="1.0.26"

////program.cs
public class Program
{

    public static IConfiguration Configuration { get; } = new ConfigurationBuilder()
       .SetBasePath(Directory.GetCurrentDirectory())
       .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
       .AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production"}.json", optional: true)
       .AddEnvironmentVariables()
       .Build();

    public static void Main(string[] args)
    {
        var logConfig = LogBuilder.BuildConfiguration(Configuration, "Logging");
        Log.Logger = logConfig.CreateLogger();

        try
        {
            Log.Information("Starting up");
            CreateWebHostBuilder(args).Build().Run();
        }
        catch (Exception ex)
        {
            Log.Fatal(ex, "Application start-up failed");
        }
        finally
        {
            Log.CloseAndFlush();
        }


    }

    public static IHostBuilder CreateWebHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
        .ConfigureWebHostDefaults(webBuilder =>
        {
            webBuilder.ConfigureKestrel((context, options) =>
            {
                options.Limits.MaxRequestBodySize = 52428800;
            })
            .UseStartup<Startup>();
        })
        .UseSerilog();
}

////scoped cs
public class D365OrgService : IOrganizationService, IDisposable
{

    private readonly IOrganizationService _client;

    private readonly IAuthDataProvider _authProvider;
    private readonly ILogger<D365OrgService> _logger;
    private string _clientId;
    public string ClientId => _clientId;

    public D365OrgService(
        IAuthDataProvider authProvider,
        ILogger<D365OrgService> logger)
    {
        _authProvider = authProvider;
        _logger = logger ?? NullLogger<D365OrgService>.Instance;
        _client = CloneServiceClient();
        _clientId = _authProvider.ClientId;
    }

    private IOrganizationService CloneServiceClient()
    {
        Stopwatch st = new Stopwatch();
        st.Restart();
        var ms01 = st.ElapsedMilliseconds;
        var clone = _authProvider.BenchmarkServiceClient.Clone();
        var ms04 = st.ElapsedMilliseconds;
        _logger.LogDebug($"Cloned client - {ms01} - {ms04}");
        st.Stop();
        return clone;
    }

///
....
/////

    public void Dispose()
    {
        (_client as ServiceClient)?.Dispose();
    }
}

[heroes3lover]

///authprovider created benchmarkserviceclient.
private ServiceClient ConstructService()
{
ServicePointManager.SecurityProtocol = ServicePointManager.SecurityProtocol | SecurityProtocolType.Tls12;
var authData = GetAuthData();
_clientId = authData.ClientId;
_logger.LogDebug("GetOrganizationService {clientId}", authData.ClientId);

        var connectionString = $"Url={authData.Resource};AuthType=ClientSecret;ClientId={authData.ClientId};ClientSecret={authData.ClientSecret};RequireNewInstance=true";

        var client = new ServiceClient(connectionString, _logger);

        if (!client.IsReady)
        {
            _logger.LogWarning("Dynamics ServiceClient is NOT ready!! client {ClientId} ", authData.ClientId);
            _logger.LogError(client.LastException, "CRM App client ID {ClientId} LastCrmError: {LastCrmError}",
                authData.ClientId, client.LastException);
            throw new Exception($"OrganisationServiceProvider Could not connect to instance {authData.Resource}");
        }
        return _client = client;
    }

[heroes3lover]

Morning @MattB-msft
I updated the code to use Scoped with .Clone plan, no luck.
but error changed from "failed connection to dataverse" to "An operation was attempted on a nonexistent network connection."
Which is exactly same error when I was using my testing console app with 500 tasks running in parallel.
this only happens when many requests come in short period, if not too many requests, it won't report any error.
From failed request logs in azure portal I can see much less 500 errors after I updated the code with clone method.

[MattB-msft]

How are you hosting it and are you capturing the detailed logs? ( you can pass a ILogger to the constructor in the service client )

[heroes3lover]

After I use clone method.

2023-01-03 19:08:07.856 [ERR] [Australia East] [] [80000045-0400-8700-b63f-84710c7967bb] CustomerController - Get
System.TimeoutException: The request channel timed out attempting to send after 00:04:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.
---> System.TimeoutException: The HTTP request to 'https://Companystaging.crm6.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.2.47.11043' has exceeded the allotted timeout of 00:04:00. The time allotted to this operation may have been a portion of a longer timeout.
at System.ServiceModel.Channels.HttpChannelFactory1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.SendRequestAsync(Message message, TimeoutHelper timeoutHelper) at System.ServiceModel.Channels.RequestChannel.RequestAsync(Message message, TimeSpan timeout) --- End of inner exception stack trace --- at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.Retrieve(String entityName, Guid id, ColumnSet columnSet) at Company.CAL.WebApi.BusinessLogic.CustomerBusinessLogic.RetrieveContact(Guid customerId, CustomerResponseOptions options) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC11-JOB1\Company.CAL.WebApi\BusinessLogic\CustomerBusinessLogic.cs:line 389 at Company.CAL.WebApi.BusinessLogic.CustomerBusinessLogic.RetrieveCustomer(Guid customerId, CustomerResponseOptions options) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC11-JOB1\Company.CAL.WebApi\BusinessLogic\CustomerBusinessLogic.cs:line 68 at Company.CAL.WebApi.BusinessLogic.CustomerBusinessLogic.RetrieveCustomer(CustomerIdentificationRequest request, CustomerResponseOptions options) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC11-JOB1\Company.CAL.WebApi\BusinessLogic\CustomerBusinessLogic.cs:line 106 at Company.CAL.WebApi.Controllers.CustomersController.Get(Nullable1 id, String customerType, String membershipCardNumber, String externalSystemCode, String externalSystemId, Nullable`1 brokerClientProfileId, Boolean returnActiveAlerts, String fields) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC11-JOB1\Company.CAL.WebApi\Controllers\CustomersController.cs:line 91
2023-01-03 19:08:07.856 [INF] [Australia East] [] [80000045-0400-8700-b63f-84710c7967bb] Response Body: {"Message":"The request channel timed out attempting to send after 00:04:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout."}
2023-01-03 19:08:07.857 [ERR] [Australia East] [] [80000045-0400-8700-b63f-84710c7967bb] HTTP GET /api/customers responded 500 in 241429.0067 ms

[heroes3lover]

Before I used clone method, the error log is

2022-12-29 17:00:54.203 [ERR] [Australia East] [] [80000004-0000-5e00-b63f-84710c7967bb] CustomerController - Get
Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException: Failed to connect to Dataverse
---> System.AggregateException: One or more errors occurred. (Request to the endpoint timed out.)
---> MSAL.NetCore.4.48.1.0.MsalServiceException:
ErrorCode: request_timeout
Microsoft.Identity.Client.MsalServiceException: Request to the endpoint timed out.
---> System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 30 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.PowerPlatform.Dataverse.Client.Auth.MSALHttpRetryHandlerHelper.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) --- End of inner exception stack trace --- --- End of inner exception stack trace --- at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Microsoft.Identity.Client.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, CancellationToken cancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, Boolean doNotThrow, Boolean retry, CancellationToken cancellationToken) --- End of inner exception stack trace --- at Microsoft.PowerPlatform.Dataverse.Client.Auth.AuthProcessor.ProcessMsalExecptionAsync(Uri serviceUrl, ClientCredentials clientCredentials, X509Certificate2 userCert, String clientId, Uri redirectUri, PromptBehavior promptBehavior, Boolean isOnPrem, String authority, Object msalAuthClient, DataverseTraceLogger logSink, Boolean useDefaultCreds, MemoryBackedTokenCache memoryBackedTokenCache, String tokenCacheStorePath, MsalException msalEx) at Microsoft.PowerPlatform.Dataverse.Client.Auth.AuthProcessor.ExecuteAuthenticateServiceProcessAsync(Uri serviceUrl, ClientCredentials clientCredentials, X509Certificate2 userCert, String clientId, Uri redirectUri, PromptBehavior promptBehavior, Boolean isOnPrem, String authority, Object msalAuthClient, DataverseTraceLogger logSink, Boolean useDefaultCreds, SecureString clientSecret, Boolean addVersionInfoToUri, IAccount user, MemoryBackedTokenCache memoryBackedTokenCache, String tokenCacheStorePath) at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.ConnectAndInitServiceAsync(OrganizationDetail orgdata, Boolean IsOnPrem, Uri homeRealmUri) at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.DoDirectLoginAsync(Boolean IsOnPrem) at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.InitServiceAsync() Inner Exception: System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 30 seconds elapsing. ---> System.TimeoutException: A task was canceled. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.PowerPlatform.Dataverse.Client.Auth.MSALHttpRetryHandlerHelper.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at Microsoft.Identity.Client.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, CancellationToken cancellationToken) at Microsoft.Identity.Client.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary2 headers, HttpContent body, HttpMethod method, ILoggerAdapter logger, Boolean doNotThrow, Boolean retry, CancellationToken cancellationToken)
StatusCode: 0
ResponseBody:
Headers:
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.GetCachedService(ConnectionService& ConnectionObject) at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.IntilizeService(ConnectionService& ConnectionObject) at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath) --- End of inner exception stack trace --- at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClientAsync externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger, String tokenCacheStorePath) at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.ConnectToService(String connectionString, ILogger logger) at company.D365.Integration.D365OrgService.ConstructService() at company.D365.Integration.D365OrgService.Retrieve(String entityName, Guid id, ColumnSet columnSet) at company.CAL.WebApi.BusinessLogic.CustomerBusinessLogic.RetrieveContact(Guid customerId, CustomerResponseOptions options) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC-JOB1\company.CAL.WebApi\BusinessLogic\CustomerBusinessLogic.cs:line 389 at company.CAL.WebApi.BusinessLogic.CustomerBusinessLogic.RetrieveCustomer(Guid customerId, CustomerResponseOptions options) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC-JOB1\company.CAL.WebApi\BusinessLogic\CustomerBusinessLogic.cs:line 68 at company.CAL.WebApi.BusinessLogic.CustomerBusinessLogic.RetrieveCustomer(CustomerIdentificationRequest request, CustomerResponseOptions options) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC-JOB1\company.CAL.WebApi\BusinessLogic\CustomerBusinessLogic.cs:line 106 at company.CAL.WebApi.Controllers.CustomersController.Get(Nullable1 id, String customerType, String membershipCardNumber, String externalSystemCode, String externalSystemId, Nullable`1 brokerClientProfileId, Boolean returnActiveAlerts, String fields) in D:\bamboo-agent-home\xml-data\build-dir\PJ-ASCFTC-JOB1\company.CAL.WebApi\Controllers\CustomersController.cs:line 91

[MattB-msft]

OK, so you ported to MSAL at the same time as porting to ServiceClient?
if that is the case, the error looks like it's coming from you MSAL token acquire code (in the second error) which implies you're not managing a token cache for your execution. MSAL token cache logic is very different from ADAL and it does not just 'do it for you' as ADAL did for Confidential client type accounts. There is considerable documentation on building token cache providers for MSAL.

Regarding the use of .clone..
the timeout your seeing is indicative of the Dataverse API taking longer to execute then the connection time allows.
that can be manipulated using the MaxTimeOut (cant remember the full name of the property) on the ServiceClient. it is a static property that must be configured before you create the service client.

[heroes3lover]

OK, so you ported to MSAL at the same time as porting to ServiceClient? if that is the case, the error looks like it's coming from you MSAL token acquire code (in the second error) which implies you're not managing a token cache for your execution. MSAL token cache logic is very different from ADAL and it does not just 'do it for you' as ADAL did for Confidential client type accounts. There is considerable documentation on building token cache providers for MSAL.

Yes, As I know latest ServiceClient is using MSAL, we definitely move to ServiceClient and MSAL at same time.
When I didn't use clone method we created ServiceClient with connection string for each request, then I saw MASL error,
I believed the authentication step broke ServiceClient and WebApp, then I checked all discussions in forum and notice clone will duplicate the token (correct me if I am wrong) and yes, clone did solve MASL error.

Regarding the use of .clone.. the timeout your seeing is indicative of the Dataverse API taking longer to execute then the connection time allows. that can be manipulated using the MaxTimeOut (cant remember the full name of the property) on the ServiceClient. it is a static property that must be configured before you create the service client.

This is our current issue, timeout, I can see it times out in 00:04:00, I will set as crmsdk MaxConnectionTimeout and compare these two. Thanks.

[heroes3lover]

Hi,
crmsdk MaxConnectionTime is 2min, ServiceClient default is 4min (comment in code says it is 2min, but not true).
Compared again during daytime with another QA environment, still same outcome.
500 requests in 100 seconds, crmsdk can return 200 hundred succeed responses. but ServiceClient returns none. even I registered ServiceClient as singleton.

[MattB-msft]

The comment / docs for ServiceClient default timeouts are being corrected in docs currently (this was in flight already)

I'm not clear on your response here, you're saying that when you hit your service with 5K requests, the service client version is not doing anything? Are you getting Error's?

[heroes3lover]

Thanks Matt,
Sorry didn't make the comment clear.
I mean use above test app, I set count to be 500 tasks.
Because the default timeout for httpclient is 100seconds,
When 100seconds passed, with crmsdk about half requests got response successfully, and none with ServiceClient.
I will write some sample solution for this testing.
But if you have any suggestion, I am very glad to take it.

[heroes3lover]

Hi Matt,

I have created testing solution
https://github.com/heroes3lover/MultipleApiCall
You can run MultipleApiCall.exe 3 times for crmsdk webapp and serviceclient webapp.
You will see big difference. 500 requests in parallel.
Thanks.

[MattB-msft]

@heroes3lover I apparently missed this post, Can you reissue that invite please?

[heroes3lover]

1.0.39 is much different and I will spend more time to test.
For 500 tasks I ran three times and all succeed.
1.0.26 didn't do that, 1.0.26 got very bad outcome at 2rd and 3rd time.

I sent invitation again for my repo.
Feel free if you see any wrong way of usage,
every suggestion is appreciated.

[MattB-msft]

@heroes3lover
Have a look at this branch : https://github.com/heroes3lover/MultipleApiCall/tree/users-mattb-variant1
I altered this a bit, Perf looked quite good to me, running on a bench test.