ExecuteAsync with ResilienceContext and CancellationToken

[AndrejNazarov]

Hello,

I wasn't able to find an ExecuteAsync method on ResiliencePipeline that allows to pass a ResilienceContext and a CancellationToken together.

This can be done with AsyncPolicy.

My use-case:

• retry an async operation that should be cancellable during retries, thus passing a CancellationToken from the caller
• working with Context (newly ResilienceContext) to pass specific ILogger into the retry code and some other properties
• my pipeline definition (i.e. OnRetry method) is in a different class than the decorated method, I want to inject it via DI (either via ResiliencePipelineRegistry<string> or ResiliencePipelineProvider<string>)

I'm unable to do this in Polly v8 - the ExecuteAsync methods on ResiliencePipeline class contain either a CancellationToken or a ResilienceContext, but never both.

Here is some sample code

public class MyService(ILogger logger, 
    ResiliencePipelineProvider<string> pipelineProvider,
    AsyncPolicy asyncPolicy,
    ISomethingAsync foo) : BackgroundService 
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
        var pipeline = pipelineProvider.GetPipeline("MyPipeline");
        while (!stoppingToken.IsCancellationRequested)
        {
            //new way
            var resilienceContext = ResilienceContextPool.Shared.Get(stoppingToken);
            resilienceContext.Properties.Set(new ResiliencePropertyKey<ILogger>("Logger"), logger);
            try
            {
                //something like this, but this method signature doesn't exist
                var resultNew = await pipeline.ExecuteAsync(async (ctx, token) =>
                {
                    ctx.Properties.Set(new ResiliencePropertyKey<string>("SomeKey"), "SomeValue");
                    var bar = await foo.GetAsync(token);
                    return bar;
                }, resilienceContext, stoppingToken);
            }
            finally
            {
                ResilienceContextPool.Shared.Return(resilienceContext);
            }
            
            //pre version 8 - works
            var context = new Context();
            context.Add("Logger", logger);
            var resultOld = await asyncPolicy.ExecuteAsync(async (ctx, token) =>
            {
                ctx["SomeKey"] = "SomeValue";
                var bar = await foo.GetAsync(token);
                return bar;
            }, context, stoppingToken);
        }
    }
}

Hope this makes sense.

[martincostello]

The reason there aren't any overloads where both are available is because the cancellation token is a member of the resilience context:

Polly/src/Polly.Core/ResilienceContext.cs

Line 34 in 0856394

public CancellationToken CancellationToken { get; internal set; }

[AndrejNazarov]

I see. Can you then suggest how to do this with ResilienceContext? If I do it this way, the token inside the lambda is different:

while (!stoppingToken.IsCancellationRequested)
{
    Console.WriteLine($"stoppingToken wait handle:{stoppingToken.WaitHandle.SafeWaitHandle.GetHashCode()}");
    
    //is this how I pass the cancellation token to the lambda?
    var resilienceContext = ResilienceContextPool.Shared.Get(stoppingToken);
    resilienceContext.Properties.Set(new ResiliencePropertyKey<ILogger<MyService>>("Logger"), logger);
    try
    {
        var resultNew = await pipeline.ExecuteAsync(async (ctx) =>
        {
            ctx.Properties.Set(new ResiliencePropertyKey<string>("SomeKey"), "SomeValue");
            var bar = await foo.GetAsync(ctx.CancellationToken);
            //this returns a different HashCode than stoppingToken above, so it's a different CancellationToken
            Console.WriteLine($"Lambda token wait handle:{ctx.CancellationToken.WaitHandle.SafeWaitHandle.GetHashCode()}"); 
            return bar;
        }, resilienceContext);
    }
    finally
    {
        ResilienceContextPool.Shared.Return(resilienceContext);
    }
}

If I do it the old way, the token inside the lambda is the same (as you would expect).

    var polly = Policy.Handle<Exception>().RetryAsync(3, (exception, retryCount, context) => Console.WriteLine($"try: {retryCount}, Exception: {exception.Message}"));
    var context = new Context();
    var resultOld = await polly.ExecuteAsync(async (ctx, token) =>{
       ctx["SomeKey"] = "SomeValue";
       var bar = await foo.GetAsync(token);
       //This returns same hash code as stoppingToken, so it is the same token
       Console.WriteLine($"Lambda token wait handle :{token.WaitHandle.SafeWaitHandle.GetHashCode()}");
       return bar;
}, context, stoppingToken);

What am I doing wrong?

[martincostello]

The token you get on the context will be different because it's internally wired up to honour timeouts related to a timeout policy/strategy etc. That's an implementation detail you shouldn't worry about - you should just use the token you are given to honour cancellations. If you want to use a specific token, then you should pass it into the context with a named value and use it directly like you are with other things.

[AndrejNazarov]

Okay. Let's say I have an async method that takes 30s to complete.

• I want it to be cancellable from outside
• I want it to run inside a Polly ExecuteAsync wrapper for retries and all the other Polly goodness
• I want to use ResilienceContext data for e.g. specific loggers, etc

What's the best way to achieve this?

This seems to pass/reference the same token:

var resilienceContext = ResilienceContextPool.Shared.Get(outsideToken); // is it necessary to pass the outsideToken here?
ResiliencePropertyKey<CancellationToken> TokenPropertyKey = new("TokenKey");
ResiliencePropertyKey<string> StringPropertyKey = new("SomeKey");
resilienceContext.Properties.Set(TokenPropertyKey, outsideToken);
try
{
    var resultNew = await pipeline.ExecuteAsync(async (ctx) =>
    {
        ctx.Properties.Set(StringPropertyKey, "SomeValue");
        var token = ctx.Properties.GetValue(TokenPropertyKey, default);
        var bar = await foo.GetAsync(token);
        Debug.Assert(outsideToken.WaitHandle.SafeWaitHandle.GetHashCode() == token.WaitHandle.SafeWaitHandle.GetHashCode(), "Tokens are different");
        return bar;
    }, resilienceContext);
}
finally
{
    ResilienceContextPool.Shared.Return(resilienceContext);
}

But this looks rather complicated compared to what was before. Is there no other/simpler way?

[martincostello]

Polly v7 had far too many method overloads which made future iteration and maintenance difficult, so we rationalised down the number of overloads for v8. We also made other changes to improve performance, such as pooling contexts.

If you want to use both features (context and cancellation) then the shared snippet is the way to achieve this. You could simplify the code to some degree by adding helper methods specific to your context to do things like get/set your values on your ResilienceContext via methods that abstract away the resilience keys etc: example.

The inner token in the lambda will/should be different if you're using strategies like timeout and hedging, so I would avoid writing your code to assume that the token is the same as your outside token, then you don't need to worry about adding and getting it to the context.

[AndrejNazarov]

The inner token in the lambda will/should be different if you're using strategies like timeout and hedging,

Thank you for this (and other) clarifications. I wasn't aware of that. I ran it through the debugger and indeed these 2 tokens are different in v8:

var res = await pipeline.ExecuteAsync(async (innerToken) =>
{
    var bar = await foo.GetAsync(innerToken);
    return bar;
}, outsideToken);

but are the same in v7 equivalent code.