Getting OrleansMessageRejectionException during k8s deployment

[olizarevichroman]

We run our application in Kubernetes (k8s) and often encounter failed messages during the deployment process.

For example, consider two grain entities of different types: grain1 and grain2.
These grains are hosted on different pods: grain1 on pod1 and grain2 on pod2.
When accessing data from grain1, it communicates with grain2.

The problem occurs when a client sends a request to grain1, which is routed to pod1. At the same time, during deployment, pod2 is being terminated. When grain1 attempts to fetch data from grain2, we encounter an OrleansMessageRejectionException.

I reviewed issues related to message retries but couldn’t find anything that sheds light on the root cause or a potential solution to this problem.
And I didn't find a way to configure automatic retries as well.

I would like to know how to handle such errors. Should we configure timeouts and membership table update frequencies differently, or is there another approach? Or I should be done at application layer with grain filter?

{
  "message": "Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Connection attempt to endpoint S10.10.43.14:11111:95402761 timed out after 00:00:05\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 219\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 106\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync|30_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 236",
  "data": {},
  "innerException": null,
  "helpLink": null,
  "source": "System.Private.CoreLib",
  "hResult": -2146233088,
  "stackTrace": [
    "at Orleans.Serialization.Invocation.ResponseCompletionSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 81",
    "at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 117",
    "at Orleans.Runtime.ActivityPropagationGrainCallFilter.Process(IGrainCallContext context, Activity activity) in /_/src/Orleans.Core/Diagnostics/ActivityPropagationGrainCallFilter.cs:line 51",
    "at Orleans.Runtime.OutgoingCallInvoker`1.Invoke() in /_/src/Orleans.Core/Runtime/OutgoingCallInvoker.cs:line 88",
    "at Orleans.Runtime.GrainReferenceRuntime.InvokeMethodWithFiltersAsync[TResult](GrainReference reference, IInvokable request, InvokeMethodOptions options) in /_/src/Orleans.Core/Runtime/GrainReferenceRuntime.cs:line 90"
}