Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AzureStorageOrchestrationService fails to store an event once OrchestrationStatus gets bigger #1185

Open
dsempi opened this issue Feb 5, 2025 · 0 comments

Comments

@dsempi
Copy link
Contributor

dsempi commented Feb 5, 2025

If the orchestration status is a more complex class, that serializes into a larger string, the following exception is thrown and orchestration does not proceed further:

DurableTask.Core[29]
TaskOrchestrationDispatcher-091002dd69a645bb89247b8f24b44b98-0: Unhandled exception with work item '8652685d247d44589559f13e3096234e': DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
 ---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge

Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}

Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8

   at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
   at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
   at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

Backing off for 10 seconds until 5 successful operations
DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage
 ---> Azure.RequestFailedException: The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.
RequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
Time:2025-02-05T11:55:49.497Z
Status: 400 (Bad Request)
ErrorCode: PropertyValueTooLarge

Content:
{"odata.error":{"code":"PropertyValueTooLarge","message":{"lang":"en-US","value":"The property value exceeds the maximum allowed size (64KB). If the property value is a string, it is UTF-16 encoded and the maximum number of characters should be 32K or less.\nRequestId:4ad3ecb0-d57a-4e74-8091-b78ea9224b7c\nTime:2025-02-05T11:55:49.497Z"}}}

Headers:
Server: Azurite-Table/3.33.0
x-ms-error-code: REDACTED
x-ms-request-id: 4ad3ecb0-d57a-4e74-8091-b78ea9224b7c
x-ms-version: REDACTED
Date: Wed, 05 Feb 2025 11:55:49 GMT
Connection: keep-alive
Keep-Alive: REDACTED
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8

   at Azure.Data.Tables.TableRestClient.MergeEntityAsync(String table, String partitionKey, String rowKey, Nullable`1 timeout, String ifMatch, IDictionary`2 tableEntityProperties, QueryOptions queryOptions, CancellationToken cancellationToken)
   at Azure.Data.Tables.TableClient.UpsertEntityAsync[T](T entity, TableUpdateMode mode, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30
   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.ClientResponseExtensions.DecorateFailure(Task`1 responseTask) in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34
   at DurableTask.AzureStorage.Storage.Table.InsertOrMergeEntityAsync[T](T tableEntity, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/Table.cs:line 103
   at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.UpdateStateAsync(OrchestrationRuntimeState newRuntimeState, OrchestrationRuntimeState oldRuntimeState, String instanceId, String executionId, Nullable`1 eTagValue, Object trackingStoreContext, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 992
   at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in /_/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1202
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 613
   at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in /_/src/DurableTask.Core/TaskOrchestrationDispatcher.cs:line 212
   at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in /_/src/DurableTask.Core/WorkItemDispatcher.cs:line 373

This limitation is not present for ServiceFabric and Emulator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant