Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not truncate activity failure info if there is not many of them #7367

Open
yiminc opened this issue Feb 20, 2025 · 1 comment
Open

Do not truncate activity failure info if there is not many of them #7367

yiminc opened this issue Feb 20, 2025 · 1 comment
Labels
enhancement New feature or request

Comments

@yiminc
Copy link
Member

yiminc commented Feb 20, 2025

Currently server would truncate activity failure if it exceeds 4KB (default threshold). This make sense if there is many pending activities (like thousands), but it does not make sense if there is only one activity.

A better solution is to only truncate if the aggregated failure from all pending activities exceeds some lager threshold.

@yycptt yycptt added enhancement New feature or request and removed potential-bug labels Feb 20, 2025
@yycptt
Copy link
Member

yycptt commented Feb 20, 2025

Some context:

We enforce a fixed 2kb activity failure size limit for each activity. This has a couple of issues:

  • The limit maybe too low if a workflow only have 1 or 2 activities. We have received requests from cluster before saying we need a higher limit.
  • The limit is too high when workflow has lots of pending activities (max 2k pending activities), causing entire mutable state size to reach limit and workflow get terminated.
  • When combined with other things like buffered events, the total mutable state size may reach the limit and get workflow terminated.

Some ideas:

  • A total failure size limit across activities.
  • When ms size reaches the limit, before directly terminating the workflow, see if we can flush buffered events or truncate activity failure message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants