Add fine control for workflow cache eviction

[antmendoza]

Is your feature request related to a problem? Please describe.

Some users want to control when workflows are evicted from the cache. For example:

• They may want to evict workflows they know will sleep for days. For non-latency-sensitive use cases, the cost of storing these workflows in the cache is higher than polling and replaying the workflow history when the timer fires. Ideally, they want to prioritize keeping in the cache workflows that they know are actively active and latency-sensitive.

• They may want to keep in cache workflows that will be queried after the workflow is closed (and evicted from cache). Querying closed workflows comes with extra latency since the workflow history has to be fetched and replayed by the worker before executing the query. Users want to control how long the workflow will be stored in the cache after the workflow is closed to prevent the extra latency that comes with fetch + replay

Describe the solution you'd like

Additional context

Per-SDK Tickets

• Go -
• Java -
• Core -
• TypeScript -
• Python -
• .NET -
• Ruby -
• PHP -
• Temporal CLI -

[cretz]

This will need some internal discussions on design, and some of the work may be server side if post-close queries don't use the sticky queue anyways.

[Quinn-With-Two-Ns]

They may want to evict workflows they know will sleep for days.

Currently SDKs use a LRU strategy for eviction, I think their is room for optimization here. For example we could factor in history size when choosing what workflow to evict. If it is worth optimizing eviction vs just asking users to increase their memory and cache size I am not sure.

and some of the work may be server side if post-close queries don't use the sticky queue anyways.

Yes their is some server side work needed to allow query to always use the workflow cache temporalio/temporal#4463.

I do think it should be opt in since keeping a closed workflow in cache by default on the chance that it may be queried, is not appropriate.