You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#137 addresses purging tasks in an error state. But long running periodic tasks can have an ever increasing list of execution errors that will never get purged. A function should be provided that can be used to trim the execution list for periodic tasks so that they don't have unbounded growth.
Another idea would be to have a max executions config setting and do a LTRIM every time we add to the execution list here:
Another use case that could create execution history issues is for a task that has many retries before failing.
For example, say there's a queue that delivers many small payloads to https://app.example.com. If that website become unavailable and the delivery task retries every minute for a day, it's possible to end up with over a thousand recorded executions for just one task. Each execution record takes up roughly as much space as the queued task, so 1 bad task could end up consuming over a thousand tasks worth of space.
For just one task it's not that bad but if thousands of tasks all fail the same way, it's easy to run out of memory in redis.
I think the best solution here is adding a max executions setting, then doing an LTRIM if it's set.
We should then store the number of executions in a separate Redis key (or in the JSON payload of the task).
Another issue is that unique or periodic tasks share one execution history. For example, if you have a periodic task that retries up to N times, then the N retries don't currently apply to each period -- they apply to the overall lifetime of the periodic task, which isn't what we want. That's because we store each failed execution under the task key and don't clean them up.
#137 addresses purging tasks in an error state. But long running periodic tasks can have an ever increasing list of execution errors that will never get purged. A function should be provided that can be used to trim the execution list for periodic tasks so that they don't have unbounded growth.
Another idea would be to have a max executions config setting and do a
LTRIM
every time we add to the execution list here:tasktiger/tasktiger/worker.py
Lines 409 to 412 in fe82e84
The text was updated successfully, but these errors were encountered: