Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality to purge periodic task execution histories #151

Open
jkemp101 opened this issue Oct 11, 2019 · 2 comments
Open

Add functionality to purge periodic task execution histories #151

jkemp101 opened this issue Oct 11, 2019 · 2 comments

Comments

@jkemp101
Copy link
Member

#137 addresses purging tasks in an error state. But long running periodic tasks can have an ever increasing list of execution errors that will never get purged. A function should be provided that can be used to trim the execution list for periodic tasks so that they don't have unbounded growth.

Another idea would be to have a max executions config setting and do a LTRIM every time we add to the execution list here:

self.connection.rpush(
self._key('task', task.id, 'executions'),
serialized_execution,
)

@AlecRosenbaum
Copy link
Contributor

Another use case that could create execution history issues is for a task that has many retries before failing.

For example, say there's a queue that delivers many small payloads to https://app.example.com. If that website become unavailable and the delivery task retries every minute for a day, it's possible to end up with over a thousand recorded executions for just one task. Each execution record takes up roughly as much space as the queued task, so 1 bad task could end up consuming over a thousand tasks worth of space.

For just one task it's not that bad but if thousands of tasks all fail the same way, it's easy to run out of memory in redis.

I think the best solution here is adding a max executions setting, then doing an LTRIM if it's set.

@thomasst
Copy link
Member

We should then store the number of executions in a separate Redis key (or in the JSON payload of the task).

Another issue is that unique or periodic tasks share one execution history. For example, if you have a periodic task that retries up to N times, then the N retries don't currently apply to each period -- they apply to the overall lifetime of the periodic task, which isn't what we want. That's because we store each failed execution under the task key and don't clean them up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants