Future Jobs & Retries

Future jobs

Future jobs are any job with a RunAfter later than right now/should not be be executed immediately. These can be newly queued jobs, or jobs that need to be retried due to failure.

New items are added to the future jobs queue during the following scenarios:

• System startup: all jobs with a future RunAfter
• When new jobs are added with a future RunAfter date
• When errors occur when handling normally scheduled jobs

Adding jobs

Jobs may get added to the future jobs queue through multiple paths

A job is Enqueued() with a RunAfter > now

• The job is added to the worker's internal state to be watched for later execution
• The job is inserted into the jobs table

During system startup

• All jobs with RunAfter > now are added to the worker's internal state and monitored for later execution

A Handler's Handle() raises an error

• Exponential backoff is calculated
• Job is updated with RunAfter set to the backoff, and error set
• Job is added to the worker's internal state and monitored for later execution

Processing future jobs

Every worker checks the jobs queue table for future jobs at startup and adds them to the worker's internal state. When any job in the worker's internal state has a future job due within the next 30 seconds, a goroutine is dedicated to monitoring the job's due timer and notifies all workers that the job is ready to be processed. When future jobs are made eligible to be processed, they're removed from the worker's internal state, relying solely on the neoq_jobs table for persistence.