20211115131438-sidekiq_stuff.org

Sidekiq stuff

For a deep 🍑 dive, read this https://www.mikeperham.com/how-sidekiq-works/

Starting it up

sidekiq -q default -q foo

sidekiq --help shows you all the startup options.

Poking around in the console

See Mike Perham’s A Tour of the Sidekiq API post. This has most of what I need.

Running a job in the console (when sidekiq isn’t running)

job = Sidekiq::Queue.new('somequeue').first
job.klass.constantize.new.perform(*job.args)

A Queue is kind of enumerable. You can convert it to_a and find the job if you need. Otherwise just, queue.clear to empty it out and start over.

Scheduled jobs

See also https://www.rubydoc.info/github/mperham/sidekiq/Sidekiq/ScheduledSet

irb(main):001:0> scheduled_set = Sidekiq::ScheduledSet.new
=> #<Sidekiq::ScheduledSet:0x00000000090f6a50 @name="schedule", @_size=1>

Add a scheduled job to the queue

See also https://www.rubydoc.info/github/mperham/sidekiq/Sidekiq/SortedEntry

s = Sidekiq::ScheduledSet.new
s.first.add_to_queue

Logging what’s in the queue

filtered_jobs = s.select { |job| job.args.last == 3 }

sorted_jobs = filtered_jobs.sort_by { |job| job.at }

s = Sidekiq::ScheduledSet.new
s.each do |job|
  puts "Job Class: #{job['class']} : Arguments: #{job.args} : Scheduled At: #{Time.at(job.at)}"
end

Retry Sets

r = Sidekiq::RetrySet.new

Enqueuing in bulk

See also https://www.rubydoc.info/github/mperham/sidekiq/Sidekiq%2FClient:push_bulk

Sidekiq::Client.push_bulk('class' => SomeWorkerClass, 'args' => args)

args is a 2D array.

Middleware

https://github.com/mperham/sidekiq/wiki/Middleware

Unique Jobs (ENT)

https://github.com/mperham/sidekiq/wiki/Ent-Unique-Jobs

• Jobs are unique for the duration set OR until the job completes successfully.
• If job retries beyond the specified duration, another job can be pushed.

Manually deleting unique job locks

Try using the redis-cli. Something like…

redis-cli --scan --pattern "*unique*" | xargs redis-cli DEL

Scheduled jobs

https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs

Scheduled jobs (including retry queue) are polled around every 5 seconds by default. This can be configured with the average_scheduled_poll_interval config property.

Rate Limiting (ENT)

https://github.com/mperham/sidekiq/wiki/Ent-Rate-Limiting

There’s several options, here is one example.

Window Limiting Example

class SomeWorker
  include Sidekiq::Worker

  def perform
    # Limit 5 per 1 second window. If limit is reached, job thread will pause
    # (sleep) for 1 second before trying again. If still limited, job will be
    # rescheduled linearly up to 20 times until finally going to retry queue.
    limiter = Sidekiq::Limiter.window('test_limit_worker', 5, :second, wait_timeout: 1)
    limiter.within_limit do
      puts "****** RUNNING THE WORKER: at #{Time.current} ********"
      raise if rand(9).zero? # Simulate some job errors
    end
  end
end

Poison Pills

Jobs that cause the process to crash are called “poison pills”. You can find these in the logs (Killed poison pill) See also https://github.com/sidekiq/sidekiq/wiki/Reliability#poison-pills

Batches

https://github.com/mperham/sidekiq/wiki/Batches

Gives you a collection of jobs that can be monitored as a group. And has callbacks for success and complete.

class BatchWorker
  include Sidekiq::Worker

  BATCH_DESCRIPTION = 'Test Limiter Batch'

  def perform
    batch = Sidekiq::Batch.new
    batch.description = BATCH_DESCRIPTION
    batch.jobs do
      100.times do
        SomeWorker.perform_async
      end
    end
  end
end

Batch Sets

Pragmatically get info about current batches (this will only show batches that are not complete)

Sidekiq::BatchSet.new.each { |batch| p batch }

• Batches can be deleted programmatically: batch.delete

Capsules

Introduced in Sidekiq v7. This should let you define concurrency options per queue. https://github.com/mperham/sidekiq/blob/v7.0.0/docs/capsule.md#sidekiq-70-capsules

Add-ons

See also Ruby gems

• sidekiq-grouping: combines individually enqueued jobs into one job with a single argument. Useful for converting a bunch of single requests to a single bulk request.

Setup

For redis configuration just set REDIS_URL env var. see https://github.com/sidekiq/sidekiq/wiki/Using-Redis for more options.

Testing

To run jobs inline in a test:

require 'sidekiq/testing'

RSpec.describe 'My Test' do
  before do
    Sidekiq::Testing.inline!
  end

  after do
    Sidekiq::Testing.fake!
  end
end