while_executing working wrong

[hiennguyenvan-agilityio]

As description, it is to make sure that a job can be scheduled any number of times but only executed a single time per argument provided to the job we call this runtime uniqueness. But it not work for me

class Job
  include Sidekiq::Worker
  sidekiq_options queue: :test, retry: true, unique: :while_executing

  def perform(arg)
    sleep 5.minutes
  end
end

Job.perform_async(1)
Job.perform_async(1)
Job.perform_async(2)

Have two Job with arg is 1 execute at the time

System information

• redis: 3.3.1
• sidekiq: 4.1.3
• sidekiq-unique-jobs: 4.0.18
• sidekiq-limit_fetch (https://github.com/brainopia/sidekiq-limit_fetch#limits): 3.1.0

Sidekiq Config

:queues:
  - test
  - default
:limits:
  test: 3

[natematykiewicz]

Have you tried it without sidekiq-limit_fetch? Its says that it's incompatible with "any other plugin that rewrites fetch strategy of sidekiq.".

This limit fetch gem is going to try to control when jobs are performed from the queue, but that's exactly what this gem is needs to do for the while_executing option.

[hiennguyenvan-agilityio]

@natematykiewicz Thank for feedback.

I just check it without sidekiq-limit_fetch. It's same result, still have two Job with arg is 1 execute at the time.

[hiennguyenvan-agilityio]

Below is list gem relate with sidekiq I used

gem 'sidekiq', '~> 4.1', '>= 4.1.2'
gem 'sidekiq-cron', '~> 0.4.0'
gem 'sidekiq-failures'
# gem 'sidekiq-limit_fetch'
gem 'sidekiq-statistic'
gem 'sidekiq-unique-jobs'

[natematykiewicz]

If I were you, I'd remove all gems except sidekiq and sidekiq-unique-jobs and then see what you get. If it works, start adding gems in until you find one that doesn't seem to be compatible with sidekiq-unique-jobs.

[hiennguyenvan-agilityio]

gem 'sidekiq', '~> 4.1', '>= 4.1.2'
# gem 'sidekiq-cron', '~> 0.4.0'
# gem 'sidekiq-failures'
# gem 'sidekiq-limit_fetch'
# gem 'sidekiq-statistic'
gem 'sidekiq-unique-jobs'

It's same result, still have two Job with arg is 1 execute at the time.

[aserafin]

I had the same experience - the only option that seems to work for me is until_executed

[cheneveld]

I was having some issues with this. I initially had one worker process with 2 queues:

worker: env DB_POOL=$WORKER_CONCURRENCY bundle exec sidekiq -q default -q keep_em_cached -c $WORKER_CONCURRENCY

And moved my unique jobs on its own process

worker: env DB_POOL=$WORKER_CONCURRENCY bundle exec sidekiq -q default -q keep_em_cached -c $WORKER_CONCURRENCY
another_worker: env DB_POOL=4 bundle exec sidekiq -q another_queue -c 4

This seemed to solve my unique issues. I didn't investigate any more than this so, I can't say what was actually happening.

[mhenrixon]

Thanks for the report @cheneveld

[hiennguyenvan-agilityio]

Sorry, I'm pretty busy recent time.

@cheneveld, @mhenrixon Tried split 2 process but no luck. It's same result.

[untoldone]

@mhenrixon This seems like a pretty core use case to be broken? Super simple repro:

class TestWorker
  include Sidekiq::Worker

  sidekiq_options unique: :while_executing

  def perform(one, two)
    sleep 10
  end
end

And then start with

TestWorker.perform_async 1, 2
TestWorker.perform_async 1, 2
TestWorker.perform_async 1, 2
TestWorker.perform_async 1, 2
TestWorker.perform_async 1, 2

Redis: 3.2.6 (single node, nothing special)
Sidekiq: 4.2.9
Sidekiq-unique-jobs: 4.0.18

Am I missing something here? If this is truly broken, should it be reflected in docs?

[jasonbuehler]

I am also seeing this behavior. Is while_executing option broken? It doesn't appear to lock other jobs out, everything with the same args enqueues all at once.

[mhenrixon]

It has been somewhat fixed in v5.0.5. Check the readme for how it currently works: https://github.com/mhenrixon/sidekiq-unique-jobs#while-executing