lock: :until_and_while_executing not working for scheduled jobs

[macmartine]

Describe the bug
Duplicate jobs are getting run when scheduling jobs using perform_at

Expected behavior
When a job is scheduled to be run in the future, and a job with the same arguments gets called, it shouldn't be added to the schedule.

Current behavior
Duplicate jobs are added and run.

Worker class

class SequenceActionWorker

  include Sidekiq::Worker
  include Sidekiq::Status::Worker

  sidekiq_options queue: 'default', retry: false, lock: :until_and_while_executing

  def perform id
    Api::ActionPerformerService.new id
  end

end

The jobs are getting executed with:

SequenceActionWorker.perform_at(Time.now + 5.minutes, some_id)

[rept]

I seem to have the same problem but with until_executed.

I'm running version 6.0.6 of the gem and have created a test job like this:

class TestUniqJob < SidekiqJob
  sidekiq_options queue: :default, retry: false, unique: :until_executed

  def perform
    sleep 3.minutes
  end
end

I use the whenever gem to plan to run this every 2 minutes, however, it should only queue or run this is it's not already queued or running.

Unfortunately it doesn't seem to do anything to prevent the double execution:

Process | TID | JID | Queue | Job | Arguments | Started
70cb7842c19e:3912:912125873e19 | 126ozc | 4df5d4b4ee51844a37ed7d05 | default | TestUniqJob |   | 1 minute ago
70cb7842c19e:3912:912125873e19 | 126pds | 42580284ad9de2f57f14544e | default | TestUniqJob |   | 24 seconds ago

[gommo]

Anyone find a solution to this?

[mhenrixon]

@rept might be a stupid question but... Did you setup the middleware? Also what is in the class SidekiqJob?

Edit: I tried your code and I can't replicate the problem with it.

In the rails console I do the following:

[2] pry(main)> 10.times { p TestUniqJob.perform_async }
"ed75904f197510f8cae53606"
nil
nil
nil
nil
nil
nil
nil
nil
nil
=> 10

The sidekiq server process logs the following:

6:12:40 PM worker.1 |  2019-04-13T16:12:40.704Z 27345 TID-ovv4e7ybp TestUniqJob JID-ed75904f197510f8cae53606 INFO: start
6:15:40 PM worker.1 |  2019-04-13T16:15:40.914Z 27345 TID-ovv4e7ybp TestUniqJob JID-ed75904f197510f8cae53606 INFO: done: 180.21 sec

Only one job starts and it runs for 3 minutes.

[mhenrixon]

@macmartine I can't replicate the problem for your example either:

SequenceActionWorker.perform_at(Time.now + 5.minutes, 1)
=> "ece62bec44a2769af68cbd5b"
SequenceActionWorker.perform_at(Time.now + 5.minutes, 1)
=> nil
SequenceActionWorker.perform_at(Time.now + 5.minutes, 1)
=> nil
SequenceActionWorker.perform_at(Time.now + 5.minutes, 2)
=> "d34950d4b74ea9e1550a21cd"
SequenceActionWorker.perform_at(Time.now + 5.minutes, 3)
=> "629bf13bab02e6aa36fa1e4e"
SequenceActionWorker.perform_at(Time.now + 5.minutes, 4)
=> "96a10e00f7dd5133da38bf0d"
SequenceActionWorker.perform_at(Time.now + 5.minutes, 4)
=> nil
SequenceActionWorker.perform_at(Time.now + 5.minutes, 4)
=> nil
SequenceActionWorker.perform_async(1)
=> nil
SequenceActionWorker.perform_async(4)
=> nil

Looks like everything is working fine to me... Much later in the sidekiq logs...

6:24:25 PM worker.1 |  2019-04-13T16:24:25.362Z 29931 TID-owtz5i293 SequenceActionWorker JID-ece62bec44a2769af68cbd5b INFO: start
6:24:31 PM worker.1 |  2019-04-13T16:24:31.493Z 29931 TID-owtz5hxg7 SequenceActionWorker JID-d34950d4b74ea9e1550a21cd INFO: start
6:24:38 PM worker.1 |  2019-04-13T16:24:38.772Z 29931 TID-owtz5i2ef SequenceActionWorker JID-629bf13bab02e6aa36fa1e4e INFO: start
6:24:38 PM worker.1 |  2019-04-13T16:24:38.777Z 29931 TID-owtywwcpz SequenceActionWorker JID-96a10e00f7dd5133da38bf0d INFO: start
6:25:25 PM worker.1 |  1
6:25:25 PM worker.1 |  2019-04-13T16:25:25.661Z 29931 TID-owtz5i293 SequenceActionWorker JID-ece62bec44a2769af68cbd5b INFO: done: 60.299 sec
6:25:31 PM worker.1 |  2
6:25:31 PM worker.1 |  2019-04-13T16:25:31.512Z 29931 TID-owtz5hxg7 SequenceActionWorker JID-d34950d4b74ea9e1550a21cd INFO: done: 60.019 sec
6:25:38 PM worker.1 |  3
6:25:38 PM worker.1 |  4
6:25:38 PM worker.1 |  2019-04-13T16:25:38.793Z 29931 TID-owtz5i2ef SequenceActionWorker JID-629bf13bab02e6aa36fa1e4e INFO: done: 60.021 sec
6:25:38 PM worker.1 |  2019-04-13T16:25:38.793Z 29931 TID-owtywwcpz SequenceActionWorker JID-96a10e00f7dd5133da38bf0d INFO: done: 60.016 sec

[mhenrixon]

@gommo happy to have a look at your problem too but you need to provide something to go on otherwise I am closing this issue.

[jozefvaclavik]

@rept why do you have there unique: :until_executed instead of lock: :until_executed I can't see anywhere such option.

Was just implementing uniq lock for scheduled jobs and this works fine for me. sidekiq_options queue: 'orders', unique_across_queues: true, lock: :until_and_while_executing

[mhenrixon]

Closing due to inactivity. I requested more information and didn't get a reply.