Runtime uniqueness when using :before_yield as unlock order

[tsubery]

I have a specific use case for sidekiq uniquness for a project I work on. We need the possibility to enqueue more jobs while a job is running but we never want two unique jobs to run concurrently. As far as i understand there is no way to do it as :after_yield unlock order would not allow to enqueue more jobs while a worker is running and :before_yield would allow simultaneous workers. I made a fork adding a locking mechanism for each running job to fix our problem. You can find it here:
https://github.com/tsubery/sidekiq-unique-jobs/tree/runtime_uniqueness
I also added some documentation describing it.
Do you think that is something that might be interesting to other users of the gem?

[mhenrixon]

This might fix the problem some people have with uniqueness and help get rid of some of the confusion I think.

[mhenrixon]

@tsubery ? Any pull request coming for this feature? :)

[tsubery]

The way I implemented it was using another gem called redis-lock.
Before starting a job i would just acquire lock and release it after the job was done or timeout occured.
I don't recall all the specific details but I remember encountering problems with some of the dependencies and ended up patching both redis-lock and mock-redis to make the tests pass.
It is very likely I'm missing some knowledge to create an appropriate solution but it makes more sense to me to use a lock inside the specific job than to use my implementation that relies on patching two extra gems.
I would be very curious to hear about alternative solutions.

[stevenjonescgm]

I had a similar use case (wanted the job to be unique in the queue but also able to reschedule itself). I solved/mitigated by writing another worker that simply queued the first so the uniqueness could bounce between them.

Before I noticed unique_unlock_order: :before_yield, I tried to be clever with my FutureWorker and set unique_job_expiration: 0.seconds so that FutureWorker could be rescheduled as soon as item[at] was passed. This caused an exception to be raised from setex: "ERR invalid expire time in setex" because when the scheduled job was picked up, it gets moved to the active queue and attempts to set expire time < 1.

I'll open a pr, but I'm no longer certain that it is a useful change since unique_unlock_order: :before_yield is better for my case and Sidekiq::Worker.perform_in protects from item['at'] < Time.now

[pik]

The current before_yield is completely arbitrary. Basically it will have a near unique behaviour if the jobs are fast to run and the queue is long and absolutely no-uniqueness guarantees in the opposite scenario.

[mhenrixon]

@pik which is also why it randomly fails in builds like https://travis-ci.org/mhenrixon/sidekiq-unique-jobs/jobs/73491936 if I am not mistaken. looks like we are finally able to make the problem visible in the tests even though it is still irregular.

[pik]

@mhenrixon I think the solution @tsubery posted is pretty good; my one big concern is that spin-lock is rather intense and it Redis is single-threaded, so I would like a default to re-queue the job or schedule it with a delay -- perhaps with an option to use the spin-lock.

e.g.

run_lock: <true/false>
run_lock_retry_interval: <num>
run_lock_num_retries: <num>
reschedule_on_lock_fail: <true/false>

I have a patch for this but I think it needs some clean-up and refactoring. Also I don't know if there are any uses for before_yield without a run_lock as multiple jobs can easily run concurrently as long as they come into the queue after the former has already started; on the other-hand the advantage of adding it as a separate sidekiq-unique option would be that it allows for it to remain backwards-compatible.

[mhenrixon]

Check version 4.0.0