undefined method `get_sidekiq_options' for "MyScheduledWorker":String

[girak]

Our Sidekiq redis instance is shared among multiple services.
So workers are available on one of the repo while not on the others.

When a scheduled or retried job is being consumed by the sidekiq poller running on another service, we need to to safely re-enqueue the task without exception.

Currently Sidekiq does re-enqueue jobs without the worker class without issue, but apparently the unique jobs middleware is not accounting for this case.

I have workaround the issue with a monkey patch, but an official fix is much appreciated. Thanks for the hard work!

[mhenrixon]

@girak could you point me to where that job is re-enqueued in sidekiq? I'm pretty sure we could mimic that behaviour.

[girak]

@mhenrixon https://github.com/mperham/sidekiq/blob/d40f7cc6af61ef9046c34429bef2932ca85f889f/lib/sidekiq/client.rb#L182

anywhere that refers get_sidekiq_options

[mhenrixon]

Ok so this happened to me as well now that I split up our backend processing in multiple different projects. I will see if I can get this solved somehow. First a failing test case.

[mhenrixon]

For us it looks like the error is with Sidekiq. Could you provide a stacktrace?

[felixyz]

I've got a quirky little RSpec spec here: https://gist.github.com/Felixyz/8781057
It test drives (synchronously) the Sidekiq retry mechanism with sidekiq-unique-jobs inserted in the middleware chain and attempts to mimic the setup we have, with two queues. The script does not reproduce the bug, from which I draw the conclusion that the bug (if it is a bug) is probably in Sidekiq's higher-level coordination classes, like Manager, Fetcher or Launcher.

Whatever it is, it's perhaps not so much a bug as an uncompatibility in sidekiq-unique-job's handling of the parameters? By which I mean: this is probably not an issue that we would report with Sidekiq itself, right?

[felixyz]

@girak Would you care to share your patch?

[felixyz]

One more observation that might be useful: we're getting fewer of these errors every day. My impression is that they might occur after a restart of the Sidekiq processes. We are running Sidekiq Pro. Are others who have been seeing this problem running Sidekiq or Sidekiq Pro?

[mhenrixon]

Yeah so @felixyz is right and that's why I was asking for a stacktrace. @girak I can't replicate that sidekiq unique jobs should be the problem in this situation. The crash actually happens in Sidekiq and the reason it happens in Sidekiq for us is because the wrong server was processing jobs it could not handle (we assigned it to the wrong queue).

[felixyz]

@mhenrixon I'm not sure you can get a great stack trace because of the async nature of Sidekiq, but (as you probably know) the line that causes the exception is:
/app/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.6.7/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:in `call'

In any case: yes, we also concluded that a worker was assigned a job from the wrong queue. However, I haven't been able to track down the reason why this happens. Any tips for that?
(The only thing I can think of is that our two kinds of workers inherit from a common base class, which is the one that mixes in Sidekiq::Worker.)

[mhenrixon]

@felixyz for us it was just me assigning the job to the wrong queue simple as that. Unfortunately we had some trouble updating Heroku so deploys would be listed as successful even though it failed in CI. Took me a while to figure out why the code changes didn't "take". We have seen a couple of incidents after but I honestly have no idea. The only thing I've been able to think of is that Sidekiq simply picks the wrong job.

[sahin]

we have the same problem , I hope this helps...

NoMethodError - undefined method get_sidekiq_options' for "DataTAWorker":String: /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sidekiq-unique-jobs-2.3.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:124:in block in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:159:inprocess_single'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:47:in push' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:86:inpush'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:260:in block (2 levels) in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:inmap'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:in block in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:inwith'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq.rb:67:in redis' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:252:inretry'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:inblock (2 levels) in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:in each' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:inblock in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:in loop' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:ineach'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:401:in retry_all' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/web.rb:162:inblock in class:Web'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:inblock in compile!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:in []' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:inblock (3 levels) in route!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:976:in `route_eval'

[girak]

@felixyz Hey guys, here's the monkey patch i did on our repos.

module SidekiqUniqueJobs
  module Middleware
    module Client
      class UniqueJobs
        def unique_enabled?
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique'] || item['unique']
          else
            false
          end
        end

        def unique_job_expiration
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique_job_expiration']
          else
            SidekiqUniqueJobs::Config.default_expiration
          end
        end
      end
    end
  end
end

[sahin]

thx a lot.... When do u think u can merge it to master?

[sahin]

we had problems using monkey patch in the new sidekiq 3.0, for now, we are doing this in sidetiq

def perform

uniq = Hash.new
Sidekiq::DeadSet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

uniq = Hash.new
Sidekiq::RetrySet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

end

[Altonymous]

Any updates on this issue?

2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: undefined method `get_sidekiq_options' for "Reporter":String
2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: /Users/nunya/.rvm/gems/ruby-2.1.0@reporting-bot/gems/sidekiq-unique-jobs-3.0.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:91:in `unique_enabled?'