Using Sidekiq across different applications

In a project I was working on, we wanted to split the big monolith application we had into several smaller repositories. In particular, we wanted the user-facing part of our application to be completely separate from the administrative backend. This is all nice and well, but then we had to consider how we can still make sure that jobs such as sending email are properly enqueued and executed. Since we were using Sidekiq, which just runs atop of a Redis store, this is not hard to do in principle: You just put some items in your queue from one repository and read from that same queue in the other repository.

Let’s quickly review how enqueuing and dequeuing works in Sidekiq. If you have the following worker in your project:

class EmailWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'email', retry: false

  # dequeuing and executing job
  def perform(recipient, message)
    send_email recipient, message
  end

  private

  def send_email(recipient, message)
    # code omitted
  end
end

Then at some other point in your code you can enqueue a job

EmailWorker.perform_async('test@example.org', 'Hello World')

If you’re in Rails, you can just run bundle exec sidekiq from your project root and Sidekiq will start and automatically pick up any work for which there is a corresponding worker in the project (for more details, check the Sidekiq Wiki).

But what happens when you have two different repositories, one which enqueues jobs and one which dequeues them? (In practice both projects could be enqueueing and dequeueing jobs, but let’s keep it simple.) If the EmailWorker is only in the project that actually executes the job, you cannot call EmailWorker#perform_async in the other repository. But remember that Sidekiq just runs atop of Redis - it just pushes jobs to some queues and reads from those queues. Instead of the more “magical” perform_async syntax, you can also drop down to a slightly lower level and do the following to enqueue a job:

require 'sidekiq'
Sidekiq::Client.push(
  class: 'EmailWorker',
  queue: 'email',
  retry: false,
  args: ['test@example.org', 'Hello World']
)

Notice that the class option is just a string here. This is nice, since it keeps the code decoupled from the actual implementation of the EmailWorker. The enqueuing repository does not need to be concerned with how the actual worker is implemented in another project - in fact it’s even possible that the EmailWorker doesn’t exist at all.

However, this approach also has a number of drawbacks:

Of course, the simplest way to mitigate these drawbacks would be to simply create a method send_email that encapsulates this behaviour. However, what if there was a way to just reuse the normal Sidekiq syntax without having to share the worker classes?

Enter worker proxies.

The solution is quite simple actually: Instead of using the worker classes directly, a project that wants to enqueue jobs for specific workers just defines proxy classes. These proxy classes follow a particular naming convention: An EmailWorkerProxy matches an EmailWorker. They can also define their default Sidekiq options for queueing, retry behaviour etc. just like regular old Sidekiq workers. This is what a worker proxy could look like:

class EmailWorkerProxy
  include WorkerProxy
  sidekiq_options queue: 'email', retry: false
end

Notice that the class doesn’t include any #perform method: That is up to the actual worker class for which this class is only a proxy. Now somewhere in your code you could just run:

EmailWorkerProxy.perform_async('test@example.org', 'Hello World')

There is just one drawback to this solution: It is not provided by Sidekiq. However, it turns out that if we muck a little with Sidekiq’s internals, it’s actually not that hard to implement the WorkerProxy module ourselves. This is what it looks like:

module WorkerProxy
  def self.included(base)
    base.send(:include, Sidekiq::Worker)
    base.extend ClassMethods
  end

  module ClassMethods
    # override
    def client_push(item)
      # get sidekiq options defined in proxy
      extended_item = get_sidekiq_options.merge(item)

      # strip 'Proxy' from class name
      proxied_class_name = to_s[0..-6]

      super(extended_item.merge 'class' => proxied_class_name)
    end
  end
end

What this does is basically setting up the including class as a normal Worker class by including Sidekiq::Worker, but then overriding the internal #client_push method. Besides merging the default options defined in the WorkerProxy file with any options passed in directly, it just replaces the ‘class’ option - which would be e.g. EmailWorkerProxy with the actual worker class, in this case EmailWorker.

Now in all fairness, utilising a private API is not really solid design and has the risk of breaking with future releases. It does work well in our current projects, though, and it has a certain elegance, at least as a proof of concept. It would be nice if Sidekiq offered such an option by default.

Tags: ruby, sidekiq, SOA