No more slow requests with Delayed Job (Rails 4)

I'm sure you've already seen pages taking forever to load... Not the best user experience huh ! This usually happens when you imported a big file or upload a picture. Those are usually long tasks that should be run in the background to keep the user experience smooth.

Here are a few examples :

• Import and process large quantity of data
• Make calls to external services
• Sending lots of emails
• Processing pictures
• 'More super long tasks...'

Now that you know what we are talking about, let's see how to avoid this kind of things in Rails !

No worries, we don't have to write it from scratch, there are some very good solutions out there. Today, we'll use Delayed Job with Active Record.

Delayed Job supports other ORM : DataMapper, IronMQ, Mongoid, MongoMapper and Redis.

We will create a simple application which will 'import' a big file. You can get the code here.

Get Delayed Job

First thing first, let's head to the Gemfile and add the gem :

# Gemfile
gem 'delayed_job_active_record'

Followed by a 'bundle install'.

Now that we have Delayed Job installed, we can generate the table that will store our queue of jobs :

rails generate delayed_job:active_record
rake db:migrate

Setup our sample app

Let's add some controller and model to our app :

rails g controller Uploads index format

And set the root for our app :

# routes.rb
Rails.application.routes.draw do
  get 'uploads/index'
  get 'uploads/format'
  root 'uploads#index'
end

If you head over to http://localhost:3000 or whatever your local url is, you should access the 'Uploads#index' view.

Let's create a document model of which we'll save a record during the upload.

rails g model document name:string imported:boolean

Go to the migration file and add a default value of 'false' to the column 'imported'.

class CreateDocuments < ActiveRecord::Migration
  def change
    create_table :documents do |t|
      t.string :name
      t.boolean :imported, default: false

      t.timestamps
    end
  end
end

And run the migration.

rake db:migrate

During the upload, we will format the document, for 5 seconds. Yup, sorry about that, it's just a sample app, no real logic here !

To do that, we add a 'format' method to our document :

# app/models/document.rb
class Document < ActiveRecord::Base

  def format
    sleep 5
    update_column :imported, true
  end

end

The column 'imported' allows us to know when the formatting is done. It will become useful when we integrate Delayed Job.

All we need now is to update our view to have a cool upload form or maybe just a link... and create a new document and 'format' it in the controller !

<!-- views/uploads/index.html.erb -->
<h1>Upload</h1>
<%= link_to 'Upload & Format an invisible file !', uploads_format_path %>

<!-- views/uploads/format.html.erb -->
Imported & Formatted !
<%= link_to 'Back', root_path %>

# app/controllers/uploads_controller.rb
class UploadsController < ApplicationController

  def index

  end

  def format
    @doc = Document.create(name: 'Invisible')
    @doc.format
  end

end

Now, go the homepage and click on the link... and nothing happens for 5 seconds ! Thanks to Turbolinks, you won't even see the page loading. I'm sure you agree that it's a pretty bad experience for a user.

Delayed Job to the rescue !

Simply change

# app/controllers/uploads_controller.rb
@doc.format

to

# app/controllers/uploads_controller.rb
@doc.delay.format

and try again ! Wow, it's instant now ! Not so fast, the document was not actually formatted, we have one thing to do first. Open a new terminal/tab/split and run

rake jobs:work

You should see :

[Worker(host:X pid:4967)] Starting job worker
[Worker(host:X pid:4967)] Job Document#format (id=1) RUNNING
[Worker(host:X pid:4967)] Job Document#format (id=1) COMPLETED after 5.0041
[Worker(host:X pid:4967)] 1 jobs processed at 0.1992 j/s, 0 failed

It's working ! If you're wondering, yes, you have to run this rake task all the time when working with Delayed Job, including on your server. But no worries, there are easy ways to manage that as we will see later.

BTW : If you want to check your jobs in rails console, the model is Delayed::Job. With that, you can list them, delete them, dance with them or whatever is on your mind.

Now we're going to improve the user experience by letting him know that we are formatting the file and update the page when it's done. To do that, we'll simply use some long-polling javascript.

Long Polling

First, we need to change our 'format' view :

<!-- views/uploads/format.html.erb -->
<div data-status="<%= uploads_status_path(@doc) %>">
  <p>Formatting...</p>
  <%= link_to 'Back', root_path, style: 'display: none;' %>
</div>

We will show 'Formatting...' until the file is formatted, then we'll display a new message and show the back button. (OMFG, inline CSS, yeah well...)

Then we need a new action on the uploads controller that will tell us when the file is formatted :

# app/controllers/uploads_controller.rb
  def status
    @doc = Document.find(params[:document_id])
    render json: { id: @doc.id, imported: @doc.imported }
  end

And the route :

get 'uploads/status/:document_id', to: 'uploads#status', as: 'uploads_status'

For the last step, we need javascript ! Create a new file in 'assets/javascripts/', I called mine 'upload' :

$(document).on 'ready page:load', ->

  poll = (div, callback) ->
    # Short timeout to avoid too many calls
    setTimeout ->
      console.log 'Calling...'
      $.get(div.data('status')).done (document) ->
        console.log 'Formatted ?', document.imported
        if document.imported
          # Yay, it's imported, we can update the content
          callback()
        else
          # Not finished yet, let's make another request...
          poll(div, callback)
    , 2000

  $('[data-status]').each ->
    div = $(this)

    # Initiate the polling
    poll div, ->
      div.children('p').html('Document formatted successfully.')
      div.children('a').show()

It's a lot of code, but we are basically just making calls every 2 seconds until the server tells us that the file has been formatted.

Try it ! After a few seconds, the text should be updated. Much better for the user ! Of course, it's even better with a progress bar or any visual indication ;)

Let's go back to Delayed Job.

Adding 'delay' everytime you want to call that method through Delayed Job is a bit annoying. But there is a simpler way : 'handle_asynchronously'

You can remove 'delay' from '@doc.delay.format'. Instead, we're going to add 'handle_asynchronously' directly next to the method :

# app/models/document.rb
class Document < ActiveRecord::Base

  def format
    sleep 5
    update_column :imported, true
  end
  handle_asynchronously :format

end

Restart the Delayed Job task. Everything should still work perfectly !

BTW : When debugging your code, you might want to call the methods without delay. To do that, just use format_without_delay instead of format.

More options

You can also pass some options to handle_asynchronously :

handle_asynchronously :get_ready, priority: 1, run_at: Proc.new { 10.seconds.from_now }, queue: 'queue1'

Priority : Lowest number are executed first

Run_at : Date to run the job at. Give it a Proc to have a date defined by the moment the job is picked up by DJ.

Queue: With queues, you can setup different group of jobs. You could have the jobs of 'queue1' processed by one process and 'queue2' by another. You can run specific queues like this :

QUEUE=queue1 rake jobs:work

or

QUEUES=queue1,queue2 rake jobs:work

Let's talk about running Delayed Job in production, first on a dedicated server, then on Heroku.

Run on your server

To run Delayed Job on your server, you will have to add the gem 'daemons' to your Gemfile. Don't forget to 'bundle install'. Now you can run :

bin/delayed_job start

or in production :

RAILS_ENV=production bin/delayed_job start

If you want to stop it, just replace 'start' by 'stop'. The logs are available in 'log/delayed_job.log'. You can check more options here.

Run on Heroku

To use Delayed Job on Heroku, you will have to use some Workers.

After pushing your code, you will need to run the following command if you don't have any worker.

heroku ps:scale worker=1

The worker should start processing the tasks.

Note that running Delayed Job on Heroku in that way is far from cost effective since you will pay for a worker 24/7 even if you process 2 tasks per day. The solution to this problem is Workless which will hire or fire workers based on your needs. I am planning to write a tutorial about Workless, since the use of multiple queues can become tricky !

The End

Now, it's time to speed up your application by setting in background everything that belongs there :)