Bloggy #3 - Creating the Yumi library, a JSON API implementation, from scratch

This is the third jutsu in the series 'Building a blogging API from scratch with Grape, MongoDB and the JSON API specification'. We are going to implement the JSON API specification we studied in the previous jutsu.

To do this, we are going to create a library named Yumi that will take care of generating the JSON document for us. I also want to show you how to do some TDD so we'll write the first class using the red/green/refactor flow. Unfortunately, I couldn't do that for each class because it would make this jutsu way too long.

By following this tutorial, you will learn:

• How to do some basic TDD development
• How to write a Ruby library from scratch
• How to isolate responsibilities to make them easier to test
• How to write an implementation of the JSON API specification

Master Ruby Web APIs [advertisement]

Mandatory mention about the book I'm currently writing: Master Ruby Web APIs. If you like what you're reading here, take a look and register to get awesome benefits when the book is released.

The Bloggy series

You can find the full list of jutsus for this series in this jutsu.

The jutsu before this one is available there.

Getting the code

You can get the source code to follow this tutorial from GitHub if you didn't follow the fifteenth jutsu.

Implementation Time

Let's get started with the implementation.

First, if you want an easy way to debug something in your code, don't forget to include the gem pry-byebug in your Gemfile. After that, a simple binding.pry will stop the execution of the server and give you a console to interact with the code at that specification breakpoint. It will also add a few more commands like step, next, finish and continue.

I did a screenjutsu on pry-byebug if you're interested.

Second, restarting the server every time we make a change is a pain in the ass. Let's use Shotgun to auto-reload our Grape application.

Shotgun

To use Shotgun, simply add the gem to your Gemfile.

# Gemfile
gem 'shotgun'

And run bundle install. After that, use shotgun config.ru to start your server instead of rackup.

Note that the default port for Shotgun is 9393 and not 9292. You can also specify the port with the option --port=9292.

Expected Output

For Bloggy, which has posts, comments and tags, we want the JSON document to look like this:

{
  "meta": {
    "name": "Bloggy",
    "description": "A simple blogging API built with Grape."
  },
  "links": {
    "self": "http://localhost:9393/api/v1/posts"
  },
  "data": [
    {
      "type": "posts",
      "id": "56caf33afef9af15b0000000",
      "attributes": {
        "slug": "super-title",
        "title": "Super Title",
        "content": ""
      },
      "links": {
        "self": "http://localhost:9393/api/v1/posts/56caf33afef9af15b0000000"
      },
      "relationships": {
        "tags": {
          "data": [],
          "links": {
            "self": "http://localhost:9393/api/v1/posts/56caf33afef9af15b0000000/relationships/tags",
            "related": "http://localhost:9393/api/v1/posts/56caf33afef9af15b0000000/tags"
          }
        },
        "comments": {
          "data": [
            { "type": "comments", "id": "56cafa8efef9af1a01000000" },
            { "type": "comments", "id": "56cc7afafef9af2d54000000" }
          ],
          "links": {
            "self": "http://localhost:9393/api/v1/posts/56caf33afef9af15b0000000/relationships/comments",
            "related": "http://localhost:9393/api/v1/posts/56caf33afef9af15b0000000/comments"
          }
        }
      }
    }
  ],
  "included": [
    {
      "type": "comments",
      "id": "56cafa8efef9af1a01000000",
      "attributes": {
        "author": "tibo",
        "email": "tibo@example.com",
        "website": "example.com",
        "content": "Awesome"
      },
      "links": {
        "self": "http://localhost:9393/api/v1/comments/56cafa8efef9af1a01000000"
      }
    },
    {
      "type": "comments",
      "id": "56cc7afafef9af2d54000000",
      "attributes": {
        "author": "thibault",
        "email": "thibault@example.com",
        "website": "devmystify.com",
        "content": "Cool"
      },
      "links": {
        "self": "http://localhost:9393/api/v1/comments/56cc7afafef9af2d54000000"
      }
    }
  ]
}

It's a bit long but it contains all the information we need to display a post and access the related resources. If we were building something like a CRM with a lot of lists, it would be super easy to automatically call the given url instead of hard-coding it in the client.

However, for our blog we don't have any dedicated view for tags or comments, only posts. That's why I decided to use a compound document to include everything related to a post in just one document.

This JSON document is what we want to output from our API. Currently, we use Grape default formatting which just calls to_json on the models and only shows the post attributes. We have a lot to do to get what we've seen above.

The Idea: The Yumi library

To build this JSON document, we're going to build a generic library called Yumi. In this library, we will have a Base presenter class from which our API presenters will inherit.

Note: Yumi is the Japanese term for a bow. 'Cause you know, we're basically shooting JSON documents.

To give you an idea of how we will use it in Bloggy, here is our future Post presenter:

module Presenters
  class Post < ::Yumi::Base

    meta META_DATA
    attributes :slug, :title, :content
    has_many :tags, :comments
    links :self

  end
end

Creating Yumi

We are going to create Yumi inside our project, in the folder app/yumi.

I believe the best way to create a library is not necessarily to create a brand new project. It's better to make it inside an existing project (that's what the lib/ folder is for in Rails) before extracting it to a gem. Plus, it's easier for a tutorial to just have one project.

Navigate to the root of your application folder and run the command:

mkdir app/yumi

And update the application.rb file to load the content of this folder:

# application.rb
# Moar
# ...
# Load files from the models and api folders
Dir["#{File.dirname(__FILE__)}/app/models/**/*.rb"].each { |f| require f }
Dir["#{File.dirname(__FILE__)}/app/api/**/*.rb"].each { |f| require f }

# Add this line
Dir["#{File.dirname(__FILE__)}/app/yumi/**/*.rb"].each { |f| require f }
# ...
# Moar

To make Yumi, we will create a bunch of classes that will each take care of generating a specific part of the JSON API document. For example, one class will be responsible for generating the hash that list the attributes. Another one will take care of the links.

Isolating responsibilities like this will make it super easy for us to write tests for each one of them.

Note that since this is a tutorial, we cannot make this library complete - it would take too long and bore you to death. Instead, I focused only on the features we really need. We will review what's missing at the end, feel free to add the features you want ;)

1. Setting up a testing environment

We are going to follow a TDD approach to build the first class of Yumi. Before doing this, we need to setup a testing environment with Rspec, Rack-Test, Factory Girl and Mongoid Cleaner. Note that Factory Girl and Mongoid Cleaner won't be used in the following tests but will be needed in the future when we write the specs for the Bloggy API.

1.1 Include the gems in your Gemfile

First, we simply need to update our Gemfile and add the following gems. Here is a quick description for each one of them:

• Rspec: My favorite testing framework.
• Rack-Test: Required since we're testing a Rack application.
• Factory Girl: Let us define factories for our model with pre-defined attributes.
• Mongoid Cleaner: Used to cleanup our database between each test.

And here is the list of gem:

# Gemfile
# Other gems
# ...
gem 'rspec'
gem 'rack-test'
gem 'factory_girl'
gem 'mongoid_cleaner'

A quick bundle install will install everything in place.

1.2 Setup Rspec

Then we need to setup Rspec using the following command:

rspec --init

This should create the spec/ folder and the spec/spec_helper.rb file. If you're missing them, just create them manually.

1.3 The Spec helper file

Below is the content for the spec/spec_helper.rb file updated with everything needed to make it work within our Rack application. Checkout the comments if you don't understand something.

# spec/spec_helper.rb

# We need to set the environment to test
ENV['RACK_ENV'] = 'test'

require 'ostruct'
require 'factory_girl'
require 'mongoid_cleaner'

# We need to load our application
require_relative '../application.rb'

# Those two files are created later in the jutsu
# but we can already include them
require_relative './factories.rb'
require_relative './support/helpers.rb'

# Defining the app to test is required for rack-test
OUTER_APP = Rack::Builder.parse_file('config.ru').first

# Base URL constant for our future tests
BASE_URL = 'http://example.org:80/api/v1'

RSpec.configure do |config|
  # Load the helpers file required earlier
  config.include Helpers

  config.expect_with :rspec do |expectations|
    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
  end

  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
  end

  # We put this to use the create & build methods
  # directly without the prefix FactoryGirl
  config.include FactoryGirl::Syntax::Methods

  # Setup Mongoid Cleaner to clean everything before
  # and between each test
  config.before(:suite) do
    MongoidCleaner.strategy = :drop
  end

  config.around(:each) do |example|
    MongoidCleaner.cleaning do
      example.run
    end
  end
end

1.4 The factories file

Next we create the file spec/factories.rb and put the following content in it. This file contains a bunch of factories for the Bloggy models.

# spec/factories.rb
FactoryGirl.define do
  factory :post do
    slug 'my-slug'
    title 'My Title'
    content 'Some Random Content.'
  end

  factory :comment do
    author 'Tibo'
    email 'thibault@example.com'
    website 'devmystify.com'
    content 'This post is cool!'
    association :post, factory: :post
  end

  factory :tag do
    slug 'ruby-on-rails'
    name 'Ruby on Rails'
    association :post, factory: :post
  end

end

We'll come back to them later in the tutorial, when we actually use them.

1.5 The helpers file

Finally, here is the content for the spec/support/helpers.rb file. This little module will allow us to write our tests faster by avoiding repeating the same thing in each test. As you can see below, it gives us shortcuts to get the response body as a Ruby hash or convert a symbolized hash to a stringified hash.

Don't forget to add the spec/support folder before creating the file!

# spec/support/helpers.rb
module Helpers

  def json
    JSON.parse(last_response.body)
  end

  def to_json(hash)
    JSON[hash.to_json]
  end

end

1.6 Updating the Mongoid config

We need to update the mongoid config and add a test environment to make our tests run. Here is the updated content for config/mongoid.config.

development:
  clients:
    default:
      database: devblast_blog
      hosts:
        - localhost:27017

test:
  clients:
    default:
      database: devblast_blog_test
      hosts:
        - localhost:27017

1.7 Checking that everything works

Let's see if we did everything correctly.

Run the rspec command and you should see the expected output without any error. If you do see an error, try to understand what's wrong and fix it. If you can't, just leave a comment and I'll see what I can do. ;)

Command:

rspec

Output:

Finished in 0.00031 seconds (files took 1.16 seconds to load)
0 examples, 0 failures

We're done setting up our testing environment. Now it's time to start writing some tests and some code!

2. The Attributes presenter class

The Attributes presenter is responsible for generating a hash of attributes for the specified resource. It will take a resource, a presenter and the list of attributes as parameters.

2.1 The Skeleton

For now, however, it's not going to do anything. We're just going to write an empty skeleton so we can start writing tests for it. Create the folder app/yumi/presenters before adding the file attributes.rb inside.

The code for this file is:

# app/yumi/presenters/attributes.rb
module Yumi
  module Presenters
    class Attributes

      def initialize(options)
        @options = options
      end

      def to_json_api
        # pending
      end

    end
  end
end

There is really nothing much for now. Let's proceed.

2.2 The Spec Skeleton

Let's also add an empty spec file for this class. Add the folders spec/yumi and spec/yumi/presenters before creating the file attributes_spec.rb.

Put this code inside the file:

# spec/yumi/presenters/attributes_spec.rb
require 'spec_helper'

describe Yumi::Presenters::Attributes do

  describe '#to_json_api' do

  end

end

Once again nothing much.

2.3 Running the tests

Let's check that we didn't break Rspec first. Run the rpsec command and if you don't see any error, proceed.

rspec

Output:

No examples found.


Finished in 0.00032 seconds (files took 1.06 seconds to load)
0 examples, 0 failures

2.4 Adding a few let

Now we can get started for real. Sorry for all the preparations. We are about to write the tests for the Yumi::Presenters::Attributes class but before that we need to define a few let that we will use in our tests.

If you don't know what's a let, it's an elegant way to define variables for your tests. To give you an idea, defining the following let:

let(:variable_name) { 'variable_value' }

Is equivalent to something like this:

def variable_name
  @variable_name ||= 'variable_value'
end

All let definitions are wiped between each test.

In a real use-case, the Attributes class will receive Ruby classes as parameters. To mimick that, we use OpenStruct because it makes easy to create objects from hashes. Those objects respond to calls with the dot notation (.) which makes them behave like regular class instances.

Here is the updated code for the attributes_spec file.

# spec/yumi/presenters/attributes_spec.rb
require 'spec_helper'

describe Yumi::Presenters::Attributes do

  let(:attributes) { [:description, :slug] }
  let(:presenter) { OpenStruct.new({}) }
  let(:resource) { OpenStruct.new({ description: "I'm a resource.", slug: 'whatever' }) }


  let(:options) do
    {
      attributes: attributes,
      resource: resource,
      presenter: presenter
    }
  end

  let(:klass) { Yumi::Presenters::Attributes.new(options) }

  describe '#to_json_api' do

  end

end

With those let definitions, we have our options hash that will be passed to the class and that contains the attributes, the resource and the presenter.

The presenter is kinda useless for now, but it will be needed soon. I included it already to avoid having to change this part of the spec later in the tutorial.

2.5 Our first test

Finally, our first test! Here we are testing the to_json_api method and we basically want it to output a hash of the list of given attributes associated with the values of the given resource.

# spec/yumi/presenters/attributes_spec.rb
# describe & let
describe '#to_json_api' do

  it 'outputs the hash with the resource attributes' do
    expect(klass.to_json_api).to eq({
      description: "I'm a resource.",
      slug: 'whatever'
    })
  end

end
# Rest of the file

Let's see how this test runs.

2.6 Running the tests (RED)

Run the rspec command and see how hard our test fails.

rspec

Output:

Failure/Error:
  expect(klass.to_json_api).to eq({
    description: "I'm a resource.",
    slug: 'whatever'
  })

  expected: {:description=>"I'm a resource.", :slug=>"whatever"}
       got: nil

  (compared using ==)

2.7 Fixing it

We can't leave that test fail, that wouldn't be professional. So let's fix it! We just need to implement enough code to make it pass. To do this, we just have to loop through the @attributes and call the method of the same name on the resource object.

Here is the code:

# app/yumi/presenters/attributes.rb
module Yumi
  module Presenters
    class Attributes

      def initialize(options)
        @options = options
        @attributes = @options[:attributes]
        @resource = @options[:resource]
      end

      # Takes the given list of attributes, loops through
      # them and get the corresponding value from the resource
      def to_json_api
        @attributes.each_with_object({}) do |attr, hash|
          hash[attr] = @resource.send(attr)
        end
      end

    end
  end
end

2.8 Re-running the tests (GREEN)

Now let's see if that works. Re-run the tests.

rspec

Output:

Finished in 0.01113 seconds (files took 1.2 seconds to load)
1 example, 0 failures

Awesome, it's working! Normally, we'd do some refactoring here but I don't see what to change in the code we wrote.

2.9 Adding the override

Unfortunately, there is a feature missing from this. We don't want Yumi users to be stuck only with their resource attributes. Maybe they also want to define something in their presenter class or override one of the resource value.

Let's add a test for this. We are also going to use contexts to keep this test from the one we already wrote because they depend on a different set of parameters.

As you can see below, we override the let(:presenter) with a new OpenStruct that contains the same key than the resource we defined (description). When klass will be called, it will use this definition of the presenter when building the options variable.

# spec/yumi/presenters/attributes_spec.rb
# describe & let
describe '#to_json_api' do

  context 'without overrides' do

    it 'generates the hash only with the resource attributes' do
      expect(klass.to_json_api).to eq({
        description: "I'm a resource.",
        slug: 'whatever'
      })
    end

  end

  context 'with overrides' do

    let(:presenter) { OpenStruct.new({ description: "I'm a presenter." }) }

    it 'outputs the hash with the description overridden' do
      expect(klass.to_json_api).to eq({
        description: "I'm a presenter.",
        slug: 'whatever'
      })
    end

  end

end
# Rest of file

2.10 Run the tests (RED)

Let's see how it goes with rspec.

rspec

Output:

Failure/Error:
  expect(klass.to_json_api).to eq({
    description: "I'm a presenter.",
    slug: 'whatever'
  })

  expected: {:description=>"I'm a presenter.", :slug=>"whatever"}
       got: {:description=>"I'm a resource.", :slug=>"whatever"}

Boom, huge fail as expected. Our first test is still passing but the new one is failing hard.

2.11 Fixing it

Luckily, we can fix this code pretty easily. First let's add the @presenter instance variable that will get a value from the options parameter. Then we just need to check if the given presenter can respond to the attribute or not. If it can, we will get the value from it else we'll get it from the resource.

# app/yumi/presenters/attributes.rb
def initialize(options)
  @options = options
  @attributes = @options[:attributes]
  @presenter = @options[:presenter]
  @resource = @options[:resource]
end

# Takes the given list of attributes, loops through
# them and get the corresponding value from the presenter
# or the resource
def to_json_api
  @attributes.each_with_object({}) do |attr, hash|
    hash[attr] = (@presenter.respond_to?(attr) ? @presenter : @resource).send(attr)
  end
end
# Rest of file

2.12 Run the tests (GREEN) \o/

One more time, just one more time, please...

rspec

Output:

Finished in 0.01382 seconds (files took 1.28 seconds to load)
2 examples, 0 failures

And yes, it works! We implemented the Attributes class the way we wanted it and we did it in the TDD way. Congratulations!

A quick break

The bad news is that we still have 8 classes to write for Yumi... And that means we cannot use TDD to write them all because it would take forever.

Honestly, this jutsu grew way bigger than I thought (7000+ words...), actually way too big to stay interesting. So I decided to extract all the code and only explain what each class does while adding links to the GitHub repository. Feel free to check the files and read the comments in each one of them.

2. The Links presenter

Like the Attributes class, the Links class only takes a hash named options in parameters. But this hash contains all the data needed to build the links for our resources.

From the given options hash, we will extract:

• The base URL
• The links wanted for this resource
• The plural name of the resource
• The actual resource

With those, we will generate the links that could look something like this:

{ self: 'http://localhost:9393/posts/1' }

Code: Links Class Tests: Links Class Tests

3. The IncludedResources presenter

The IncludedResources class is responsible for generating the array of included resources that will be available at the top-level of our JSON document. To do so, it will need:

• The base URL
• The resource
• The resource relationships defined in the presenter

The instances of this class use a hash named included_resources to keep the resources in the returned array unique. The key for each resource is the association of its type and its id so we're sure we cannot have duplicates.

Code: IncludedResources Class Tests: IncludedResources Class Tests

4. The Relationships presenter

The Relationships class will call the as_relationship method of each related presenter.

• The base URL
• The resource
• The resource plural name
• The resource relationships defined in the presenter

Code: Relationships Class Tests: Relationships Class Tests

5. The Data presenter

The Data class is responsible for generating the main part of our JSON document. It generates the resource data hash that contains its type, its id, its attributes, its links and its relationships.

To do this, it needs the following:

• The base URL
• The resource
• The resource plural name

This class is pretty simple in itself and will call other presenters (Attributes, Links, Relationships) to get its job done.

Code: Data Class Tests: Data Class Tests

6. The Resource presenter

The Resource class is also pretty simple. Its responsibility? Check if the given resource is a collection or a single resource and call the Data presenter in the right way.

To do this, it just needs the resource.

Code: Resource Class Tests: Resource Class Tests

7. The ClassMethods module

This is not actually a class. It's a small module that will give us those neat methods to call in our presenters and allow us to write a presenter liks this:

module Presenters
  class Post < ::Yumi::Base
    meta META_DATA
    attributes :slug, :title, :content
    has_many :tags, :comments
    links :self
  end
end

See the meta, attributes, has_many and links? ;)

We are going to use the extend keyword in the Base class to load the module methods as class methods.

Code: ClassMethods Class Tests: ClassMethods Class Tests

8. The Base class

And the last one! This is the actual base class from which our future presenters will inherit. It contains the public API of our library with the 3 methods as_json_api, as_relationship and as_included.

It takes 3 parameters which are:

• The base URL, needed to build the link
• The resource
• An optional prefix when instantiating a presenter to call the as_relationship method

This class is responsible for creating the options hash that will be passed around and that contains all the required variable for each other class. After that, its job is just to call the appropriate class and define the top-level layout of the JSON document.

Code: Base Class Tests: Base Class Tests

Retrospective

Wow, we're finally done. That was a long jutsu. I hope you learned a few things while reading, if not please let me know.

Now, as promised, I want to share how to improve this little library. Since we only built what we needed for the rest of this jutsu series, it's kind of lacking in a few areas.

Here are a few improvement ideas for you:

• Extracting Yumi into a gem
• Adding the belongs_to relationship
• Adding support for pagination links

Anyway, we did it! Good luck if you add more features.

Yumi is done! Long live Yumi.

Source Code

The source code is available on GitHub.