Janos Mucsi's Got Ruby?

We need to write a test to detect possible memory leaks in some Rails code. The test itself is written in Ruby.

Testing GETs

I simply put the URLs in an array:

uris = %w(/admin
/users)

Then I make GET calls using the net API that comes with Ruby:

uris.each do |uri|
  Net::HTTP.get(uri)
end

To determine the virtual memory used by Rails, execute:

memory_check = `ps -eo vsz,args | grep script/server`
m = memory_check.match(/(\d+)/)
virtual = m[0].to_i

If you don’t manually start the server, replace the string after grep with
the name of your process.

Testing POSTs

Operations that modify data are usually more involved to set up because the POST
data is sent in the body of the request and also because there may be constraints
in the database. We, therefore, modify the datastructure to include the POST data
separately from the URI:

uris = ["/admin",
        ["/admin/users", {"client_id" => "387", "email"=>"test@mycompany.com", "permissions_select"=>"standard" },
        Proc.new { |hash| hash["email"] = "test#{Time.now.to_i.to_s}@mycompany.com"} ]]

We have an Array now. The first element will be still the URI. The second element contains
the submit data – Rails style. Finally, we specify a Proc object to modify the data
in a way that lets us send the request multiple times without getting constraint or
validation errors. The example Proc above ensures that each email is unique by using
the current time to build it.
Then in the code we inspect the first element of each URI to detect whether it’s a GET
or a POST:

uris.each do |uri|
  post = false
  case uri
    when String
      full_uri = uri
    when Array
      # POST with a hash of values
      full_uri = uri[0]
      post = true
  end
  # test code here
end

Before submitting, we check the post flag and act accordingly. We invoke the supplied
Proc object passing in the Hash containing the POST data for modification:

url = URI.parse full_uri
  Net::HTTP.get(url) if !post
  if post
    # let's customize
    uri[2].call(uri[1])
    Net::HTTP.post_form(url, uri[1])
  end
end

Hitting each URL multiple times

The memory test is only realistic with many hits. For this, we introduce a variable
called @how_many into our mini-framework that the caller can set.

uris = [ "/admin" ]
module Net
  class HTTP
    class << self

      attr_accessor :prefix
      attr_accessor :how_many

      def fetch(uris)
        data = []

        uris.each do |uri|
          post = false
          case uri
            when String
              full_uri = prefix + uri
            when Array
              # POST with a hash of values
              full_uri = prefix + uri[0]
              post = true
          end

          url = URI.parse full_uri
          @how_many.times do
            Net::HTTP.get(url) if !post
            if post
              # let's customize
              uri[2].call(uri[1])
              Net::HTTP.post_form(url, uri[1])
            end

            memory_check = `ps -eo vsz,args | grep script/server`
            m = memory_check.match(/(\d+)/)
            virtual = m[0].to_i
          end
        end
      end
    end
  end
end
Net::HTTP.how_many = 50
Net::HTTP.prefix = 'http://localhost:3000'
Net::HTTP.fetch(uris)

We’ve also added a prefix to the URI so that we can test against any host/port
combination.

Data collection and graph

We use an array to store the memory used after each hit.

def fetch(uris)
  data = []
  # processing goes here
  memory_check = `ps -eo vsz,args | grep script/server`
  m = memory_check.match(/(\d+)/)
  virtual = m[0].to_i
  data << virtual
  # more processing
end

Once we looped through all URIs and hit them as many times as specified, we have an array that we
can use the draw a graph. gruff is well-suited for this purpose. First we need to
install ImageMagick, then we implement a module to encapsulate graphing code:

require 'gruff'

module Net
  module Grapher

    def draw_graph(title, url_count, times, data)
      puts 'drawing graph!'
      g = Gruff::Line.new
      g.title = title

      g.data("#{url_count.to_s}/#{times.to_s}", data)

      g.labels = {0 => '0', 40 => '40', data.size => "#{data.size}"}

      file_name = "#{title.gsub(/\//, '')}_#{url_count.to_s}_#{times.to_s}.png"
      File.delete(file_name) if File.exists? file_name
      g.write(file_name)
    end
  end
  class HTTP
    extend Grapher
    # more code here
  end
end

The draw_grap method is available now. It takes a title, and some statistical information.
The last argument is the Array itself containing the data. Since the number of hits
and the number of URIs is important to evaluate the graph, we use that information
to construct the name of the file name:

file_name = "#{title.gsub(/\//, '')}_#{url_count.to_s}_#{times.to_s}.png"

Running the test

Let’s repeat the code in its entirety:

#!/usr/local/bin/ruby
require 'gruff'

uris = ["/admin",
        ["/admin/projects", {"admin_project[change_id]"=>"0",
                              "admin_project[new_project_support_contact_attributes][][work_phone]"=>"",
                              "admin_project[new_project_support_contact_attributes][][last_name]"=>"",
                              "admin_project[new_project_support_contact_attributes][][first_name]"=>"",
                              "admin_project[new_project_support_contact_attributes][][mobile_phone]"=>"209-346-4294",
                              "admin_project[client_id]"=>"387",
                              "admin_project[description]"=>"",
                              "admin_project[project_name]"=>"This is test4448",
                              "commit"=>"Save",
                              "client_id"=>"387"}, Proc.new { |hash| hash["admin_project[project_name]"] = "This is test#{Time.now.to_i.to_s}" } ]
]

module Net
  module Grapher

    def draw_graph(title, url_count, times, data)
      puts 'drawing graph!'
      g = Gruff::Line.new
      g.title = title

      g.data("#{url_count.to_s}/#{times.to_s}", data)

      g.labels = {0 => '0', 40 => '40', data.size => "#{data.size}"}

      file_name = "#{title.gsub(/\//, '')}_#{url_count.to_s}_#{times.to_s}.png"
      File.delete(file_name) if File.exists? file_name
      g.write(file_name)
    end
  end
  class HTTP
    extend Grapher
    class << self

      attr_accessor :prefix
      attr_accessor :how_many

      def fetch(uris)
        data = []

        uris.each do |uri|
          post = false
          case uri
            when String
              full_uri = prefix + uri
            when Array
              # POST with a hash of values
              full_uri = prefix + uri[0]
              post = true
          end

          url = URI.parse full_uri
          @how_many.times do
            Net::HTTP.get(url) if !post
            if post
              # let's customize
              uri[2].call(uri[1])
              Net::HTTP.post_form(url, uri[1])
            end

            memory_check = `ps -eo vsz,args | grep script/server`
            m = memory_check.match(/(\d+)/)
            virtual = m[0].to_i
            data << virtual
          end
        end

        draw_graph(uris[0], uris.size, @how_many, data)
      end
    end
  end
end

Net::HTTP.how_many = 50
Net::HTTP.prefix = 'http://localhost:3000'
Net::HTTP.fetch(uris)

Make sure the file has the x flag set so it’s executable. If we save the code to
a file called test_me.rb and run ./test_me.rb, we get a nice graph like this:

Evaluation

This test was run on a Linux machine with 4G of memory. We hit 5 URLs, each a 100 times. The difference between the highest
and lowest memory usage is less than 4K so we see a nice garbage collection pattern.
No memory leak so far!

How do we test a filter in ApplicationController? We want to bypass all the logic in the controllers that subclass it,
but we still want a realistic test with a request and a response. Such tests are
called functional tests in the Rails world.

Setting up the test

Our ApplicationController that has the filter is typical, with a before_filter:

class ApplicationController < ActionController::Base
  before_filter :check_logged_on
..
end

We create a test in test/functional/application_controller_test.rb:

require File.dirname(__FILE__) + '/../test_helper'

class FooControllerTest < ActionController::TestCase
  def setup
  end

  def test_before_filter
  end
end

We do need a real controller for a real test, but it can not be any of the existing ones. So let’s create a controller on the fly:

class FooController < ApplicationController
    def index
    render :text => "index called"
  end
end

For simplicity, we just render text, so we don’t have to create a view.
Now we can write the setup method:

class FooControllerTest < ActionController::TestCase
  def setup
    @controller = FooController.new
    @request    = ActionController::TestRequest.new
    @response   = ActionController::TestResponse.new
    ActionController::Routing::Routes.draw do |map|
      map.resources :foo
    end
  end
end

We needed to add the routing code, so we can hit the controller. Now we are ready to test something.
Let’s say, the filter renders an error if the user is not logged in:

class ApplicationController < ActionController::Base
  def check_logged_on
      unless @logged_on
        render :status => 500
      end
  end
  # more code here
end

Of course, in the real world, this method would be more complicated, but this example illustrates the point. Our first test looks like this:

class FooControllerTest < ActionController::TestCase
  def setup
    @controller = FooController.new
    @request    = ActionController::TestRequest.new
    @response   = ActionController::TestResponse.new
    ActionController::Routing::Routes.draw do |map|
      map.resources :foo
    end
  end

  def test_bypass
    @controller.logged_on = true
    get :index
    assert_response :success
  end
end

For this to work, we need an accessor in FooController:

class FooController < ApplicationController
  prepend_before_filter :check_logged_on
  attr_accessor :logged_on

  def index
    render :text => "index called"
  end
end

If you run this, you will get success because FooController#index has been called and by default the status code returned is 200 which is :success.
The second test, will run the full code in the filter:

def test_render_error
  @controller.logged_on = false
  get :index
  assert_response :error
end

In this case the code under unless will execute and FooController#index will not be called. The returned status code will correspond to :error.

Running the test

To run the test, type the following from command line and you should get no errors:

janos@janos-laptop:/work/myapp$ rake test:functionals TEST=test/functional/application_controller_test.rb
(in /work/myapp)
Loaded suite /usr/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake/rake_test_loader
Started
..
Finished in 0.23019 seconds.

2 tests, 2 assertions, 0 failures, 0 errors

Finally they put up a video…

…and a test test drive

I am pleased to announce that after six months of development our successful project myfitv.com is live. You can see recommendations:

Find out what’s on:

…and of course, you can watch a show, a movie or just a clip:

We have seen the documentation on Rails caching. So the real question about caching, then, is where to put the code? Caching is a crosscutting concern, it can occur anywhere in the code, because anything can be cached. It does not go well with other modules. Caching code stays separate.

Almost all calls end up in ActiveRecord::Base. There is also first, last, but let’s focus on find for now. All queries go through Base.find – a class method.
We need to alias it so we can do more. The new method will be called find, but not before the old method is renamed to ‘old_find’.

alias old_find find

Nobody will know about the new find method, and that’s the right thing to do. When it’s called, it will check the cache to see whether the requested thing is there. If it’s there, it will return the value and never call the original cache. If there is a miss, the requested item is not there, the find method will call old_find. If old_find returns a value, it will cache that value.

value=Rails.cache.read(key)
 if !value
    value=old_find(*args)
     Rails.cache.write(key, value, :expires_in => 1.hours) if value
end

How do I generate the key? If we assume that all queries return different results, then one way to go is to use the method arguments. I will write a method like this:

def generate_key(*args)
    key="#{self.name}#{args.join}"
    key
end

I made it two lines for illustration so that you can put a breakpoint there and inspect the value in the debugger. I also use the name of the class since it’s possible to have a Video with id=94 and the Customer with id=94.

Preloading

You might be out of luck when referencing a cached object’s collections. This is because the connection to the database may be closed since the ‘parent’ object was cached. So if your videos have actors, for example, you might just want to call actors on your video to force the proxy to load the actors. Since finders either return a single object or multiple objects in an Array, I need to handle that:

def preload(result)
   if result.class==Video
      result.actors
      result.images
   elsif result.class==Array
      result.each do |video|
         video.actors
         video.images
     end
   end
end

Notice how you can do this for multiple collections.
Let’s put this all together:

if ENV['RAILS_ENV']=='development'
  RAILS_DEFAULT_LOGGER.debug "environment.rb: Enabling finder interceptor for Videos."
  class ActiveRecord::Base
     class << self
        alias old_find find
        def find(*args)
          begin
            #if defined?(@cache_finds) and @cache_finds
            if self.name=='Video'
              key=generate_key(*args)
              value=Rails.cache.read(key)
              if value
                logger.debug ">>>>>>>>> Returning "<<key<<" from cache."
              end
              if !value
                value=old_find(*args)
                preload value
                Rails.cache.write(key, value, :expires_in => 1.hours) if value
              end
            else
              value=old_find(*args)
            end
            value
          rescue Exception => e
              logger.error e
          end
        end

        def generate_key(*args)
          key="#{self.name}#{args.join}"
          key
        end

        def preload(result)
          if result.class==Video
            result.series
            result.images
          elsif result.class==Array
            result.each do |video|
              video.actors
              video.images
            end
          end
        end
      end
  end
end

Put this code in an appropriate location in Rails.
The beauty of the solution is that it keeps all other parts of the code clean – it is non-invasive. You can remove it, without affecting other places in the code. You can use any caching technology such as Redis, if you like. Or just use the MemoryStore that comes with Rails.

If you installed mySQL with ports, do the following:

cd /Library/LaunchDaemons
sudo launchctl unload -w org.macports.mysql5.plist
sudo launchctl load -w org.macports.mysql5.plist

It’s often necessary to paginate in Ajax applications. For example, let’s say we have a tool for editors that lets them search for videos. The search is performed in one area and in an other area the user selects the page he wants to populate. Then he can just drag the chosen videos over the selected page. Full-page reload on pagination would be a problem, because the user would have to find the page he wants to edit again. We need to to able to paginate within the search results while leaving the rest of the page intact.

I did not want to implement pagination on the back end myself, so I took advantage of the mislaw-will_paginate gem.
The first task was to capture the pagination links. We render the links into a local variable like this:

@videos=Video.paginate :page=>params[:page]
pagination=ActionView::InlineTemplate.new("<%= will_paginate @videos %>", nil).render(@template, {})

The code to query the database would typically be more complicated, there would be chained named scopes, for example, and the call to ‘paginate’ would come last:

@videos=Video.most_popular.paginate :page=>params[:page]

We use JSON to communicate between the browser and the server. The video results are in an array generated by ActiveRecord. Let’s add the pagination string in the first position in the array:

@videos.unshift pagination # let's stick the pagination information in the first position
render({ :json => @videos.to_json(..) })

This does not disrupt the json encoding, the links stay a string while the rest of the array’s members get transformed according to the rules given in the to_json method.

On the JavaScript side, assuming that I have a div somewhere that looks like this:

<div id="pagination">
</div>

I can take resolve the pagination information and stick it to the div:

function onSearchResultsReady(results) {
  var pagination=results.shift();
  //processing here
  // ..
  $('#pagination').append(pagination);
  ajaxifyLinks();
}

The final step is to prevent the browser from reloading the page so that it instead makes an Ajax call when the user clicks on a page link. Notice the call to ajaxifyLinks. That function overrides the links’ default behavior:

function ajaxifyLinks() {
    $('#pagination a').click (function(){
        $.getJSON(this.href,
          {},
          onSearchResultsReady);
        return false; // don't follow the link
    });
  }

With minimal amount of work, we were able to make the gem do all the heavy-lifting, and even got our ajaxified pagination as well:

Autotest is a convenient way to keep your tests passing without having to continuously issue the rake rspec commands.
To use autotest, create a Rails project if you have not done so already:

rails huhu

cd into the new directory and enable RSpec:

./script/generate rspec

Now generate two RSpec-enabled models:

./script/generate rspec_model book title:string
./script/generate rspec_model author full_name:string

Notice how now you have two tests in spec/models.
Before running the tests, make sure your test database is ready:

rake db:migrate RAILS_ENV=test

Now from the root directory of the Rails application, execute:

./script/autospec

You should see all tests passing. Observe that the script does not exit.
Open a second shell and add a test like the following to spec/models/author_spec.rb:

it "should be stuff" do
    false.should== true
  end

Obviously, you would write a meaningful test but it illustrates the point. As soon as you save the file, you will see autotest execute in the first shell window. Here is the relevant output:

'Author should be stuff' FAILED
expected: true,
     got: false (using ==)
./spec/models/author_spec.rb:15:

Finished in 0.087639 seconds

2 examples, 1 failure

Now correct the test so that the offending line reads:

false.should==false

You’ll see how all the tests are rerun and there are no failures.
Edit the other test file, book_spec.rb, and add a passing test. You will see autotest execute only the tests in book_spec.rb.
However, if you add a failing test to book_spec.rb, then you correct it, you will see that this time autotest first executes the tests in book_spec.rb to check whether they all pass, then it goes on to execute all tests in all spec files.

It takes a little time to get used to the work style autotest offers, but it’s well worth the peace of mind to know that all your tests are passing.

These notes describe how to install Passenger in a typical environment.
First get the gem:

sudo gem install  passenger

Run the installer following its intructions:

sudo apt-get install apache2-prefork-dev
sudo passenger-install-apache2-module

The installer now will build and install the Apache module.

Once, that’s done, add the snippet given by the installer to /etc/apache2/apache2.conf. For example, it might look like this:

LoadModule passenger_module /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7/ext/apache2/mod_passenger.so
PassengerRoot /usr/lib/ruby/gems/1.8/gems/passenger-2.2.7
PassengerRuby /usr/bin/ruby1.8

Let’s say we want to make the application available on port 3000. Add the port to /etc/apache2/ports.conf:

NameVirtualHost *:3000
Listen 3000

Create a file in /etc/apache2/sites-available choosing any name for it. In this example, let’s call it rails:

<VirtualHost *:3000>
  ServerName myserver.thruhere.net:3000
  CustomLog /var/www/ruby/log/access_log common
  ErrorLog /var/www/ruby/log/error_log
  DocumentRoot /somewhere/myrailsapp/public
</VirtualHost>

Notice how the port matches the entry in ports.conf.
Now go to /somewhere/ and type rails myrailsapp. Once the application is generated, restart Apache:

sudo /etc/init.d/apache2 restart

Now if you go to the url specified in the file rails, you should see the Rails welcome page:

http://myserver.thruhere.net:3000

Enable url rewriting like this:

sudo a2enmod rewrite

Then restart Apache:

sudo /etc/init.d/apache2 restart

In the file /etc/apache2/sites-available/default, in the section <Directory /var/www> make sure that ‘AllowOverride All’ appears:

<Directory /var/www/>
          Options Indexes FollowSymLinks MultiViews
           AllowOverride All
           Order allow,deny
           allow from all
</Directory>