rb_iterate replaced by rb_block_call API between Ruby 1.8 and 1.9

You may have noticed that some Ruby libraries using C extensions were suddenly having core dumps when compiled using Ruby 1.9, whereas compiling with Ruby 1.8 got no problems.
A good culprit might be the usage of rb_iterate method in the C extension: rb_iterate API has been deprecated and changed since Ruby 1.9, in favor of rb_block_call. In fact this new method is more intuitive and easy to use than its predecessor.
Although I have no problems with it being deprecated, I think that breaking its behavior was a rough move taken by Ruby developers.

I will illustrate the differences using the following code, written in Ruby, then in C using Ruby 1.8 rb_iterate API, and in C using Ruby 1.9 rb_block_call API:

# object, object_param and context_var are already initialized as 3 Ruby objects
object.method(object_param) do |yield_param|
  # Access yield_param and context_var
end
// iterate_args: Ruby Array containing arguments given as 2nd parameters of rb_iterate call
VALUE call_method_from_iterate(VALUE iterate_args) {
  VALUE object = rb_ary_entry(iterate_args, 0);
  VALUE object_param = rb_ary_entry(iterate_args, 1);
  return rb_funcall(object, rb_intern("method"), 1, object_param);
}

// yield_args: Ruby Array containing arguments given by the yield call (implemented in the "method" method of "object")
// context_args: Ruby Array containing arguments given as 4th parameters of rb_iterate call
VALUE yielded_block(VALUE yield_args, VALUE context_args) {
  VALUE yield_param = rb_ary_entry(yield_args, 0);
  VALUE context_var = rb_ary_entry(context_args, 0);
  // Access yield_param and context_var
}

// object, object_param and context_var are already initialized as VALUE
rb_iterate(
  call_method_from_iterate, // C method to be called once yield is ready to be used
  rb_ary_new3(2, // Ruby Array giving parameters to the previously defined C method
    object,
    object_param),
  yielded_block, // C method to be called when "yield" is called
  rb_ary_new3(1, // Ruby Array that will also be given to the yielded method: pass context parameters here
    context_var)
);
// yielded_object: First argument given to the yield call (implemented in the "method" method of "object"). Same as yield_argv[0].
// context_args: Ruby Array containing arguments given as 4th parameters of rb_iterate call
// yield_argc: Number of arguments given by the yield call
// yield_argv: C Array containing arguments given by the yield call
VALUE yielded_block(VALUE yielded_object, VALUE context_args, int yield_argc, VALUE yield_argv[]) {
  VALUE yield_param = yield_argv[0]; // Could be "= yielded_object" too
  VALUE context_var = rb_ary_entry(context_args, 0);
  // Access yield_param and context_var
}

// object, object_param and context_var are already initialized as VALUE
VALUE method_args[1];
method_args[0] = object_param;
rb_block_call(
  object, // Object to call the method on
  rb_intern("method"), // Method ID to be called on the object, once yield is ready to be used
  1, // Number of arguments given to the method
  method_args, // C Array of arguments given to the method
  RUBY_METHOD_FUNC(yielded_block), // C method to be called when "yield" is called
  rb_ary_new3(1, // Ruby Array that will also be given to the yielded method: pass context parameters here
    context_var)
);

We can see that using rb_block_call spares an extra method definition, and is more readable.

C, Howto, Ruby , , , , , , , , , , ,

Code surprise for today

A little code surprise today for all my beloved readers 🙂
You find out what it does 😉

Hint: executing it might help and is harmless

Proc.new {
def d(s) $stdout << s << "\r"; sleep 0.05 end
m, r, o = 37, (65..90), 0
s = ' '*m
z = [ [ lambda{|x| x*(x*(x*(x - 306) + 34953) - 1766360) + 33321600}, 1, 0, [2, 3], 4 ],
  [ lambda{|x| x*(x*(x - 234) + 18171) - 468234}, 1, 0, 2 ],
  [ lambda{|x| x*(x*(x*(x - 305) + 34697) - 1744867) + 32731530}, 2, 1, 3, 0 ]
].each do | f, *l |
  n = 0
  r.each_with_index do |c, i|
    if (f.call(c) == 0)
      [l[n]].to_a.flatten.each do |p|
        t, k = nil, 0
        i.step(p,(p-i)/(p-i).abs) do |j|
          s[o+k] = t if !t.nil?
          t, k = s[o+j], j
          s[o+j] = c.chr
          d s
        end
      end
      n += 1
    else
      s[o+i] = c.chr.downcase if s[o+i].ord < 65
    end
    d s
  end
  o += l.flatten.size + 1
  s[o-1..m-1] = ' '*(m-o+1)
  d s
end
$stdout << "\n" << m*s.size+(o+o+z[0][3][1]*z[0][4])*z.flatten.size+(z[2][2].to_s+z[0][4].to_s).to_i << "!\n"
}.call

Enjoy, with all my best wishes for this year!

Muriel

My life, Ruby , , , , ,

Migrating big applications from Rails 2 to Rails 3

Today will have a rather long and technical post, focused on all the steps needed for migrating big applications from Rails 2 to Rails 3.

First question: why bother?
Rails 2 is great, but Rails 3 is awesome.
Apart from all new goodies given by Rails 3 (assets, increased security, bundler…), more and more Rails libraries ship only for Rails 3.
Not upgrading is no longer an option for an application that is bound to be maintained in the long run.

I got a bunch of Rails 2 applications, and when reading on-line resources, I got strong confidence: upgrading from Rails 2 to 3 is easy! Right?

Well, considering your Rails 2 application kept the simplicity of the default Rails’ blog example, yes: it will be easy.
For real world applications, situation is a bit more complex.

So this post tries to enumerate all the steps I had to take, and all the problems I got when migrating big Rails applications from 2.3.x to 3.2.9.

If not already the case, make sure your application is currently running using the latest version of Rails 2.3.x. If not, migrate first to this version.
Then there are some steps you can perform while your application is still using Rails2: details here. This will greatly simplify your migration afterwards.

  1. First steps: Rails 3 and upgrade setup
  2. Upgrade basics
    1. Check points to upgrade
    2. Generate Rails3 files
    3. Merge files with Rails2 backups
    4. Session secret setting
    5. Cookie verifier secret
    6. Routing
    7. Bundler used to handle dependencies
    8. Test helpers path
    9. Application configuration
    10. filter_parameter_logging deprecated
    11. Block helpers in ERB
    12. Deprecated RAILS_ENV, RAILS_ROOT, RAILS_DEFAULT_LOGGER
    13. Remove useless migration plugin
    14. Remove useless .rails2 files
  3. Other tasks, not as straightforward
    1. Use raw to avoid escaping HTML characters in views
    2. Use assets to serve your public images, stylesheets, javascript files
    3. No more stylesheets controllers to generate CSS files
    4. Writing helpers using concat does not work anymore
    5. Changing table names for models has changed
    6. Recognizing routes has changed
    7. Rails3 declare more MIME types
    8. Accessing current controller from the environment has changed
    9. Migrate to Devise gem version >= 2.0
    10. Use Prototype library
    11. link_to_function API changed and is deprecated
    12. Models’ attributes can’t be mass assigned anymore by default
    13. Form error messages don’t work anymore
    14. The Mailer interface has changed

First steps: Rails 3 and upgrade setup

First thing to do is to install Rails 3.

> gem install rails

Rails developers have provided a little helper script that will diagnose your Rails 2 application and tell you what files need attention.
This script is installed this way:

> ruby script/plugin install git://github.com/rails/rails_upgrade.git
Initialized empty Git repository in /path/to/railsapp/vendor/plugins/rails_upgrade/.git/
remote: Counting objects: 27, done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 27 (delta 3), reused 20 (delta 3)
Unpacking objects: 100% (27/27), done.
From git://github.com/rails/rails_upgrade
 * branch            HEAD       -> FETCH_HEAD
Thanks for installing the Rails upgrade plugin.  This is a set of generators and analysis tools to help you upgrade your application to Rails 3.  It consists of three tasks...

To get a feel for what you'll need to change to get your app running, run the application analysis:

    rake rails:upgrade:check

This should give you an idea of the manual changes that need to be done, but you'll probably want to upgrade some of those automatically.  The fastest way to do this is to run 'rails .', which will simply generate a new app on top of your existing code.  But this generation also has the effect of replacing some existing files, some of which you might not want to replace.  To back those up, first run:

    rake rails:upgrade:backup

That will backup files you've probably edited that will be replaced in the upgrade; if you finish the upgrade and find that you don't need the old copies, just delete them.  Otherwise, copy their contents back into the new files or run one of the following upgraders...

Routes upgrader
===============

To generate a new routes file from your existing routes file, simply run the following Rake task:

    rake rails:upgrade:routes

This will output a new routes file that you can copy and paste or pipe into a new, Rails 3 compatible config/routes.rb.

Gemfile generator
=================

Creating a new Gemfile is as simple as running:

    rake rails:upgrade:gems

This task will extract your config.gem calls and generate code you can put into a bundler compatible Gemfile.

Configuration generator
=======================

Much of the configuration information that lived in environment.rb now belongs in a new file named config/application.rb; use the following task to generate code you can put into config/application.rb from your existing config/environment.rb:

    rake rails:upgrade:configuration

Now we are ready to use all those goodies to help us upgrading our application.

Upgrade basics

Check points to upgrade

As the upgrade script told us, let’s start with checking what needs immediate attention.

> rake rails:upgrade:check
Deprecated session secret setting
Previously, session secret was set directly on ActionController::Base; it's now config.secret_token.
More information: http://lindsaar.net/2010/4/7/rails_3_session_secret_and_session_store

The culprits: 
        - config/initializers/session_store.rb

Old router API
The router API has totally changed.
More information: http://yehudakatz.com/2009/12/26/the-rails-3-router-rack-it-up/

The culprits: 
        - config/routes.rb

Deprecated test_help path
You now must require 'rails/test_help' not just 'test_help'.
More information: http://weblog.rubyonrails.org/2009/9/1/gem-packaging-best-practices

The culprits: 
        - test/test_helper.rb

Deprecated filter_parameter_logging calls
The list of filtered parameters are now stored in /config/application.rb. For example: config.filter_parameters += [:password]
More information: http://de.asciicasts.com/episodes/224-controller-in-rails-3

The culprits: 
        - app/controllers/application_controller.rb

New file needed: config/application.rb
You need to add a config/application.rb.
More information: http://omgbloglol.com/post/353978923/the-path-to-rails-3-approaching-the-upgrade

The culprits: 
        - config/application.rb

Deprecated ERb helper calls
Block helpers that use concat (e.g., form_for) should use <%= instead of <%.  The current form will continue to work for now, but you will get deprecation warnings since this form will go away in the future.
More information: http://weblog.rubyonrails.org/

The culprits: 
        - app/views/activities/_form.html.erb
        - app/views/user_carts/_checkout_forms.html.erb

Deprecated constant(s)
Constants like RAILS_ENV, RAILS_ROOT, and RAILS_DEFAULT_LOGGER are now deprecated.
More information: http://litanyagainstfear.com/blog/2010/02/03/the-rails-module/

The culprits: 
        - app/controllers/application_controller.rb
        - app/helpers/application_helper.rb

Wow! This is quite a big list of culprits.
Let’s backup some files before updating them. Here again, a helper script comes in handy.

> rake rails:upgrade:backup

* backing up .gitignore to .gitignore.rails2
* backing up app/controllers/application_controller.rb to app/controllers/application_controller.rb.rails2
* backing up app/helpers/application_helper.rb to app/helpers/application_helper.rb.rails2
* backing up config/routes.rb to config/routes.rb.rails2
* backing up config/environment.rb to config/environment.rb.rails2
* backing up config/environments/development.rb to config/environments/development.rb.rails2
* backing up config/environments/production.rb to config/environments/production.rb.rails2
* backing up config/database.yml to config/database.yml.rails2
* backing up doc/README_FOR_APP to doc/README_FOR_APP.rails2
* backing up test/test_helper.rb to test/test_helper.rb.rails2

This is a list of the files analyzed and backed up (if they existed);
you will probably not want the generator to replace them since
you probably modified them (but now they're safe if you accidentally do!).

- .gitignore
- app/controllers/application_controller.rb
- app/helpers/application_helper.rb
- config/routes.rb
- config/environment.rb
- config/environments/development.rb
- config/environments/production.rb
- config/environments/staging.rb
- config/database.yml
- config.ru
- doc/README_FOR_APP
- test/test_helper.rb

Generate Rails3 files

If you didn’t backup your files previously (using rake rails:upgrade:backup or other means), be careful when conflicts are found during the new files generation.

Generating Rails3 files is done this way in your application’s directory:

> rails new .
       exist
      create  README.rdoc
    conflict  Rakefile
Overwrite /path/to/railsapp/Rakefile? (enter "h" for help) [Ynaqdh] Y
       force  Rakefile
      create  config.ru
...
Using rails (3.2.9)
Using sass (3.2.3)
Using sass-rails (3.2.5)
Using sqlite3 (1.3.6)
Using uglifier (1.3.0)
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Merge files with Rails2 backups

The backup step has created a bunch of .rails2 files everywhere in your application.
You have to review each one of them, and port their content into their corresponding Rails3 migrated file.

For example: port the content of app/controllers/application_controller.rails2.rb into app/controllers/application_controller.rb, respecting the structure of the newly generated file.

Part of the merging conflicts is covered in the following sections.

Session secret setting

Use Railsapp::Application.config.session_store instead of ActionController::Base.session to store the session store key, and eventually move the secret to an extra file. Details are given here.

# Be sure to restart your server when you modify this file.

# Rails 2 - Obsolete
# ActionController::Base.session = { :key => ‘mykeysession’, :secret => ‘somereallylongrandomkey’ }
# Rails 3
Railsapp::Application.config.session_store :active_record_store, :key => 'mykeysession'

The cookie verifier secret is usually set in file config/initializers/cookie_verification_secret.rb. The way to set it has changed:

# Rails2
# ActionController::Base.cookie_verifier_secret = 'A BIG STRING';
# Rails3
ActionController::Base.config.secret_token = 'A BIG STRING';

Routing

Routing format has completely changed, but here comes a little helper script in handy to migrate most of them:

> rake rails:upgrade:routes >config/routes.rails3.rb

Then, you can review your new Routes file, and replace the content of config/routes.rb with the one of config/routes.rails3.rb.
Details of these modifications can be found here.

Here is a little example of a migrated routes.rb file:

Railsapp::Application.routes.draw do
  
  # Resources
  resources :my_resources
  match 'resources/feed' => 'resources#feed', :as => :feed_resources, :format => 'atom'

  resources :items do
    member do
      get :display
    end
  end

  # Statistics
  match 'statsdisplay' => 'stats#display', :as => :statsdisplay

  # User carts
  resources :user_carts do
    resources :cart_contents
  end

  # Users
  devise_for :users, :controllers => { :registrations => 'registrations' }
  resources :users do
    resources :user_event_subscriptions
    resources :user_carts
    member do
      get :edit_external_password
      get :current_cart
    end
  end

  # Static pages
  match 'live' => 'home#live', :as => :live
  root :to => 'home#index'

  # Default pages
  match '/:controller(/:action(/:id))'
  match '*path' => 'home#handle404'

end

Bundler used to handle dependencies

If you don’t know yet about bundler, take a look at this. This is the tool Rails3 now uses to handle Gem dependencies.
If you haven’t done it already, migrate your application to bundler: a little helper script does everything:

> rake rails:upgrade:gems >Gemfile

You can review your Gemfile, and add eventually add missing dependencies that were formerly declared in your config/environment* files. Here is an example:

source 'https://rubygems.org'

gem 'rails', '3.2.6'

# Bundle edge Rails instead:
# gem 'rails', :git => 'git://github.com/rails/rails.git'

gem 'sqlite3'
gem 'json'

# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails',   '~> 3.2.3'
  gem 'coffee-rails', '~> 3.2.1'

  # See https://github.com/sstephenson/execjs#readme for more supported runtimes
  gem 'therubyracer'

  gem 'uglifier', '>= 1.0.3'
end

gem 'jquery-rails'
gem 'prototype-rails'

# To use ActiveModel has_secure_password
# gem 'bcrypt-ruby', '~> 3.0.0'

# To use Jbuilder templates for JSON
# gem 'jbuilder'

# Use unicorn as the app server
# gem 'unicorn'

# Deploy with Capistrano
# gem 'capistrano'

# To use debugger
#gem 'ruby-debug19'

gem 'mongrel', '1.2.0.pre2'
gem 'nokogiri'
gem 'mechanize'
gem 'warden'
gem 'devise'
gem 'devise-encryptable'
gem 'cancan'
gem 'recaptcha', :require => 'recaptcha/rails'
gem 'friendly_id'
gem 'dynamic_form'
gem 'rUtilAnts', '>= 1.0', :require => false
gem 'rails-ajax'

group :test do

  # RSpec
  gem 'rspec'
  gem 'rspec-rails'

  # Cucumber
  gem 'cucumber-rails', :require => false
  gem 'database_cleaner'
  gem 'launchy'

  # Capybara-webkit
  gem 'capybara-webkit'
  
  # Spork
  gem 'spork-rails'

  # Guard
  gem 'guard'
  gem 'guard-cucumber'
  gem 'guard-spork'

end

Once your Gemfile has been created, you have to run bundle install to make sure all dependencies are met.

Test helpers path

Require rails/test_help instead of test_help (usually found in test/test_helper.rb).

ENV["Rails.env"] = "test"
require File.expand_path('../../config/environment', __FILE__)
require 'rails/test_help'

class ActiveSupport::TestCase
  # Setup all fixtures in test/fixtures/*.(yml|csv) for all tests in alphabetical order.
  #
  # Note: You'll currently still have to declare fixtures explicitly in integration tests
  # -- they do not yet inherit this setting
  fixtures :all

  # Add more helper methods to be used by all tests here...
end

Application configuration

Most of the application configuration has been moved from config/environment.rb to a new file called config/application.rb. A little helper script helps us generate this file:

> rake rails:upgrade:configuration >config/application.rb

Review the configuration file config/application.rb. Here is an example:

require File.expand_path('../boot', __FILE__)

require 'rails/all'

if defined?(Bundler)
  # If you precompile assets before deploying to production, use this line
  Bundler.require(*Rails.groups(:assets => %w(development test)))
  # If you want your assets lazily compiled in production, use this line
  # Bundler.require(:default, :assets, Rails.env)
end

module Railsapp
  class Application < Rails::Application
    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration should go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded.

    # Custom directories with classes and modules you want to be autoloadable.
    # config.autoload_paths += %W(#{config.root}/extras)

    # Only load the plugins named here, in the order given (default is alphabetical).
    # :all can be used as a placeholder for all plugins not explicitly named.
    # config.plugins = [ :exception_notification, :ssl_requirement, :all ]

    # Activate observers that should always be running.
    # config.active_record.observers = :cacher, :garbage_collector, :forum_observer

    # Set Time.zone default to the specified zone and make Active Record auto-convert to this zone.
    # Run "rake -D time" for a list of tasks for finding time zone names. Default is UTC.
    # config.time_zone = 'Central Time (US & Canada)'

    # The default locale is :en and all translations from config/locales/*.rb,yml are auto loaded.
    # config.i18n.load_path += Dir[Rails.root.join('my', 'locales', '*.{rb,yml}').to_s]
    # config.i18n.default_locale = :de

    # Configure the default encoding used in templates for Ruby 1.9.
    config.encoding = "utf-8"

    # Configure sensitive parameters which will be filtered from the log file.
    config.filter_parameters += [
      :password,
      :password_clear,
      :password_verify
    ]

    # Use SQL instead of Active Record's schema dumper when creating the database.
    # This is necessary if your schema can't be completely dumped by the schema dumper,
    # like if you have constraints or database-specific column types
    # config.active_record.schema_format = :sql

    # Enforce whitelist mode for mass assignment.
    # This will create an empty whitelist of attributes available for mass-assignment for all models
    # in your app. As such, your models will need to explicitly whitelist or blacklist accessible
    # parameters by using an attr_accessible or attr_protected declaration.
    # config.active_record.whitelist_attributes = true

    # Enable the asset pipeline
    config.assets.enabled = true

    # Version of your assets, change this if you want to expire all your assets
    config.assets.version = '1.0'

  end
end

filter_parameter_logging deprecated

Parameters filtered from log files are now defined in the previously generated config/application.rb file.

Delete all filter_parameter_logging calls (usually found in app/controllers/application_controller.rb), and update your config/application.rb file accordingly.

Block helpers in ERB

This is one of the most annoying changes in big projects: in ERB views, block helpers now return strings instead of silently concatenating their result to the output.

This means that using form_for, form_tag, fields_for and link_to (even in its block form) are now called using <%= instead of <%.

<%= form_for(resource) do |f| %>
  <p>
    <%= f.label :name %><br />
    <%= f.text_field :name %>
  </p>
  <p>
    <%= f.submit button_name %>
  </p>
<% end %>

<%= link_to resource_path do %>
  <img src="myimg.png"/>
<% end %>

You have to replace all former calls in all your views.

Deprecated RAILS_ENV, RAILS_ROOT, RAILS_DEFAULT_LOGGER

This one can be seen as simply as a little text replacement in your source files (details here):

Replace with:
RAILS_ENV Rails.env
RAILS_ROOT Rails.root
RAILS_DEFAULT_LOGGER Rails.logger

Remove new_rails_default

The file config/initializers/new_rails_default was intended for Rails2 to Rails3 migration preparation. Every configuration it contains is now a default in Rails3.

You can simply delete this file.

Remove useless migration plugin

The migration plugin installed at the beginning of this tutorial is inside vendor/plugins/rails_upgrade. However this location is marked as deprecated for plugins, and this directory can be removed, now that we won’t use rake rails:upgrade:* tasks anymore.

Remove useless .rails2 files

At that point, you have completed all the common migration tasks. For simple websites, that will be all.

You can now remove all .rails2 files.

Next lie extra steps that bigger websites might need to finalize their migration correctly.

Other tasks, not as straightforward

And now comes the candy: all the tasks that were not listed as part of a Rails3 migration.
Some of them might not apply to your application, as it can deal with external gems, or specific usages of the Rails API.

Use raw to avoid escaping HTML characters in views

For big projects using heavy HTML generation in views, this can be a real pain.

In Rails2, you could use view helpers with HTML characters. For example, to display an image as a link:

<%= link_to '<img src="myimg.png"/>', resource_path %>

However in Rails3 for security reasons, this will escape all HTML characters, and instead of having a link on your image, you will have a link on the text <img src="myimg.png"/>.

If you want to force keeping your string as you intended, use raw method:

<%= link_to raw('<img src="myimg.png"/>'), resource_path %>

The same goes for any Ruby string that is returned directly into a view:

<%= raw '<p>I don\'t want the p tag to be escaped, therefore I use raw.</p>' %>

Therefore the same goes for any view helper you might have defined that was bound to return HTML snippets:

def add_bold(text)
  bolded_text = "<b>#{text}</b>"
  # Rails2
  # return bolded_text
  # Rails3
  return raw bolded_text
end

Use assets to serve your public images, stylesheets, javascript files

Assets are really awesome, and you should definitely use them.
More info on assets can be found here.

Although Rails3 can continue to deal with all the files stored in the public/ folder the same way Rails2 did, I strongly recommend migrating those files to the assets pipeline.

However when doing so in production mode, Rails3 alters the name of your files served as assets. When you access them from your code, you cannot hard-code them anymore ; instead you will use the asset_path helper.

# Rails2
mycss_url = "#{ENV['RAILS_RELATIVE_URL_ROOT']}/css/mystylesheet.css"
# Rails3
mycss_url = asset_path('mystylesheet.css')

Remember to check that your assets are enabled in file config/application.rb:

    # Enable the asset pipeline
    config.assets.enabled = true
    # Version of your assets, change this if you want to expire all your assets
    config.assets.version = '1.0'

No more stylesheets controllers to generate CSS files

When your application needs CSS generation, a common trick in Rails2 was to define a stylesheets controller, and make it render a CSS file like a view.

With Rails3 you don’t need that anymore: CSS files are served using assets, and this mechanism allows CSS files to be generated using lots of different engines: SASS, LESS, ERB

Writing helpers using concat does not work anymore

If you have written view helpers that concatenate data to the output (the same way form_for did in Rails2), you certainly have used the concat methods.
In Rails3 you will have to change this behavior: your helper methods have to return the HTML strings, and not bother anymore about concatenating them to the output.

# Encapsulate an inner block into a section div
def section(&iBlock)
  inner_html = capture(&iBlock)
  # Rails2
  # concat("<div class=\"section\">#{inner_html}</div>")
  # Rails3
  return raw "<div class=\"section\">#{inner_html}</div>"
end

And this change is reflected in your views that now get the string directly from the call, using <%= section do instead of <% section do:

<%= section do %>
  <p>My section content</p>
<% end %>

Changing table names for models has changed

If you wanted your model to use a specific table name in the database, you were using set_table_name. It has been replaced with self.table_name = .

class MyModel < ActiveRecord::Base
  # Don't use the default table name
  # Rails2
  # set_table_name 'my_table_for_model'
  # Rails3
  self.table_name = 'my_table_for_model'
end

Recognizing routes has changed

In Rails2, when you wanted to recognize a path (ie. translating a URL into the hash giving controller, action), you could use ActionController::Routing::Routes.recognize_path.
In Rails3, the same call is Rails.application.routes.recognize_path.

# Rails2
# url_params = ActionController::Routing::Routes.recognize_path('/path/to/resource/15/edit')
# Rails3
url_params = Rails.application.routes.recognize_path('/path/to/resource/15/edit')

Rails3 declare more MIME types

By default, Rails2 was missing some important MIME types to be declared. You had to do it by yourself.
In Rails3, common MIME types are already declared by default.

If you were declaring MIME types by yourself, you can comment out the following ones as they have been added into Rails3:

# Rails3: Already declared MIME types
#Mime::Type.register 'image/x-windows-bmp', :bmp
#Mime::Type.register 'image/gif', :gif
#Mime::Type.register 'image/jpeg', :jpeg
#Mime::Type.register 'video/mpeg', :mpeg
#Mime::Type.register 'application/pdf', :pdf
#Mime::Type.register 'image/png', :png
#Mime::Type.register 'image/x-tiff', :tiff
#Mime::Type.register 'multipart/x-zip', :zip

Accessing current controller from the environment has changed

Some plugins require that your initializers define callbacks that will need access to the current controller. This is ugly, but unfortunately sometimes there is no choice.

In Rails2, this could be done using env['action_controller.rescue.response'].template.controller.
In Rails3, you have to use env['action_controller.instance'].

Migrate to Devise gem version >= 2.0

If you were using the Devise gem, you will have to migrate it to a version >= 2.0.
This can be quite painful, all the more so as you customized it deeply.
The best way to do it is to generate from scratch new controllers and views from Devise’ generators after having updated it. More information can be found here.

Basically, here are the points to consider when migrating Devise:

  • :http_authenticatable and :activatable modules have been removed.
  • Views files have been reorganized, in a app/views/devise/ directory (devise_mailer/, confirmations/, passwords/, registrations/, sessions/, shared/ and unlocks/ directories have been moved there).
  • Locales have been slightly changed.
  • Signing out users is done using :delete HTTP method instead of :get.
  • Users updating their email does not change them in the database, but set a unconfirmed_email column instead.
  • Some changes have to be done on the database. Here is a migration file I used for this purpose. Don’t forget to adapt it to your own needs:
    class MigrateDeviseTo204 < ActiveRecord::Migration
      def self.up
        change_table(:users) do |t|
          t.datetime :reset_password_sent_at
          t.string   :unconfirmed_email # Only if using reconfirmable
        end
        remove_column :users, :remember_token
      end
    
      def self.down
        change_table(:users) do |t|
          t.string :remember_token
        end
        remove_column :users, :reset_password_sent_at
        remove_column :users, :unconfirmed_email
      end
    end
    
  • If you were using cancan with Devise, you may have to change your app/models/ability.rb file, and add devise/ to any Devise action. Here is an example:
    # User and sessions
    can [:new, :create, :edit, :update], 'devise/passwords'.to_sym
    can [:new, :create, :show], 'devise/confirmations'.to_sym
    can [:new, :create], [
      :registrations,
      'devise/sessions'.to_sym,
      User
    ]
    can :destroy, 'devise/sessions'.to_sym
    

Use Prototype library

By default, Rails2 was packaged with Prototype library.
Rails3 ships with jQuery by default. However it is very easy to add Prototype back if you need it in your application.

First, add the prototype dependency in your Gemfile:

gem 'jquery-rails'
gem 'prototype-rails'

Run bundle install to make sure it is installed.

Then add the prototype inclusions in your asset pipeline:

//= require jquery
//= require jquery_ujs
//= require prototype
//= require_tree .

Method link_to_function from JavaScriptHelper now takes 2 mandatory arguments instead of 1, and does not accept blocks anymore.
The solution is to not use it anymore, as it is going to be deprecated.
Here is a usage example:

# Rails 2
link_to_function('link text') do |page|
  page << 'alert(\'Hello\');'
end

# Rails 3
content_tag(:a, 'link text',
  :href => '#',
  : onclick => 'alert(\'Hello\'); return false;')

Models’ attributes can’t be mass assigned anymore by default

When your models are being saved using update in your controllers, all the attributes given in the hash will be updated.
In Rails3 this is not possible anymore for security reasons: you have to declare in your Models which attributes can be updated, using the attr_accessible method:

class User < ActiveRecord::Base

  # Setup accessible (or protected) attributes for your model
  attr_accessible :email, :name

end

Form error messages don’t work anymore

In Rails2, you could call the method error_messages in your forms to display error messages. This method does not exist anymore.

<%= form_for(@message) do |f| %>
  <%= f.error_messages %>

  <% ... %>

<% end %>

Simplest way is to add it back, which can be done using the dynamic_form gem.

# ...

gem 'dynamic_form'

# ...

The Mailer interface has changed

Here are the differences:

  • Mailers are defined in the app/mailers directory instead of app/models.
  • Variables accessible in Mailer views are directly set using @my_variable instead of @body['my_variable'].
  • Sending mails is done using MyMailer.my_mail_method(my_args).deliver instead of MyMailer.deliver_my_mail_method(my_args).

That’s all for what I have found so far.
I will update this list as I find more migration steps.

Howto, Ruby on Rails, Web development , , , , , , , , , , , , ,

The Ruby C API – Basics

In my 2 previous posts, we saw how to write C extensions, and how to package them. Now time has come to see a little bit of the Ruby C API.

This post will deal with the basics, next ones will go deeper.
Normally a little C knowledge will be enough to understand most of it.

The goal of these posts is not to be exhaustive, but rather to learn quickly the most useful techniques through simple examples.

In all the examples below, I give the Ruby and C version, for you to better understand the behavior.

  1. C API usage
  2. Ruby objects
  3. Declaring modules, classes and methods
  4. Instantiating objects of a given Ruby class
  5. Calling methods of existing Ruby objects
  6. Numeric types conversions between C and Ruby
  7. Strings
  8. Arrays
  9. Hashes
  10. Next step

C API usage

The Ruby C API can be accessed through the inclusion of 1 single file: ruby.h.

#include <ruby.h>

This file declares the whole Ruby C API, types…

Then every C extension has to implement a function named Init_xxxxx (xxxxx being your extension’s name), which is being executed during the “require” of your extension.

void Init_xxxxx() {
  // Code executed during "require"
}

Ruby objects

The types system is soooooo easy: every Ruby type/class is a VALUE C type. In fact VALUE is a just a uintptr_t C type, or void* to simplify things. This is how Ruby’s loose typing is implemented in C.

There is also another type representing Ruby symbols (like :my_symbol): this is the C type ID. To get a Ruby symbol from its name (as a C string), we will use the rb_intern method:

ID sym_myclass = rb_intern("MyClass");

Therefore, each time we use the Ruby C API, we have to use VALUE types to handle Ruby objects, and ID types to handle symbols.

Then the Ruby C API defines a bunch of handful C constants defining standard Ruby objects:

C Ruby
Qnil nil
Qtrue true
Qfalse false
rb_cObject Object
rb_mKernel Kernel
rb_cString String

You guessed it: most of Ruby standard modules are named rb_mModuleName and standard classes are named rb_cClassName.

Declaring modules, classes and methods

A good example is better than thousand words.

What we want to achieve in Ruby:

module MyModule
  class MyClass
    def my_method(param1, param2, param3)
    end
  end
end

How we can get it in C:

static VALUE myclass_mymethod(
  VALUE rb_self,
  VALUE rb_param1,
  VALUE rb_param2,
  VALUE rb_param3)
{
  // Code executed when calling my_method on an object of class MyModule::MyClass
}

void Init_xxxxx()
{
  // Define a new module
  VALUE mymodule = rb_define_module("MyModule");
  // Define a new class (inheriting Object) in this module
  VALUE myclass = rb_define_class_under(mymodule, "MyClass", rb_cObject);
  // Define a method in this class, taking 3 arguments, and using the C method "myclass_method" as its body
  rb_define_method(myclass, "my_method", myclass_mymethod, 3);
}

We can notice the following:

  • The use of the VALUE type for every Ruby object (modules and classes are Ruby objects).
  • The use of rb_define_module and rb_define_class_under API methods to define new Ruby modules and classes.
  • The definition of the body of our method as a C function.
  • The argument rb_self, always put as the first C function’s argument for classes’ methods. This contains the Ruby self object.

Instantiating objects of a given Ruby class

Ruby way:

myobject = MyModule::MyClass.new
mystring = String.new("With argument")

C way:

// Get symbol for our module's name
ID sym_mymodule = rb_intern("MyModule");
// Get the module
VALUE mymodule = rb_const_get(rb_cObject, sym_mymodule);
// Get symbol for our class' name
ID sym_myclass = rb_intern("MyClass");
// Get the class
VALUE myclass = rb_const_get(mymodule, sym_myclass);
// Create a new object, using the default initializer, having 0 argument
VALUE argv[0];
VALUE myobject = rb_class_new_instance(0, argv, myclass2);

// Use String initializer with 1 argument
VALUE strargv[1];
strargv[0] = rb_str_new2("With argument");
VALUE mystring = rb_class_new_instance(1, strargv, rb_cString);

A few points to notice there:

  • Usage of rb_intern to convert C strings to Ruby symbols.
  • Usage of rb_const_get to get a Ruby constant object.
  • Usage of rb_class_new_instance, taking the number of arguments, then a C array containing arguments given to the initializer, and last the class name.
  • The use of rb_str_new2 to create a string (see further section on Strings).

Calling methods of existing Ruby objects

Ruby way:

result = myobject.my_method(nil, true, false)
puts "Hello world!"

C way:

// Get the method's symbol
ID sym_mymethod = rb_intern("my_method");
// Call the method, giving 3 parameters
VALUE result = rb_funcall(myobject, sym_mymethod, 3, Qnil, Qtrue, Qfalse);

// Get the puts method's symbol
ID sym_puts = rb_intern("puts");
// Call puts, from Kernel
rb_funcall(rb_mKernel, sym_puts, 1, rb_str_new2("Hello world!"));

Here we note:

  • The call to rb_funcall, taking the object on which the method is called, then the method’s symbol, arguments count, and arguments. Generally methods called in Ruby without specifying the class are called from Kernel (rb_mKernel) or self.

Numeric types conversions between C and Ruby

Ruby has types that translate pretty well into numerical C types, such as integers or floats. Convenient methods are given by the C API to convert one way or the other:

// rb_myint is a Ruby integer (Fixnum)
int c_myint = FIX2INT(rb_myint);
// Create a new Ruby Fixnum based on the C int
VALUE rb_mynewint = INT2FIX(c_myint + 42);

// rb_myfloat is a Ruby float (Numeric)
double c_myfloat = NUM2DBL(rb_myfloat);
// Create a new Ruby Numeric based on the C double
VALUE rb_mynewfloat = DBL2NUM(c_myfloat + 0.5);
  • FIX2INT and INT2FIX are used to convert C integers from and to Ruby Fixnum.
  • NUM2DBL and DBL2NUM are used to convert C doubles from and to Ruby Numeric.

Strings

The Ruby C API has it all to handle strings.

// rb_mystring is a Ruby String. Get the corresponding C string (! Not guaranteed to be NULL terminated)
char* c_mystring = RSTRING_PTR(rb_mystring);
int c_mystring_len = RSTRING_LEN(rb_mystring);
// Create a Ruby String from a C string (NULL terminated this time)
VALUE rb_mynewstring = rb_str_new2("This is a C string");
  • RSTRING_PTR is used to get the underlying C string: a pointer to the stored string, with no copy or memory allocation. Beware however: you can’t be rely on the NULL terminated character, first because your Ruby string can have NULL characters, and then because the Ruby C API does not guarantee it.
  • RSTRING_LEN is used to get the string’s length. This is the safest method to assume your string’s length. Do not use C method strlen as it can be confused by extra or missing NULL characters.
  • rb_str_new2 is used to create a Ruby string from a NULL terminated C string.

Arrays

Here again, every array manipulation can be done using the C API:

Ruby way:

rb_myelement = rb_myarray[3]
rb_myarray << true
rb_mynewarray = [ nil, true, false ]

C way:

// rb_myarray is a Ruby array containing at least 4 elements
int c_myarray_len = RARRAY_LEN(rb_myarray);
// Access the 4th element
VALUE rb_myelement = rb_ary_entry(rb_myarray, 3);
// Push the element "true" at the end of the array
rb_ary_push(rb_myarray, Qtrue);

// Create an array from scratch, containing nil, true, false
VALUE rb_mynewarray = rb_ary_new3(3, Qnil, Qtrue, Qfalse);
// Do the same using a C array
VALUE c_myarray[3];
c_myarray[0] = Qnil;
c_myarray[1] = Qtrue;
c_myarray[2] = Qfalse;
VALUE rb_mynewarray2 = rb_ary_new4(3, c_myarray);
  • RARRAY_LEN returns the number of elements in the array.
  • rb_ary_entry returns the element stored at the given index.
  • rb_ary_push adds an element at the end of the array.
  • rb_ary_new3 creates an array from the list of arguments.
  • rb_ary_new4 creates an array from a C array.

Hashes

Hashes are also easily accessed:

Ruby way:

rb_four = rb_myhash['foo']
rb_myhash['yop'] = 42
rb_mynewhash = {}

C way:

// rb_myhash is { 'foo' => 4, 'bar' => 2 }
// Get the value of key "foo"
VALUE rb_four = rb_hash_aref(rb_myhash, rb_str_new2("foo"));
// Add a new pair 'yop' => 42
rb_hash_aset(rb_myhash, rb_str_new2("yop"), INT2FIX(42));
// Create a Hash from scratch
VALUE rb_mynewhash = rb_hash_new();
  • rb_hash_aref returns the value indexed by a given key.
  • rb_hash_aset sets the value of a given key.
  • rb_hash_new creates a new hash.

Next step

Now we’ve seen how to access basic Ruby structures and how to extend Ruby by creating new modules, classes and methods. This is the basics.
Next we’ll go deeper into it: Memory management, blocks, wrapping C structures, and many more await!

Stay tuned.

C, Howto, Ruby , , , , , , , , , , , , , , , , , , , , , , , ,

Packaging Ruby C extensions in nice gems

In the first part of this series on Ruby and C, we’ve seen how easy it is to write a C extension and use it in Ruby environment.

However, in real world problems, a Ruby library will have several C extensions, bundled together with other Ruby libraries.
Packaging and distributing Ruby gems containing C extensions is fairly easy, and this post explains how.

Here we will build a complete Ruby gem (named mygem), containing an executable with several Ruby libraries and C extensions.

You can get the full source code of this tutorial here.

Our example Ruby gem structure

By convention, Ruby libraries are in a lib/ folder, and C extensions are in a ext/ folder.

Here is the structure of our Ruby gem, respecting conventions:

.
+- bin/
|  +- exec.rb
+- lib/
|  +- mygem/
|     +- myfirstrubylib.rb
|     +- mysecondrubylib.rb
+- ext/
   +- mygem/
      +- myfirstcext.c
      +- extconf.rb
      +- mycsubmodule/
         +- mysecondcext.c
         +- extconf.rb

For our example, running our executable calls the 2 Ruby libraries, that respectively call their corresponding C extension. Each C extension defines 1 method displaying a little message.

Please note the 2 extconf.rb files, responsible for compiling their respective C extension.

The code

Each C extension defines a function displaying a little message:

#include <ruby.h>

static VALUE object_first_cext_call(VALUE rb_self) {
  printf("First C extension call\n");
}

void Init_myfirstcext() {
  // Define method Object#first_cext_call, corresponding to our C function object_first_cext_call; it has 0 argument.
  rb_define_method(rb_cObject, "first_cext_call", object_first_cext_call, 0);
}
#include <ruby.h>

static VALUE object_second_cext_call(VALUE rb_self) {
  printf("Second C extension call\n");
}

void Init_mysecondcext() {
  // Define method Object#second_cext_call, corresponding to our C function object_second_cext_call; it has 0 argument.
  rb_define_method(rb_cObject, "second_cext_call", object_second_cext_call, 0);
}

Do not forget the corresponding extconf.rb files:

require 'mkmf'
create_makefile('myfirstcext')
require 'mkmf'
create_makefile('mysecondcext')

Each Ruby library calls its corresponding C extension:

require "mygem/myfirstcext"

def first_ruby_call
  puts 'First Ruby call'
  first_cext_call
end
require "mygem/mycsubmodule/mysecondcext"

def second_ruby_call
  puts 'Second Ruby call'
  second_cext_call
end

Finally, our executable calls our 2 libraries:

#!/bin/env ruby
require 'mygem/myfirstrubylib'
require 'mygem/mysecondrubylib'

first_ruby_call
second_ruby_call

Test mygem locally

First, we compile our 2 C extensions:

ext/mygem> ruby extconf.rb
creating Makefile

ext/mygem> make
compiling myfirstcext.c
linking shared-object myfirstcext.so
ext/mygem/mycsubmodule> ruby extconf.rb
creating Makefile

ext/mygem/mycsubmodule> make
compiling mysecondcext.c
linking shared-object mysecondcext.so

Then we run our executable. Do not forget to include both lib/ and ext/ in your Ruby path (using -I).

>ruby -Ilib -Iext bin/exec.rb
First Ruby call
First C extension call
Second Ruby call
Second C extension call

So now that you have a running Ruby application, let’s package it into a nice gem.

Packaging Ruby C extensions

As C extensions need to be compiled to be executed, a Ruby gem including C extensions has 2 packaging possibilities: either including the compiled C extension, or including the C extension’s source files.

Including the compiled C extension means that the resulting Ruby gem will be platform dependent. In other words, you’ll need to package several gems: 1 for each platform you want it to run on.
Including the C extension source produces a platform independent Ruby gem (cool), but every time a user will install it, the Ruby gem installation process will try to compile it (not cool). This means that your users must have a C development environment on their host, and this can be a real pain, especially on Windows platforms.

Platform-dependent Ruby gem

When shipping the compiled C extension, you have to take care of 3 things when writing your GemSpec:

  • Add your compiled C extension files (exactly the same way you do for normal Ruby files)
  • Add ext directory to the required paths
  • Set the platform as current

Here is the gemspec file packaging our example gem in a platform-dependent way:

Gem::Specification.new do |spec|
  spec.name = 'mygem'
  spec.version = '0.1'
  spec.summary = 'Summary'
  spec.description = 'Description'
  spec.email = 'myemail@provider.com'
  spec.homepage = 'http://myapp.com'
  spec.author = 'me'
  spec.bindir = 'bin'
  spec.executable = 'exec.rb'
  spec.files = Dir['lib/**/*.rb'] + Dir['bin/*'] + Dir['ext/**/*.so'] + Dir['ext/**/*.dll']
  spec.platform = Gem::Platform::CURRENT
  spec.require_paths = [ 'lib', 'ext' ]
end

And now we can generate the gem:

> gem build mygem.platformdependent.gemspec.rb 
  Successfully built RubyGem
  Name: mygem
  Version: 0.1
  File: mygem-0.1-x86-cygwin.gem

You will notice that your gem has been packaged for your current platform only (here: x86-cygwin). If you want it on another platform, you will have to first compile it on the other platform, and then build a new gem using this gemspec on the same platform.

Now we can try to install it:

> gem install mygem-0.1-x86-cygwin.gem
Successfully installed mygem-0.1-x86-cygwin
1 gem installed
Installing ri documentation for mygem-0.1-x86-cygwin...
Installing RDoc documentation for mygem-0.1-x86-cygwin...

And run it:

> exec.rb
First Ruby call
First C extension call
Second Ruby call
Second C extension call

Everything works like a charm, but on your platform only.

Platform-independent Ruby gem

When shipping the C extension source files, you have to take care of 4 things when writing your GemSpec:

  • Add your C extension source files only: that is the .c and extconf.rb files only. No need for Makefile and compiled files.
  • Add ext directory to the required paths
  • Keep the platform as Ruby (independent)
  • Register the C extensions you are including: this will tell RubyGems that there is a compilation step upon installation. This is done by giving all the paths to extconf.rb files.

Here is the gemspec file packaging our example gem in a platform-independent way:

Gem::Specification.new do |spec|
  spec.name = 'mygem'
  spec.version = '0.1'
  spec.summary = 'Summary'
  spec.description = 'Description'
  spec.email = 'myemail@provider.com'
  spec.homepage = 'http://myapp.com'
  spec.author = 'me'
  spec.bindir = 'bin'
  spec.executable = 'exec.rb'
  spec.files = Dir['lib/**/*.rb'] + Dir['bin/*'] + Dir['ext/**/*.c'] + Dir['ext/**/extconf.rb']
  spec.platform = Gem::Platform::RUBY # This is the default
  spec.require_paths = [ 'lib', 'ext' ]
  spec.extensions = Dir['ext/**/extconf.rb']
end

And now we can generate the gem:

> gem build mygem.platformindependent.gemspec.rb 
  Successfully built RubyGem
  Name: mygem
  Version: 0.1
  File: mygem-0.1.gem

You will notice that your gem has been packaged for no specific platform, and will be installable on all platforms.

Now we can try to install it (first uninstall the previous one if needed using gem uninstall mygem):

> gem install mygem-0.1.gem 
Building native extensions.  This could take a while...
Successfully installed mygem-0.1
1 gem installed
Installing ri documentation for mygem-0.1...
Installing RDoc documentation for mygem-0.1...

Here you will notice that RubyGems has compiled the C extensions as the first installation step. This you you can see with the line Building native extensions. This could take a while...: it runs all the extconf.rb scripts in the background. Therefore, platforms that won’t have a C development environment won’t be able to install your gem.

Now we can run it:

> exec.rb
First Ruby call
First C extension call
Second Ruby call
Second C extension call

Everything works like a charm once again, using a platform-independent gem this time.

Next step

We have seen how to write and package C extensions.
Next step goes into learning the C API!.
Don’t be afraid, it’s easy.

Stay tuned!

C, Howto, New gem, Ruby , , , , , , , ,

How to write C extensions easily in Ruby

Lots of Ruby libraries suffer from performance. It is in fact one of the biggest criticism that is being made to Ruby.

There are lots of ways to improve Ruby’s performance: use latest Ruby version, use JRuby, use Ruby Entreprise Edition, optimize your code…
A great comparison of the different Ruby interpreters can be found here. It is a bit old, but helps getting the picture.

However the most drastic performance gain you can get is by dropping Ruby and using C instead.

The best part is that you just need to drop Ruby for your critical algorithms that need performance, and keep it for the rest of your application.
The even-more-best part is that it is really easy to mix C and Ruby together. The learning curve is quite simple, and this blog post is here to remove natural fears in bridging 2 different languages.

This post is the first part of a series dealing with C extensions in Ruby. Others will be linked here.

What is a C extension?

A C extension is just a compiled file (.so or .dll) that is used exactly the same way as a Ruby library (.rb).
A C extension has to define just 1 C method that will be called upon its load. This method can then define classes, modules, functions and also execute arbitrary code.
Simple, no ?

A C extension needs of course its source written in C, and then needs to be compiled using a C compiler before being usable by your Ruby environment.

By convention, Ruby libraries are stored in a lib/ folder from a Ruby project.
By the same convention, C extensions are stored in a ext/ folder from a Ruby project.

What do I need to create a C extension?

You need Ruby, and a C development environment ([g]cc, ld, [g]make…).
By default, this is already present in most *nix systems.

For Windows, you will have to install those. You can try the following environments:

How to setup your development environment lies beyond the scope of this post: Google indexes already plenty of resources dealing with those topics.

The Hello World from C

First things first, let’s write a Hello World in C and invoke it from Ruby. Our C extension is named myext in this tutorial.

  1. Let’s start by writing our Hello World in C:
    #include "ruby.h"
    
    void Init_myext() {
      printf("Hello Ruby from C!\n");
    }
    

    Notice the Init_myext function: this is the 1 needed function.
    Notice also the include directive that is mandatory for any C extension: it declares the Ruby/C API that will be needed later.

  2. Now we write a little Ruby program that will generate a Makefile able to compile our C extension. By convention, this file is named extconf.rb:
    require 'mkmf'
    create_makefile('myext')
    
  3. Then we generate the makefile using our script:
    > ruby extconf.rb
    creating Makefile
    
  4. And finally compile our extension by invoking make:
    > make
    compiling myext.c
    linking shared-object myext.so
    
  5. Now we can require our C extension from Ruby, the same way we do with Ruby files:
    > ruby -I. -e "require 'myext'"
    Hello Ruby from C!
    

    Notice that by default, you don’t need to specify your library’s extension (.so or .rb): Ruby is intelligent enough to find out by itself.

Next step

That’s all for the first part. You can now create C extensions and make C code run in a Ruby application.

Next we will see how to package our C extensions in nice gems.

C, Howto, Ruby , , , , , , , , , ,

Understanding Spam … figures, business model, techniques, protection

Lately I got really bothered by Spam in my various forums, blogs and emails.
It regularly costs me hours of anti-Spam protection setup, filtering false positives, reconfiguring my Captchas, and cleaning my databases.

So I decided to better understand this phenomenon, why it still exists, how does it work, how much it represents, and how to get rid of it efficiently.

  1. Why does Spam exist?
  2. How does it work?
    1. Direct email
    2. Links posted in forums, blogs and social networks
    3. Search engines
    4. Social networks
    5. Google Ads
  3. How does it translate into figures?
  4. How to fight against spam?

Why does Spam exist?

Basically, Spam aims at making people buy things. Simple: this is just a selling strategy.

There is however a big difference between Spam and other selling strategies: Spam achieves its goal by blindly reaching billions of people repeatedly, using unsolicited media.

A selling strategy con only be viable if its running costs are negligible compared to the sales it gains – in other terms: the Return-On-Investment has to be positive. Therefore, reaching billions of people using phone calls or regular mail would cost too much (both in time and money).

That’s why Spam uses different techniques for that: direct email, links posted in forums and blogs, search engines, social networks, Google Ads. Those means are really low costs and can reach billions of people very easily.

How does it work?

First, Spammers usually work for a website selling products (drugs, boots, bags, books, miracle pills…). Then the goal is to direct people to this website.

Several means:

Direct email

Automated bots scrape websites, forums, blogs to harvest email addresses. Another way is to infiltrate people’s address books – either in their social networks accounts or in their own computer – to get more email addresses. This is usually done with computer viruses (trojans…), phishing emails mimicking social networks, bugs exploits on softwares.

Once millions of email addresses have been harvested, Spam email is sent directly to them, including links to the selling website – this is the Spam emails we all get in our Spam folders. This is also done using bot-nets, in order to not be easily traceable. Bot-nets are usually made of computers that have been infected by viruses, and send emails (zombie style) without their owner being aware.

During web crawling, automated bots (and sometimes human ones) try to post messages containing links to the selling website. Targeted platforms are very often forums, blogs and social networks.

If anonymous posting is disabled, they try to create a user account.

If there is a Captcha protection, they try to pass it. And usually they succeed, at least for the most common Captchas (reCaptcha, phpBB default Captchas…).

Then they try to fool moderators by first waiting for a few days before posting, then posting messages without links to gain their trust, and then posting messages with links to the selling website. As a moderator, when I first got Spam messages without any link (and sometimes just containing random characters), I laughed, thinking “Spam, you’re doing it wrong”, but then I understood their strategy.

A great way to direct people to the selling website is make it highly ranked on search engines. This is done by creating a network of false blogs (also called Splogs) that reference each others and sometimes reference the selling website. As search engines value blogs and sites that are referenced by others, Spammers manage to increase their selling website and splogs’ page rank using this technique. This also leads to posting links to those splogs in forums and blogs messages all over the web.

Another use of this technique is through blogs track-backs, also called Spings.

Social networks

False Twitter or MySpace accounts trying to attract the most followers are also part of this strategy. The more those accounts link to other accounts, the better they will be ranked – in search engines and in the social network itself. Then Spammers post links to selling websites or splogs from those accounts, and it is spread automatically to plenty of people’s walls.

Google Ads

This one is even more insidious. The Spammer buys a Google Adword that is quite irrelevant (like a random suite of characters) for its selling website. Then Spam is posted including the Google Adword, with no link needed (using comments on blogs, forums, social networks, direct emails).

People might not understand the purpose of this Spam including no link, but if by any chance a Google Ad is present on the page displaying the Spam message, there will be a very high probability for the Google Ad to display a link to the Spammer’s selling website. For example using online email clients (GMail, Yahoo, Hotmail…), or any website having some ads displayed somewhere along with its members’ comments.

How does it translate into figures?

Most of the Spam messages are rather stupidly written, some even use gibberish, and most Spam filters are able to catch them.
So why is Spam still so active ?

I stumbled upon a study made in 2008 by researchers from Berkeley that found something very interesting:

  • They set up a false selling website for drugs, with items sold around $100 each.
  • They used a bot-net containing 75800 infected computers: they had a great firing power to mass-send emails and post messages. This bot-net is rented for $80 to send 1 million e-mails.
  • They sent 350 million emails containing direct links to their selling website, during 26 days. The cost of this sending is $28000.
  • They got 28 people that effectively bought from the selling website: representing only 0,000008 % of the Spam sent, and making them earn $2800 in 26 days.

At this point, there is a problem: using the bot-net costs $28000 and they earned $2800. They are clearly in the red: Easy Spamming is not a lucrative activity if you have to rent a bot-net.

So what happens if you are a Spammer that owns a bot-net?

Things are different: from the same study, the researchers rented just 1,5% of the bot-net’s power. This means that they could have sent much more Spam, leading to much more sales at the end if they could use the bot-net completely.

Here the study evaluates the owner’s earning up to $7000 a day, and more than $2 million a year.

Now it is easily understandable that Spam business is not for the little fishes, but is profitable for a few having the infrastructure capable of Spamming in large quantities for nearly no costs.
The fact that Spam usually targets the same industry (pharmacy online drugs and pills) also confirms this explanation.

How to fight against spam?

Several means:

  • Spam filters: Those are tricky, as depending on the filter you use, it can capture lots of false positive. Email clients usually have embedded Spam filters – that’s why you have a Spam folder there. For forums and blogs, I tested Akismet and I am very happy with it so far.
  • Captchas: I learned that common Captchas are becoming more and more useless (reCaptcha, default blogs and forums Captchas…). Bots manage to get pass them, and they annoy most of your users. However, having a customized Captcha is quite efficient.

    My favorite is using Q/A, where I write the questions and answers. Try not asking simple questions, as Bots manage to find the answers on Google. Questions like “Type the 5 first letters of the capital of England” are better than “How much is 4+5 ?”.

    You will also have to change your customized Captcha configuration regularly, especially if your site has enough visibility for human Spammers to have an interest in it – they could adapt their Bots to your Q/A.

  • No automatic follow on social networks: Spams on Twitter and other social networks will follow you (they usually have hot girls as avatars, and their posts are either empty or the same bullshit repeated over and over). Do not follow them back: following them is giving them power.

For phpBB users, you have great recommendations here.
Google is also full of good practices.

On a funnier note, there is a smart way to profit from Spam. Check this out 😉

Hope this explanation will be useful to many.

Antispam, Howto, My life, Web development , , , , , , , , , ,

My daily software stack

Hi

This is a recurring question: “What soft are you using for your day-to-day activities ?”

First, “day-to-day activities” for me mean lots of different things as a freelancer, developer and CEO. So I will try to cope with each one of them.
Then my OS is Windows7 64bits. For some development tasks that are easier to handle on a Unix like OS, I use either Cygwin (less and less I must say) or an Ubuntu on a VirtualBox.
I also have an Android smart-phone.

Please note that I always try and encourage Open Source or free software when I am looking for a new soft. I also try to use cross-platform software, as I also want my soft to work on Unix and MacOS.

So here is my software task for day-to-day activities, in no particular order.

General

  • Administrative, writing and presentation tasks requiring documents: LibreOffice. This office suite is stable and efficient enough to cover all my needs. When I need to communicate with people requiring old legacy formats and software, it can also convert to Microsoft Office files format, and also in PDF. I also use it for my presentations: neat, simple, efficient. Used it (and its genitor OpenOffice) for more than 5 years now, and I never had to go back.
  • My browser: Firefox. Lately it has become really powerful: I browse regularly with at least 50 opened tabs, some of them using heavy Javascript and Flash, and it handles them in a good way (usually <1Gb memory, <10% CPU, responsive).
  • Editing graphics: I use Gimp. It is quite powerful for my needs.
  • Taking quick notes that should be deleted soon after: Windows Notepad is cool: quick and easy to launch. And if I need persistence for those notes, my Desktop is a cool place to save them: I hate having too much stuff on it ; so this pushes me to handle those notes quickly.
  • Using quick computations: Windows Calc. This is something I use very often (unit conversions, sums…).
  • Writing quick scripts for various tasks (bulk files renaming, web scraping, maintenance…): This I do with Ruby and its shell: irb. This is really my daily swiss knife tool.
  • My calendar is Google Calendar. This way I can also access it on my smart-phone, and I have it on my iGoogle page.

My work

  • Keeping track of my time spent: No need of a complex software or framework. LibreOffice Calc is more than what I need. I created a nice worksheet where I report all my activity spent in hours, categorized. Then lots of formulas will show me how much time is spent on each category, how it progresses over time and sprints, and all this with nice graphs that are really simple to maintain.
  • Storing my sensible working files: I use a home-made Ruby script that encrypts/decrypts the contents from a USB key using a password. This way, my data is safe and I can bring it everywhere as Ruby is cross-platform.
  • Dealing with software projects development: For this part I have setup a Redmine server, along with the Redmine backlogs plugin, running on Heroku. This way I have a great environment to track my agile developments.
  • Editing source code: I have 2 winners on this one. I was using Netbeans for more than 5 years, but its recent lack of support for Ruby made me first, keep the 6.9.1 version, and then look for alternatives. So I discovered a great editor, both lightweight and handling big projects: SublimeText. I am using it more and more, and I really think that it will replace Netbeans in the short and long run.
  • My favourite version control system: Git with a nice GUI integrating in Windows explorer: Tortoise Git. So easy to use, so flexible. I not only use it to deliver source code, but also any repository needing synchronizations between several places.
  • Storing my tons of passwords and accounts I have on the Internet (and banks…): KeePass X. It is a cross-platform passwords’ safe that also encrypts/decrypts its database using a password (also works on my smart-phone). I store it on the same USB key as my work files. I made a little script using AutoHotKey that opens KeePassX, looks for desired credentials and paste them in the current active form: this way it is really easy to use it in browsers and applications.
  • Accessing FTP, SFTP and SCP accounts is done using WinSCP. Manages a list of configured hosts, with key-based or password authentication.
  • Connecting using SSH: for a long time, I used the SSH terminal from Cygwin. Now that I am using less and less Cygwin, I tend to use Putty.

My social activity

  • Communicating with chat: I use Pidgin for most of it. It handles plenty of different protocols (GTalk, Yahoo, AIM, Facebok, ICQ, Sametime…), gathering all of them in 1 simple interface, cross-platform. For people used to Skype, I also use chat in Skype.
  • Communicating with video call: The winner is Skype. Easy to use.
  • Following Twitter: This I do using Twitter lists and TweetDeck. This way lists are managed correctly, and I can read structured streams in my browser and on my smart-phone.
  • Reading my email: Done using GMail. I use it to gather emails from my other email accounts all in one place. I use labels to sort them. Pretty efficient.
  • Reading some RSS news: Done using Google Reader. I can follow easily all my RSS feeds that I have categorized in folders, on my iGoogle page.

I aligned shortcuts for all those softwares in my task bar. Therefore it is very easy to get to them.

That’s it: I have lived for more than 5 years with this list, and I must say I am happy with it.

Entrepreneurship, My life, Ruby, Web development, Windows , , , ,

Howto add email notifications to your Redmine install on Heroku

After my last post (Howto install Redmine on Heroku), I wanted to enable email notifications on my Redmine install on Heroku.
Here is the quick howto.

Please note that Heroku does not have its own SMTP server (at least for the free offer). Therefore you’ll have to use an external SMTP server.

  1. Create (by copying config/configuration.yml.example) or edit your config/configuration.yml file. This file will contain your SMTP configuration:
    # specific configuration options for production environment
    # that overrides the default ones
    production:
      email_delivery:
        delivery_method: :smtp
        smtp_settings:
          address: your_smtp_server.com
          port: 25
          domain: your_smtp_server.com
          authentication: :plain
          enable_starttls_auto: false
          user_name: "your_smtp_user_name"
          password: "your_smtp_password"
    

    For those of you that have more complex authentication methods, or special SMTP parameters (for GMail for example), you can check this page.

  2. Add this configuration file in Git. For this, edit .gitignore and make sure that config/configuration.yml is not part of it.
  3. Commit your files and push them:
    • git add .
    • git commit -m"Setting up email"
    • git push heroku master
  4. Test it, by going to your Heroku application, under Administration -> Settings -> Email notifications then click on “Send a test email” at the bottom of the page. This will send you a mail, or display an error. Googling the errors helped me a lot in debugging it.

Done. Simply.

Heroku, Howto, Web development , , , , ,

Howto install Redmine on Heroku

A little howto to get Redmine running on Heroku:

  1. Install Redmine locally. Follow instructions given on the Redmine Install page. I used development mode only in this step.
  2. Setup git in the repository with git init
  3. Edit the .gitignore file, and do the following operations in it:
    • remove Gemfile.lock,
    • remove /config/initializers/secret_token.rb,
    • add .svn
  4. Edit the Gemfile file, and change the part declaring gems for SQLite3 and PostgreSQL this way:
    # Database gems
    platforms :mri, :mingw do
      group :production do
        gem "pg", ">= 0.11.0"
      end
    
      group :development do
        gem "sqlite3"
      end
    end
    
  5. Run bundle install
  6. Edit config/environment.rb to comment out the exit in case of plugins in vendor:
    # Load the rails application
    require File.expand_path('../application', __FILE__)
    
    # Make sure there's no plugin in vendor/plugin before starting
    vendor_plugins_dir = File.join(Rails.root, "vendor", "plugins")
    if Dir.glob(File.join(vendor_plugins_dir, "*")).any?
      $stderr.puts "Plugins in vendor/plugins (#{vendor_plugins_dir}) are no longer allowed. " +
        "Please, put your Redmine plugins in the `plugins` directory at the root of your " +
        "Redmine directory (#{File.join(Rails.root, "plugins")})"
      #exit 1
    end
    
    # Initialize the rails application
    RedmineApp::Application.initialize!
    
  7. Add files to the Git index with git add .
  8. Commit files with git commit -m"Blank Redmine"
  9. Create the Heroku application with heroku create
  10. Push the commit to Heroku with git push heroku master
  11. Migrate database on Heroku with heroku run rake db:migrate
  12. Load default values on Heroku with heroku run rake redmine:load_default_data
  13. Restart your Redmine server on Heroku with heroku ps:restart

This way you have a Redmine server running just fine on Heroku.

For the record, I used Ruby 1.9.3, Rails 3.2.8, Redmine 2.1.0 and Heroku Toolbelt 2.32.4 (and also tried successfully with Redmine 2.0.3 on Rails 3.2.6).

Git, Heroku, Howto, Ruby, Ruby on Rails, Web development , , , , ,