Basic tech stuff

Programming and Linux administration

Archive for the ‘rails’ Category

Never fetch data from views

Posted by Daniel Brahneborg on 2007 August 22

Jeez, when am I going to learn?

I know very well that all data that should be displayed in view must be fetched by the controller.

View must never, ever, under any circumstances, fetch data by themselves.

Still, I manage to break that rule all too often. “It’s just going to fetch THIS little piece of data”. This not only includes SQL statements, but even all kinds of accessors that do more than fetch complete objects from a hashtable or similar.

A while ago I wrote a little find_safely() helper method for my ApplicationController, that handles “Record not found” errors. All find() accesses went through that method, which meant that when the database had to be separated into different systems, which meant that most tables had to have a system_id column, only that single method had to be changed. Yes, the value had to be set in a few places, but it never fetched data for the wrong system.

Unfortunately some views fetched data themselves, which meant that I had to have the system_id check in both ApplicationController and ApplicationHelper. This quickly got out of hand, so it didn’t take long until I moved all those accesses back to the proper controller. When this was done, it was also possible to test that the views were showing the correct data. The normal Rails test harness doesn’t have access to the data that the views fetch, which means it can’t be tested. After the refactoring, it could.

This brings back another of my favourite arguments:

You should write tests, not only to verify that the application does what it is supposed to (you’ll find that out soon enough anyway), but because it forces you to build the application in a better way.

Anyway, the “data from views” problem came back to bite me once again today. The application stores a list of the most recently accessed business objects of each type. This was tested by directly checking the contents of the array where they were stored.

It worked fine, until the objects grew too large, overflowing the “data” column in the sessions table, which in turned caused a “marshal data too short” error too appear in the Mongrel log file. Most pages recommended using a larger column type such as mediumtext, but this is of course a bad idea. Having the poor application read and parse several hundred KB or more on each request, only to serialize it back and update the database afterwards kills performance. A couple of KB is ok, but the 64KB limit of the “text” type is there for a reason.

The proper solution is of course to only store the object id’s, and instanciate the objects only when needed. The application worked again, but now the hard coded test cases started to break. To get them to work, the controllers had to fetch this data, which in turn meant that the views didn’t have to.

Instead of an ugly Module that was imported by both ApplicationController and ApplicationHelper, the “latest objects” code could be merged back into ApplicationController where it belong. The code became simpler, the response time decreased, the database errors disappeared, and a larger part of the code got proper unit tests. Sure, just using “mediumtext” would have been faster in terms of work hours, but the code quality would have suffered.

Andra bloggar om: , , .

Posted in rails | 2 Comments »

Rails fix for nested forms

Posted by Daniel Brahneborg on 2007 May 26

In Rails there is a nice helper function button_to. It uses Javascript in the onclick event to create a new form which is immediately submitted. Perfect when a button is all you really need.

Or so I thought, until I installed the HTML Validator extension to Firefox. It has a super strict HTML parser that tells you when something is wrong with your code. This way there is a much greater chance that the pages will look the way you want in as many browsers as possible.

So what’s the problem? Well, I had a form surrounding a table. The form was really only for the last row in the table, but since a form must either be outside of the table tag or within a td, the only option was to let the form surround the table. On all rows but the one with the real form fields, I wanted that new button. When the new form was created, the HTML validator became a bit upset about the fact that a new form was created within an existing form. This isn’t allowed. (There are sure a lot of things that aren’t allowed in HTML.)

The solution was to patch actionpack/lib/action_view/helpers/url_helper.rb around line 367, changing

    "this.body.appendChild(f); f.method = 'POST'; f.action = this.href;"


    "document.body.appendChild(f); f.method = 'POST'; f.action = this.href;"

This makes the new form live as a child to the top level body tag instead. Now the validator was happy again.

Posted in programming, rails | Leave a Comment »

Reasons for automatic tests

Posted by Daniel Brahneborg on 2007 May 22

As everybody should know by now, there are plenty of reasons for writing automatic tests. One of them, of course, being the fact that you get to see whether your application actually works, at least for some cases. That’s just a small part of the story, though.

Reason number two is design. To be able to write both small unit tests, large system tests and everything in between, the application simply must be well designed. Otherwise it will be impossible to test one thing at a time, and mock out the parts that should be faked. Knowing where to draw the lines between the modules in a system can be difficult, but by simply trying to write nice tests for them, it becomes much easier.

The third reason is refactoring, which I personally got bitten by this weekend when making a change to my RSS/Ping service, written in Ruby on Rails. In the first versions the “ping” logic was a separate program, but the users wanted it merged with the web application. So, I very carefully moved one function at a time into the classes where they belonged, and made a little button in the web interface. Despite being exactly the same code, with no uninitialized variables or anything like that, it simply refused to work.

Today I found the problem. The standalone program used a couple of require statements to import modules for RSS and Atom parsing and whatnot, things that weren’t used in the web application. This made the function fail, even though everything was technically fully correct. The standalone program worked, so tests wasn’t really necessary, I thought.

Lesson learned? Automatic tests are good, and should aim for full code coverage. This makes the programming part so much easier.

Andra bloggar om: , , .

Posted in programming, rails, testing | Leave a Comment »

Rails: belongs_to :polymorphic and inheritance

Posted by Daniel Brahneborg on 2007 May 10

The flag :polymorphic option for belongs_to associations is extremely useful, especially when implementing an authorization system. Let’s say that I have a Permission class that points to either a House or a Car:

class Permission < ActiveRecord::Base; belongs_to :authobject, :polymorphic => true; end
class House < ActiveRecord::Base; has_many :permissions, :as => 'authobject'; end
class Car < ActiveRecord::Base; has_many :permissions, :as => 'authobject'; end

This works just fine, making it possible both to find the object that a Permission is for, and the relevant Permissions for a certain object. In the column authobject_type you get the strings “House” and “Car”, respectively.

Now the problem: We want to replace the Car class with an abstract Vehicle class, with subclasses Car and Boat:

class Vehicle < ActiveRecord::Base; has_many :permissions, :as => 'authobject'; end
class Car < Vehicle; end
class Boat < Vehicle; end

In Rails version 1.2.3 this puts the string “Vehicle” in the Permissions.authobject_type column, which causes lots of stuff to fail. In my case, a bunch of test cases that simply created a Permission and made sure it could be found again. Suddenly it couldn’t.

The problem was this bug:, with a patch that makes sure that the real class name in the authobject_type field. The Rails documentation says you should store the base class in the authobject_type field, but that simply isn’t right, since it makes it impossible to load the right subclass. Since we still want to store permissions to Houses, it’s important that we store the exact class name.

Edit 2007-05-16: The problem is still not completely solved, since :dependent => :destroy still uses the base class name instead of the correct one. Since following the relationship works, you have to loop and destroy the objects manually.

Andra bloggar om: , , .

Posted in programming, rails | Leave a Comment »

Web services are dead, long live web applications

Posted by Daniel Brahneborg on 2007 March 3

Many years ago SOAP was born. To be able to write clients and servers, it got a specification called WSDL. Unfortunately it became too general, allowing for all kinds of call patterns, SMTP transfers, lots of different encodings and styles and whatnot. This made it too big, so there are still no tools that fully support it. JAXP probably comes closest, or maybe Dotnet.

The version that everybody uses is WSDL 1.1. WSDL 1.2 was supposed to be released in 2003. There is still no support for it. Now they’re talking about WSDL 2.0, which is even more useless.

In the real world, people used XMLRPC instead, just as they used Spring and Hibernate instead of the stupid “container based management” which only made things too complex. The latest version of Ruby on Rails gave support for REST, and now it seems that the two worlds are coming together, in something called WADL. It’s very similar to WSDL, but with the useless stuff thrown out, and only for REST style applications. The specification is less than 40 pages, which means it’s actually possible to implement tools for it.

I really, really hope something like this will catch on, because the current situation with web services is a nightmare. Everybody wants it, but for everything that is more complex than a Hello World, it’s useless. If you want to support other languages than Java you have to hand craft every single XML request. Completely insane.

Andra bloggar om: , ,

Posted in programming, rails, wadl, webservice | Leave a Comment »

Standalone code using a Rails database

Posted by Daniel Brahneborg on 2007 February 22

I’m using a Rails application to update a database, but I also want to run an external program on that database once in a while. While it might be done by creating a new thread within the Rails application, I wanted a standalone program. However, it should still use the same database configuration, Ruby models, etc.

First I need to load the Mysql driver, YAML parser, and ActiveRecord:

require 'yaml'
require 'rubygems'
require_gem 'mysql'
require_gem 'activerecord'

Then the load path for ‘require’ must be extended to include my models:

$: << File.join(File.dirname(__FILE__), '../../app/models')

To connect to the database, make sure ActiveRecord uses this connection, and that it gets closed afterwards, I use code like this. The application code is sent as a block.

def with_database(config)
  dbh = nil
    dbh = Mysql.real_connect(config["host"],
      config["username"], config["password"], config["database"])
    ActiveRecord::Base.establish_connection config
    dbh.close unless dbh.nil?

First I load the database config. The “development” profile is hard coded, but could easily be changed into using an environment variable or something.

filename = File.join(File.dirname(__FILE__), '../../config/database.yml')
dbconfig = YAML.load(
dev = dbconfig["development"]

Then I can run the application code:

with_database(dev) {
# yadda yadda

When the function completes, either normally or by raising an exception, the database is closed.

Posted in programming, rails | Leave a Comment »