Caius Theory

Now with even more cowbell…

Adding XHTML output validation to Cucumber stories

At the 2009 Barcamp Leeds I attended a talk by Neil Crosby where he talked about automated testing, and about how he felt there was a gap in everything that people were testing. Everyone has unit tests, and people are doing full stack testing too, but no-one (so he feels) does XHTML/CSS/JS validation as part of their automated test suite. And certainly from what I've seen on the mainstream Ruby site's about testing, I agreed with him.

So after his talk I had a quick look at his frontend test suite, and started wondering where exactly I would fit frontend validation testing into my workflow. Would it be part of my unit tests (RSpec), or part of the full stack tests (Cucumber)? As you've probably guessed by the title of this post, its ended up going into my cucumber tests. Since the initial play its been something I've mused about occasionally, but not something I've actively looked into how to implement as part of my test workflow.

Fast-forward a few weeks from Barcamp Leeds and I see a news article in my feed reader entitled "Easy Markup Validation" which gets me hopeful someone's solved this frontend validation thing easily for Rubyists. A quick read through and I'm sold on it and installing the gem. Opened an existing project I'm working on which has a fairly extensive test suite (both unit tests & full stack tests) and tried to slot the validation into my controller unit tests.

Problem with doing this is by default RSpec-rails doesn't generate the views in your controller specs. At that point I realised I was already generating the full page when I was doing a full stack test using culerity and cucumber. So why not just add a cucumber step in my stories to validate the HTML on each page I visit? Mainly because its not enough of a failure for this app to have invalid XHTML markup. Having valid markup would be nice, but I'd rather have it as a separate test to my stories in some way.

Currently I just do that by only validating if ENV["VALIDATION"] is set to anything, so a normal run of my cucumber stories will just test the app does what its supposed to do. If I run them with VALIDATION=true then it will check my markup is valid as well.


require "markup_validity" if ENV["VALIDATION"]


Then %r/the page is valid XHTML/ do
  $browser.html.should be_xhtml_strict if ENV["VALIDATION"]


Feature: Logging in
  In order to do stuff
  As a registered user
  I want to login

  Scenario: Successful Login
    Given there is a user called "Caius"

    When I goto the homepage
    Then the page is valid XHTML

    When I click on the "Login" link
    Then I am redirected to the login page
    And the page is valid XHTML

    When I enter my login details
    And I click "Login"
    Then I am redirected to my dashboard
    And the page is valid XHTML

Now when I run cucumber features/logging_in.feature, it doesn't validate the HTML, it just makes sure that I can login as my user and that I am redirected to the right places. But if I run VALIDATION=true cucumber features/logging_in.feature, then it does validate my XHTML on the homepage, the login page and on the user's dashboard. If it fails validation then it gives you a fairly helpful error message as to what it was expecting and what it found instead.

From a quick run against a couple of stories in my app I discovered that I've not been wrapping form elements in an enclosing element, so they've been quickly fixed and now they validate. Now I realise this gem is only testing XHTML output, and doesn't include CSS or JS validation, but from a quick peek at the gem's source it should be fairly easy to add both of those in I think, although again they aren't major errors for me yet in this app.

Merry Testing

Just a few examples of the same test written in a few languages. Its testing setting the date on an object that is created in the tests' setup method already. These fall under the unit testing, rather than full-stack testing.

Testing in ObjC with OCUnit

// Add a date and time
- (void)testSettingDate
    NSDate *theDate = [NSDate date];        
    STAssertNoThrow([calc setDate:theDate], @"Shouldn't raise an exception");
    // And it should match when pulled out as well
    STAssertEqualObjects([calc date], theDate,
                         @"%@ should match %@",
                         [calc date], theDate);


Testing in Ruby using RSpec

it "should set the date successfully" do
  the_date = = the_date
  # And it should match when pulled out as well == the_date

Testing in Ruby using Test::Unit

def test_setting_date
  the_date = = the_date
  # And it should match when pulled out as well
  assert_equal(, the_date)

Testing in PHP using PHPUnit

function testSettingDate()
    $date = date();
    $calc->date = $date;
    # And it should match when pulled out as well
    $this->assertEquals($calc->date, $date);

Create a blank rails app including plugins

When I create a rails app from scratch I like to include certain plugins to help me write the app, such as the Rspec testing framework instead of the built-in Test::Unit and jQuery instead of prototype.

And here are the commands in the order I run them to create the blank app.

# Create the rails app
cd ~/Sites/apps/
rails myapp
cd myapp

# Setup a git repo
git init
# Add all files and make the initial import
git add .
git commit -m "Initial Import"

# Add the plugins as git submodules
git submodule add git:// vendor/plugins/rspec
git submodule add git:// vendor/plugins/rspec-rails
git submodule add git:// vendor/plugins/cucumber
git submodule add git:// vendor/plugins/webrat
git submodule add git:// vendor/plugins/demeters_revenge

# Commit the changes
git ci -am "Adding all needed submodules"

# Replace TestUnit with rspec
git rm -r test/
ruby script/generate rspec
# Replace stories with cucumber features
rm -rf stories/
ruby script/generate cucumber

# Add the changes to git
git add .
git ci -m "Committing initial rspec/cucumber files"

# Install jRails, we have to install it using script/plugin
# Remove existing javascript files
git rm public/javascripts/*
mkdir public/javascripts
# Add jrails
ruby script/plugin install
git add vendor/plugins/jrails/ public/javascripts
git ci -m "Adding jRails to replace Prototype"

And now you have a blank app waiting for you to write using features for full stack testing, and rspec for testing model and controller code.

Updated 2008-11-04

Added demeters revenge and jRails plugins.

Update 2008-11-05

I've also blogged the .gitignore file I use with rails apps as well. Usually add it into my apps before running git init