At the 2009 Barcamp Leeds I attended a talk by Neil Crosby where he talked about automated testing, and about how he felt there was a gap in everything that people were testing. Everyone has unit tests, and people are doing full stack testing too, but no-one (so he feels) does XHTML/CSS/JS validation as part of their automated test suite. And certainly from what I've seen on the mainstream Ruby site's about testing, I agreed with him.
So after his talk I had a quick look at his frontend test suite, and started wondering where exactly I would fit frontend validation testing into my workflow. Would it be part of my unit tests (RSpec), or part of the full stack tests (Cucumber)? As you've probably guessed by the title of this post, its ended up going into my cucumber tests. Since the initial play its been something I've mused about occasionally, but not something I've actively looked into how to implement as part of my test workflow.
Fast-forward a few weeks from Barcamp Leeds and I see a news article in my feed reader entitled "Easy Markup Validation" which gets me hopeful someone's solved this frontend validation thing easily for Rubyists. A quick read through and I'm sold on it and installing the gem. Opened an existing project I'm working on which has a fairly extensive test suite (both unit tests & full stack tests) and tried to slot the validation into my controller unit tests.
Problem with doing this is by default RSpec-rails doesn't generate the views in your controller specs. At that point I realised I was already generating the full page when I was doing a full stack test using culerity and cucumber. So why not just add a cucumber step in my stories to validate the HTML on each page I visit? Mainly because its not enough of a failure for this app to have invalid XHTML markup. Having valid markup would be nice, but I'd rather have it as a separate test to my stories in some way.
Currently I just do that by only validating if ENV["VALIDATION"] is set to anything, so a normal run of my cucumber stories will just test the app does what its supposed to do. If I run them with VALIDATION=true then it will check my markup is valid as well.
features/support/env.rb
require "markup_validity" if ENV["VALIDATION"]
features/step_definitions/general_steps.rb
Then %r/the page is valid XHTML/ do
$browser.html.should be_xhtml_strict if ENV["VALIDATION"]
end
features/logging_in.feature
Feature: Logging in
In order to do stuff
As a registered user
I want to login
Scenario: Successful Login
Given there is a user called "Caius"
When I goto the homepage
Then the page is valid XHTML
When I click on the "Login" link
Then I am redirected to the login page
And the page is valid XHTML
When I enter my login details
And I click "Login"
Then I am redirected to my dashboard
And the page is valid XHTML
Now when I run cucumber features/logging_in.feature, it doesn't validate the HTML, it just makes sure that I can login as my user and that I am redirected to the right places. But if I run VALIDATION=true cucumber features/logging_in.feature, then it does validate my XHTML on the homepage, the login page and on the user's dashboard. If it fails validation then it gives you a fairly helpful error message as to what it was expecting and what it found instead.
From a quick run against a couple of stories in my app I discovered that I've not been wrapping form elements in an enclosing element, so they've been quickly fixed and now they validate. Now I realise this gem is only testing XHTML output, and doesn't include CSS or JS validation, but from a quick peek at the gem's source it should be fairly easy to add both of those in I think, although again they aren't major errors for me yet in this app.
markup_validation looks awesome. However, I think having explicit steps for this in your Cucumber features is noisy. I would implement it differently I think.
I use webrat for most of my projects, and the way you get the response in Webrat is via the #response_body method.
Wouldn't it be better if #response_body did the validation transparently? Then you don't have to be explicit about it in your features.
Yeah I'm seriously not convinced that adding steps is the way to go, but some kind of integration with the cucumber stack is better than writing explicit URLs for it to validate I think. Mainly because you're already loading the content, so might as well validate it at the same time.
Easiest way to get it into the stack was to create a step for it, I've just remembered that culerity adds a
assert_successful_responsemethod to the end of any steps that load pages, and I've got that at the end of steps that check that js has executed.So I should probably add the validation in that method.
Here is how I added automatic XHTML validation to cucumber. Unfortunately this hack edits the webrat_steps file. There is probably a better way, maybe using Ruby's dynamic features to wrap the webrat methods to add the functionality using Ruby's magic.
Hopefully validation will be added to future versions of webrat.
This solution validates every web page. You can also turn validation on and off in cucumber. You don't have to change any of your features unless you need to turn validation off for that feature. I have used a similar approach in the past when running integration tests in Rails.
Here is what I did. I hacked webrat_steps (at my own peril). Note that I set it to validate against xhtml_transitional since that is what my html files have as doctype.
In webrat steps I added the test to every given that seemed to bring up a new page. I added a step to turn validation on and off as well.
env.rb
webrat_steps.rb
*** Add this line wherever a new page would come up.
common_steps.rb is my own step file:
Here is a case where I don't want validation. (Not very Ruby like as I set validation off and then have to remember to turn it on at the end)
foo.feature
Thats it! I like the validator gem, its much much faster than the previous approach of using the w3c validator! I'm really enjoying Cucumber!!