Caius Theory

Now with even more cowbell…

Install capybara-webkit gem on Ubuntu

Dear future Caius searching for this issue,

The apt package you need to install to use the capybara-webkit rubygem on ubuntu (tested on 10.04 and 11.10) is libqt4-dev. That is, to gem install capybara-webkit, you need to run aptitude install libqt4-dev.

Yours helpfully,
Past Caius

Abusing Ruby 1.9 & JSON for fun

Ever since I found out about the new hash syntax you can use in ruby 1.9, and how similar that syntax is to JSON, I've been waiting for someone to realise you can just abuse eval() for parsing (some) JSON now.

For example, lets say we have the following ruby hash, which could be generated by a RESTful api:

thing = {
    :person => {
        :name => "caius"
    }
}

If we pull in the JSON gem and dump that out as a string, we get the following:

jsonstr = thing.to_json
# => '{"person":{"name":"caius"}}'

That's… not quite what we wanted. It's not going to turn back into valid ruby as it is. Luckily javascript will parse objects without requiring the attributes to be wrapped in quotes, eg: {some: "attribute"}. We could build a JSON emitter that does it properly, or we could just run it through a regular expression instead. (Lets also add a space after the colon to aid readability.)

jsonstr.gsub!(/"([^"]+)": /, '\1: ')
# => '{person: {name: "caius"}}'

Okay, so now we've turned a ruby hash into a JSON hash, that can still be parsed by the browser. Here's a screenshot to prove that:

Valid JSON 'thing'

As you can see, it parses that into a valid JS object, complete with person and then (nested) name attributes. If we wanted to, thing["person"]["name"] or thing.person.name would access the nested value "caius" just fine.

Now then, we've proved that is successfully parsed into javascript objects by the browser, generated from a ruby hash. No great shakes there, that's fairly simple and has worked for ages. Now for my next trick, I'm going to turn that string of JSON back into a ruby hash, all without going anywhere near the JSON gem.

Some of you might have guessed what I'm about to do and have started hoping you've guessed wrongly — just for the record I don't condone doing this except for fun and games. The JSON gem is there for a reason ;) With that little disclaimer out the way, here we go!

thing2 = eval(jsonstr)
# => {:person=>{:name=>"caius"}}
thing2 == thing
# => true

Oh snap! We just turned javascript objects back into valid ruby objects, in one simple method call. And we'd be able to access the "caius" value by calling thing2[:person][:name], or creating OpenStructs from the hashes and calling thing2.person.name. Which is uncannily like the JS!

Updated 2011-02-07: Jens Ayton pointed out unquoted keys aren't strictly valid JSON, which is correct. Amended to say they're parsed as javascript objects instead, with no mention of it being valid JSON.

Adding XHTML output validation to Cucumber stories

At the 2009 Barcamp Leeds I attended a talk by Neil Crosby where he talked about automated testing, and about how he felt there was a gap in everything that people were testing. Everyone has unit tests, and people are doing full stack testing too, but no-one (so he feels) does XHTML/CSS/JS validation as part of their automated test suite. And certainly from what I've seen on the mainstream Ruby site's about testing, I agreed with him.

So after his talk I had a quick look at his frontend test suite, and started wondering where exactly I would fit frontend validation testing into my workflow. Would it be part of my unit tests (RSpec), or part of the full stack tests (Cucumber)? As you've probably guessed by the title of this post, its ended up going into my cucumber tests. Since the initial play its been something I've mused about occasionally, but not something I've actively looked into how to implement as part of my test workflow.

Fast-forward a few weeks from Barcamp Leeds and I see a news article in my feed reader entitled "Easy Markup Validation" which gets me hopeful someone's solved this frontend validation thing easily for Rubyists. A quick read through and I'm sold on it and installing the gem. Opened an existing project I'm working on which has a fairly extensive test suite (both unit tests & full stack tests) and tried to slot the validation into my controller unit tests.

Problem with doing this is by default RSpec-rails doesn't generate the views in your controller specs. At that point I realised I was already generating the full page when I was doing a full stack test using culerity and cucumber. So why not just add a cucumber step in my stories to validate the HTML on each page I visit? Mainly because its not enough of a failure for this app to have invalid XHTML markup. Having valid markup would be nice, but I'd rather have it as a separate test to my stories in some way.

Currently I just do that by only validating if ENV["VALIDATION"] is set to anything, so a normal run of my cucumber stories will just test the app does what its supposed to do. If I run them with VALIDATION=true then it will check my markup is valid as well.

features/support/env.rb

require "markup_validity" if ENV["VALIDATION"]

features/step_definitions/general_steps.rb

Then %r/the page is valid XHTML/ do
  $browser.html.should be_xhtml_strict if ENV["VALIDATION"]
end

features/logging_in.feature

Feature: Logging in
  In order to do stuff
  As a registered user
  I want to login

  Scenario: Successful Login
    Given there is a user called "Caius"

    When I goto the homepage
    Then the page is valid XHTML

    When I click on the "Login" link
    Then I am redirected to the login page
    And the page is valid XHTML

    When I enter my login details
    And I click "Login"
    Then I am redirected to my dashboard
    And the page is valid XHTML

Now when I run cucumber features/logging_in.feature, it doesn't validate the HTML, it just makes sure that I can login as my user and that I am redirected to the right places. But if I run VALIDATION=true cucumber features/logging_in.feature, then it does validate my XHTML on the homepage, the login page and on the user's dashboard. If it fails validation then it gives you a fairly helpful error message as to what it was expecting and what it found instead.

From a quick run against a couple of stories in my app I discovered that I've not been wrapping form elements in an enclosing element, so they've been quickly fixed and now they validate. Now I realise this gem is only testing XHTML output, and doesn't include CSS or JS validation, but from a quick peek at the gem's source it should be fairly easy to add both of those in I think, although again they aren't major errors for me yet in this app.

Migrating Rubygems to Ruby 1.9.x

So I just installed ruby 1.9.1 through MacPorts and wanted to easily migrate my rubygems across from 1.8 to see which ones would fail to install.

Thought about it for a while, then came up with the following bash one-liner to do it:

gem list | grep "(" | awk '{ print $1 }' | xargs -L 1 gem1.9 install

NB: Installing Ruby 1.9.1 through macports sudo port install ruby19 means I get ruby1.9, gem1.9 and rake1.9 installed alongside my usual 1.8 ruby, gem and rake.

That grabs the list of installed gems from gem, searches for lines containing "(" so it only grabs the gem names, spits out the first section of the line, which is the name of the gem, and finally calls gem1.9 install for each line via xargs -L 1. Make sure to run it as root or prefix gem1.9 with sudo. (Or let it install in your home folder, but I hate that.)

From my quick run of the above snippet, 75% of my gems installed (73 out of 98) and the other few that failed to install were ones like Hpricot that require native extensions compiling. You can see the entire list of failures and successes of the gems in this pastie

Install Mysql Gem on Leopard

So, I keep having to reinstall mysql5 and rubygems from time to time for various reasons. I always install mysql5 through MacPorts as a dependency for the php5 port (along with various other bits for the LA*P stack).

sudo port install php5 +mysql5 +pear +readline +sockets +apache2 +sqlite

Once this is installed then I have mysql and can setup my databases, etc.

Ignoring the rest of the LAMP stack, I then need to connect Ruby to the Mysql I just installed through MacPorts. Its quite simple to do, once you know the right argument to pass to it. The easiest way is to just tell it where the mysql5_conf file is and let it figure out the rest for itself.

sudo gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5

Hopefully this will save me 10 minutes of googling next time I need to do this!

Update 2009-01-21

I'm an idiot and typed the gem install command by hand, and ended up with --with-mysql-conf instead of --with-mysql-config. Updated now.

Update 2009-10-19

On Snow Leopard I needed to tell rubygems to install the gem as a 64-bit binary. Hattip to Philipp

sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- \
  --with-mysql-config=/opt/local/bin/mysql_config5

Removing non-existent source from rubygems

I just came to move some ruby scripts onto my mac mini, and to do so I needed to install a couple of gems. Now I realised I hadn't installed or updated rubygems on the machine for a while, so I figured it was best to update gem before installing the gems I wanted. Easier said than done.

At some point in the past I had added http://gems.datamapper.org as a source to rubygems. Since then the datamapper project has discontinued using this gem source to serve up gems, so I was getting the following output:

mm:daemons caius$ sudo gem update --system
Updating installed gems
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 404 fetching http://gems.datamapper.org/yaml

Eeek! I can't update because the source no longer exists. So I figured I'd remove the source before updating, that should work right? Wrong. It updates the sources before removing the source from the config it would appear.

mm:daemons caius$ sudo gem sources
** CURRENT SOURCES ***

http://gems.rubyforge.org
http://gems.datamapper.org

mm:daemons caius$ sudo gem sources -r http://gems.datamapper.org
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 404 fetching http://gems.datamapper.org/yaml

Oh balls. So how do I remove the source without updating it first. I need to update it to remove it, but to remove it I need to update from it. Gotta love catch 22s!

I remembered that gem install has an option not to update sources, --no-update-sources. So I figured thats gotta work when removing a source as well, but it doesn't.

mm:daemons caius$ sudo gem sources -r http://gems.datamapper.org --no-update-sources
ERROR:  While executing gem ... (OptionParser::InvalidOption)
    invalid option: --no-update-sources

Oh crap. Now what do I do? Take my usual tactic and google for a hint of course! I'd considered trying to find where the gem config was and remove the source by hand, but I figured that wouldn't be that simple. After hitting a couple of sites that weren't relevant I ended up on the edge of complexity where he mentions the command nano ~/.gemrc. Which made me wonder if that file contains the sources.

mm:daemons caius$ cat ~/.gemrc
--- 
:update_sources: true
:verbose: true
:bulk_threshold: 1000
:sources: 
- http://gems.rubyforge.org
- http://gems.datamapper.org
:backtrace: false
:benchmark: false

All I needed to do was remove the - http://gems.datamapper.org line and poof, gem was working again. One quick gem update --system later and I was upgraded from gem 1.1.1 to 1.3.1 and installing the gems I needed.