Caius Theory

Now with even more cowbell…

#to_param and keyword slugs

Imagine you've got a blogging app and it's currently generating URL paths like posts/10 for individual posts. You decide the path should contain the post title (in some form) to make your URLs friendlier when someone reads them. I know I certainly prefer to read http://caiustheory.com/abusing-ruby-19-and-json-for-fun vs http://caiustheory.com/?id=70. (That's a fun blog post if you're into (ab)using ruby occasionally!)

Now you know all about how to change the URL path that rails generates—just define to_param in your app. Something simple that generates a slug consisting of hyphens and lowercase alphanumerical characters. For example:

# 70-abusing-ruby-1-9-json-for-fun
def to_param
  "#{id}-#{title.gsub(/\W/, "-").squeeze("-")}".downcase
end

NB: You might want to go the route of storing the slug against the post record in the database and thus generating it before saving the record. In which case the rest of this post is sort of moot and you just need to search on that column. If not, then read on!

Now we're generating a nice human-readable URL we need to change the way we find the post in the controller's show action. Up until now it's been a simple @post = Post.find(params[:id]) to grab the record out the database. Problem now is params[:id] is "70-abusing-ruby-1-9-json-for-fun", rather than just "70". A quick check in the String#to_i docs reveals it "Returns the result of interpreting leading characters in str as an integer base base (between 2 and 36)." Basically it extracts the first number it comes across and returns it.

Knowing that we can just lean on it to extract the id before using find to look for the post: @post = Post.find(params[:id].to_i). Fantastic! We've got nice human readable paths on our blog posts and they can be found in the database. All finished… or are we?

There's still a rather embarassing bug in our code where we're not explicitly checking the slug in the URL against the slug of the Post we've extracted from the database. If we visited /posts/70-ruby-19-sucks-and-python-rules-4eva it would load the blog post and render it without batting an eyelid. This has caused rather a few embarrassing situations for some high profile media outlets who don't (or didn't) check their URLs and just output the content. Luckily there's a simple way for us to check this.

All we want to do is render the content if the id param matches the slug of the post exactly, and return a 404 page if it doesn't. We already know the id param (params[:id]) and have pulled the Post object out of the database and stored it in an instance variable (@post). The @post knows how to generate it's own slug, using #to_param.

So we end up with something like the following in our posts controller, which does all the above and correctly returns a 404 if someone enters an invalid slug (even if it starts with a valid post id):

def show
  @post = Post.find(params[:id].to_i)
  render_404 && return unless params[:id] == @post.to_param
end

def render_404
  render :file => Rails.root + "public/404.html", :status => :not_found
end

And going to an invalid path like /posts/70-ruby-19-sucks-and-python-rules-4eva just renders the default rails 404 page with a 404 HTTP status. (If you want the id to appear at the end of the path, alter to_param accordingly and do something like params[:id].match(/\d+$/) to extract the Post's id to search on.)

Hey presto, we've implemented human readable slugs that are tamper-proof (without storing them in the database.)

(And bonus points if in fact you spotted I used my blog as an example, but that it isn't a rails app. (Nor contains the blog post ID in the pretty URL.) It's actually powered by Habari at the time of posting!

Capitalise "ringer" on the iPhone Volume Bezel

Backstory: Got myself a first generation iPhone second hand and unlocked it to work on my existing T-Mobile (Official iPhone network in the UK is O2.) Noticed after a week or so of owning it that when you change the volume on the phone, the bezel that comes up says "ringer" across the top. But when you have headphones plugged in, it says "Headphones". (Note the capitalisation difference.)

Now I'm not usually bothered by stuff like this (honest!) but as soon as I'd noticed the "bug", I couldn't help but think of it everytime I changed the volume, whether I was looking at the screen or not. Seeing as I'm running a jailbroken phone, and therefore have SSH access to it, I figured the string would be defined in a .strings file somewhere in the /System folder. And I'd be able to change it!

Fast-forward a few months and I install the iPhone OS 3.0 update (jailbroken of course), and finally decide to turn the phone's SSH server on and go looking for the setting. To do so I figured I'd just need grep installed on the phone - I could copy the file itself to my mac and edit it there.

So I connect to the phone, have a poke around the filesystem and then start a search to find the correct file:

# On the iPhone
$ cd /System/Library/
$ grep -r "ringer" *
Binary file CoreServices/SpringBoard.app/English.lproj/SpringBoard.strings matches
Binary file CoreServices/SpringBoard.app/M68AP.plist matches
Binary file CoreServices/SpringBoard.app/SpringBoard matches
Binary file Frameworks/CFNetwork.framework/CFNetwork matches
Binary file Frameworks/CFNetwork.framework/da.lproj/Localizable.strings matches
Binary file Frameworks/CFNetwork.framework/no.lproj/Localizable.strings matches
Binary file Frameworks/Foundation.framework/da.lproj/URL.strings matches

At which point I stopped the grep search (^C) because I know the home screen of the iPhone is the SpringBoard.app, so I figured it would be in the file SpringBoard.app/English.lproj/SpringBoard.strings. Making sure to have SSH enabled on your mac, a simple scp CoreServices/SpringBoard.app/English.lproj/SpringBoard.strings user@your_mac.local: later and the file is sat in my home folder on my mac.

Switching to the mac, now I try and open the file with TextMate, only to realise its in binary format. I need it in the nice XML format to edit it, so a quick google later and I've found a hint on MacOSXHints telling me how to convert from binary to xml plist format.

# On the mac
$ plutil -convert xml1 SpringBoard.strings

Then opening the file in TextMate was a bit more successful! I can actually understand what its defining now. Search through the file for "ringer" and I found the following lines:

<key>RINGER_VOLUME</key>
<string>ringer</string>

Change the "ringer" to "Ringer" between the <string> and my editing work is complete! Yes, it really is that easy to edit an interface string that is defined in a .string. Now I just need to convert the file back to binary, and copy it back to the phone. Converting back to binary file is one line, just change the xml1 in the previous command to binary1.

# On the mac
$ plutil -convert binary1 SpringBoard.strings

And then scp it back to the phone, make a backup of the existing file, and overwrite the existing file with the new one I've edited:

# On the iPhone
$ cd ~
$ scp user@mac_name.local:SpringBoard.strings .
$ cd /System/Library/CoreServices/SpringBoard.app/English.lproj/
$ mv SpringBoard.strings SpringBoard.strings.bak
$ cp ~/SpringBoard.strings SpringBoard.strings

And then restart the phone, either in the usual manner or just run reboot on the phone via SSH. Lo and behold once its rebooted and I changed the volume, it read "Ringer"!

Screenshot of Volume bezel

Quantum Javascript Bug

So I've got some js I've written to update a couple of <select> lists in a form, and it was all working fine for me (under Safari.) John happened to mention it wasn't working for him under Firefox, so I fired up Firefox and took a look. Could reproduce it perfectly, changing the first popup was populating the second one, but then wasn't selecting the right value from the list.

Having no idea what was happened I figured I'd enable firebug and watch it execute to figure out what was happening. Enabled firebug, reloaded the page, selected from the first popup… and voila! It updated the second one and selected the correct row! WTF!!!

Turned firebug off and it didn't work, turned it back on and it worked. Figured it might be something buggy in the Firefox 3.0.5 js runtime, so I grabbed a copy of the new beta 3.5 and tried it in there—still failed to update the page as it should.

Then started poking around the javascript code, the function that was seemingly failing to run was being triggered by a setTimeout() call set to 1 second. We figured it might be the timing causing it, so started playing around with the time, tried anything from ½ a second up to 4 seconds but still no joy in firefox with firebug turned off.

Then John went looking for the javascript errors in firefox (with firebug off) and discovered that it was throwing an error because window.console didn't exist. All of a sudden it made perfect sense! Safari has window.console.log() for writing to the console log, as does firebug. But of course firefox without firebug doesn't!

So the function was just exiting on that error. It was very weird initially to have it work perfectly as soon as the developer tools were enabled!

Validating Data with Regular Expressions in Ruby

I happened to be sent a link to the OWASP paper on Rails Security recently and started reading it. Partway in there's a section on Regular Expressions, which opens with the following line:

A common pitfall in Ruby's regular expressions is to match the string's beginning and end by ^ and $, instead of \A and \z.

Now I've never used \A and \z in my regular expressions to validate data, I've only ever used ^ and $ assuming they matched the start and end of the string. This becomes an issue with validating data in rails, because %0A (\n URL encoded) is decoded by rails before passing the string to your model to validate.

Testing our expectations

Lets say we want to validate the string as a username for our app. A username is 5 characters long and consists only of lowercase letters.

regex = /^[a-z]{5}$/

First we make sure it matches the data we want it to:

"caius".validate(regex) # => true

Excellent, that validated. Now we'll try a shorter string, which we expect to fail.

"cai".validate(regex) # => false

Once more, it behaves how we expected it to. The shorter string was rejected as we wanted it to be. Now, what happens if we test a string with a newline character in it? We'll make sure the data before the \n is valid, and then add some more data after the newline.

"caius\nfoo".validate(regex) # => true

Uh oh! That validated and would've been saved as a username?!

Lets have a look at exactly what's happening there, the $ matches the \n character, so the regex is only matching the first 5 characters of the string, and just ignores anything after the \n. As it turns out, this is exactly what we've asked the regex to match, but we didn't want this behaviour.

Now you might be thinking, "So what? someone can have a username with a newline in it." For starters this will probably display weirdly anywhere you use their username, but more importantly it opens your application to an injection attack. Suppose they took advantage of this by setting their username to include some javascript on the page which stole your login cookie and sent it to them. You view their account in the admin section and oh no! They can login as your admin account and do what they want.

Simple example of this is just having it output an alert dialog. (This is actually the code I'll use to test an application as its not malicious, but blindingly obvious if the javascript is executed or not.)

"caius\n<script>alert('hello')</script>".validate(regex) # => true

Ok, so that was the result we were expecting this time, although it's still not the outcome we wanted. Anytime their username is viewed (providing you aren't escaping the data to HTML entities) you'll see the following:

javascript alert dialog

The Solution

Having realised from our testing above that ^$ matches the beginning/end of a line in ruby not the beginning and end of a string, I hear you cry, "How do we make sure we're matching the entire string?!"

The answer is pretty simple. Just swap out ^$ for \A\z. Lets go ahead and try this with the same data as we have above, but with the modified regular expression.

new_regex = /\A[a-z]{5}\z/
"caius".validate(new_regex) # => true

That's a good start, the valid string still matches.

"cai".validate(new_regex) # => false

Looks like it's going well, invalid string is invalid.

"caius\nfoo".validate(new_regex) # => false

Oh Excellent! It's validating this one correctly now.

And just for consistency, lets test it with a more likely attack string.

"caius\n<script>alert('hello')</script>".validate(new_regex) # => false

Fantastic! We've fixed the security hole in our validation of the user's username.


If you want to actually run the code above you'll need the following at the start of the ruby script to patch the validate method into String.

class String
  def validate regex
    !self[regex].nil?
  end
end

Update: I had \Z in the new_regex rather than the \z it should've been. Thanks Ciarán.

Fix Mail.app crashing after 10.5.6 upgrade

When you upgrade to Mac OS 10.5.6, Mail.app might start crashing a few seconds after starting due to the GPG Bundle.

The solution is to grab the updated version of the GPG bundleGPGMail_d55_Leopard.dmg

Removing non-existent source from rubygems

I just came to move some ruby scripts onto my mac mini, and to do so I needed to install a couple of gems. Now I realised I hadn't installed or updated rubygems on the machine for a while, so I figured it was best to update gem before installing the gems I wanted. Easier said than done.

At some point in the past I had added http://gems.datamapper.org as a source to rubygems. Since then the datamapper project has discontinued using this gem source to serve up gems, so I was getting the following output:

mm:daemons caius$ sudo gem update --system
Updating installed gems
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 404 fetching http://gems.datamapper.org/yaml

Eeek! I can't update because the source no longer exists. So I figured I'd remove the source before updating, that should work right? Wrong. It updates the sources before removing the source from the config it would appear.

mm:daemons caius$ sudo gem sources
** CURRENT SOURCES ***

http://gems.rubyforge.org
http://gems.datamapper.org

mm:daemons caius$ sudo gem sources -r http://gems.datamapper.org
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 404 fetching http://gems.datamapper.org/yaml

Oh balls. So how do I remove the source without updating it first. I need to update it to remove it, but to remove it I need to update from it. Gotta love catch 22s!

I remembered that gem install has an option not to update sources, --no-update-sources. So I figured thats gotta work when removing a source as well, but it doesn't.

mm:daemons caius$ sudo gem sources -r http://gems.datamapper.org --no-update-sources
ERROR:  While executing gem ... (OptionParser::InvalidOption)
    invalid option: --no-update-sources

Oh crap. Now what do I do? Take my usual tactic and google for a hint of course! I'd considered trying to find where the gem config was and remove the source by hand, but I figured that wouldn't be that simple. After hitting a couple of sites that weren't relevant I ended up on the edge of complexity where he mentions the command nano ~/.gemrc. Which made me wonder if that file contains the sources.

mm:daemons caius$ cat ~/.gemrc
--- 
:update_sources: true
:verbose: true
:bulk_threshold: 1000
:sources: 
- http://gems.rubyforge.org
- http://gems.datamapper.org
:backtrace: false
:benchmark: false

All I needed to do was remove the - http://gems.datamapper.org line and poof, gem was working again. One quick gem update --system later and I was upgraded from gem 1.1.1 to 1.3.1 and installing the gems I needed.

Use datamapper sessions with merb & datamapper

Issue

Can't use merb sessions with datamapper & mysql, get back an error about needing an id on the text column or something (I had the error a couple of days ago.)

Solution

I suggest grabbing merb_datamapper svn source to fix this in. To do so make sure you have subversion installed on your machine (I'm assuming a Unix based machine here.)

  1. Checkout the source

     svn co http://svn.devjavu.com/merb/plugins/merb_datamapper
    
  2. Open up the affected file in your favourite editor (I use TextMate)

     cd merb_datamapper
     mate lib/merb/sessions/data_mapper_session.rb
    
  3. Find line 25 that contains

     `property :session_id, :text, :lazy => false, :key => true`
    

    and remove :text, :lazy => false to replace it with :string

     `property :session_id, :string, :key => true`
    

    Save and close the file, thats the editing done. Now to install the gem.

  4. Build the gem

     rake gem
    
  5. Install the gem

     sudo gem install pkg/merb_datamapper-0.5.gem
    

And you're away with the fix installed. Now just run merb to create your sessions table in the db. Hope this helped!