Caius Theory

Now with even more cowbell…

Potty Training YAML

Ran into a problem today where I have a class with a few attributes on it, but I only want a certain three of those attributes to appear in the YAML dump of a class instance.

Diving straight into a code example–lets say we have a Contact class, and we only want to dump the name, email and website attributes.

require "yaml"

class Contact
  attr_accessor :name, :email, :website, :telephone

  # helper method to make setting up easy
  def initialize(params={})
    params.each do |key, value|
      meffod = "#{key.to_s}="
      send(meffod, value) if respond_to?(meffod)
    end
  end
end

# And create an instance for us to play with
caius = Contact.new(
  :name => "Caius",
  :email => "dev@caius.name",
  :website => "http://caius.name/",
  :telephone => "12345"
)

As we'd expect when dumping this, all instance variables get dumped out:

print caius.to_yaml
# >> --- !ruby/object:Contact 
# >> email: dev@caius.name
# >> name: Caius
# >> telephone: "12345"
# >> website: http://caius.name/

Initially I tried to override to_yaml and unset the instance variables I didn't want showing up, but that just made them show up empty. After digging around a bit more, I happened across the Type Families page in the yaml4r docs, which right at the bottom mentions to_yaml_properties.

Turns out to_yaml_properties returns an array of instance variable names (as strings) that should be dumped out as part of the object. A quick method definition later, and we're only dumping the variables we want. (See my Ruby Shortcuts post if you don't know what %w() does)

class Contact
  def to_yaml_properties
    %w(@name @email @website)
  end
end

And now we dump the class, expecting only the three attributes to be outputted:

print caius.to_yaml
# >> --- !ruby/object:Contact 
# >> name: Caius
# >> email: dev@caius.name
# >> website: http://caius.name/

Success!

Read Later in a keystroke

I use a wonderful service for saving text to be read later, instapaper.com. It's gotten more wonderful as time has gone on and other applications/service's have gained the ability to save links/articles/webpages there for me to pick up later.

For instance, I'm out and about checking twitter on my iPhone using tweetie and someone tweets a link. Rather than wait for it to load and having to read it then and there I can just hit "Read Later" and it's saved in my instapaper account for me to read as and when I choose to. Recently the legendary mac feed reader NetNewsWire gained this ability too.

There's a few ways to send a feed item to instapaper from within NNW. Firstly you can right-click and click "Send to Instapaper".

Send to Instapaper from contextual menu
View Original on Flickr

Secondly there's a menu item for it in the News menu, which also provides my chosen way of instapapering an item—the keyboard shortcut! ⌃P (control-P).

Send to Instapaper from News menu
View Original on Flickr

So, in NNW I'm happily sending stuff to instapaper with the handy ⌃P shortcut, but that doesn't exist in the third place I mark things to read later–Safari! Up until now I've been using the standard "Read Later" bookmarklet that instapaper.com provides, and it's got a spot on my Bookmarks Bar so I can easily click it.

That doesn't really help with the fact I'm hitting ⌃P in NNW, and it doesn't work in Safari. Quite often I noticed myself hitting the key combination in Safari and wondering for a split second why it wasn't sending the item to instapaper. Then the solution hit me!

In OS X you can setup (and/or override) menu items with custom key combinations! Why hadn't I remembered this before. Because the "Read Later" bookmark(let) is nested under the Bookmarks menu, it is a menu item! A quick trip into the Keyboards Prefpane in System Preferences and a new binding later and voilâ, "Read Later" in Safari is bound to ⌃P and I can use it in both Safari and NNW.

Filling in the form to bind the keyboard shortcut
View Original on Flickr

Capitalise "ringer" on the iPhone Volume Bezel

Backstory: Got myself a first generation iPhone second hand and unlocked it to work on my existing T-Mobile (Official iPhone network in the UK is O2.) Noticed after a week or so of owning it that when you change the volume on the phone, the bezel that comes up says "ringer" across the top. But when you have headphones plugged in, it says "Headphones". (Note the capitalisation difference.)

Now I'm not usually bothered by stuff like this (honest!) but as soon as I'd noticed the "bug", I couldn't help but think of it everytime I changed the volume, whether I was looking at the screen or not. Seeing as I'm running a jailbroken phone, and therefore have SSH access to it, I figured the string would be defined in a .strings file somewhere in the /System folder. And I'd be able to change it!

Fast-forward a few months and I install the iPhone OS 3.0 update (jailbroken of course), and finally decide to turn the phone's SSH server on and go looking for the setting. To do so I figured I'd just need grep installed on the phone - I could copy the file itself to my mac and edit it there.

So I connect to the phone, have a poke around the filesystem and then start a search to find the correct file:

# On the iPhone
$ cd /System/Library/
$ grep -r "ringer" *
Binary file CoreServices/SpringBoard.app/English.lproj/SpringBoard.strings matches
Binary file CoreServices/SpringBoard.app/M68AP.plist matches
Binary file CoreServices/SpringBoard.app/SpringBoard matches
Binary file Frameworks/CFNetwork.framework/CFNetwork matches
Binary file Frameworks/CFNetwork.framework/da.lproj/Localizable.strings matches
Binary file Frameworks/CFNetwork.framework/no.lproj/Localizable.strings matches
Binary file Frameworks/Foundation.framework/da.lproj/URL.strings matches

At which point I stopped the grep search (^C) because I know the home screen of the iPhone is the SpringBoard.app, so I figured it would be in the file SpringBoard.app/English.lproj/SpringBoard.strings. Making sure to have SSH enabled on your mac, a simple scp CoreServices/SpringBoard.app/English.lproj/SpringBoard.strings user@your_mac.local: later and the file is sat in my home folder on my mac.

Switching to the mac, now I try and open the file with TextMate, only to realise its in binary format. I need it in the nice XML format to edit it, so a quick google later and I've found a hint on MacOSXHints telling me how to convert from binary to xml plist format.

# On the mac
$ plutil -convert xml1 SpringBoard.strings

Then opening the file in TextMate was a bit more successful! I can actually understand what its defining now. Search through the file for "ringer" and I found the following lines:

<key>RINGER_VOLUME</key>
<string>ringer</string>

Change the "ringer" to "Ringer" between the <string> and my editing work is complete! Yes, it really is that easy to edit an interface string that is defined in a .string. Now I just need to convert the file back to binary, and copy it back to the phone. Converting back to binary file is one line, just change the xml1 in the previous command to binary1.

# On the mac
$ plutil -convert binary1 SpringBoard.strings

And then scp it back to the phone, make a backup of the existing file, and overwrite the existing file with the new one I've edited:

# On the iPhone
$ cd ~
$ scp user@mac_name.local:SpringBoard.strings .
$ cd /System/Library/CoreServices/SpringBoard.app/English.lproj/
$ mv SpringBoard.strings SpringBoard.strings.bak
$ cp ~/SpringBoard.strings SpringBoard.strings

And then restart the phone, either in the usual manner or just run reboot on the phone via SSH. Lo and behold once its rebooted and I changed the volume, it read "Ringer"!

Screenshot of Volume bezel

Quantum Javascript Bug

So I've got some js I've written to update a couple of <select> lists in a form, and it was all working fine for me (under Safari.) John happened to mention it wasn't working for him under Firefox, so I fired up Firefox and took a look. Could reproduce it perfectly, changing the first popup was populating the second one, but then wasn't selecting the right value from the list.

Having no idea what was happened I figured I'd enable firebug and watch it execute to figure out what was happening. Enabled firebug, reloaded the page, selected from the first popup… and voila! It updated the second one and selected the correct row! WTF!!!

Turned firebug off and it didn't work, turned it back on and it worked. Figured it might be something buggy in the Firefox 3.0.5 js runtime, so I grabbed a copy of the new beta 3.5 and tried it in there—still failed to update the page as it should.

Then started poking around the javascript code, the function that was seemingly failing to run was being triggered by a setTimeout() call set to 1 second. We figured it might be the timing causing it, so started playing around with the time, tried anything from ½ a second up to 4 seconds but still no joy in firefox with firebug turned off.

Then John went looking for the javascript errors in firefox (with firebug off) and discovered that it was throwing an error because window.console didn't exist. All of a sudden it made perfect sense! Safari has window.console.log() for writing to the console log, as does firebug. But of course firefox without firebug doesn't!

So the function was just exiting on that error. It was very weird initially to have it work perfectly as soon as the developer tools were enabled!

Validating Data with Regular Expressions in Ruby

I happened to be sent a link to the OWASP paper on Rails Security recently and started reading it. Partway in there's a section on Regular Expressions, which opens with the following line:

A common pitfall in Ruby's regular expressions is to match the string's beginning and end by ^ and $, instead of \A and \z.

Now I've never used \A and \z in my regular expressions to validate data, I've only ever used ^ and $ assuming they matched the start and end of the string. This becomes an issue with validating data in rails, because %0A (\n URL encoded) is decoded by rails before passing the string to your model to validate.

Testing our expectations

Lets say we want to validate the string as a username for our app. A username is 5 characters long and consists only of lowercase letters.

regex = /^[a-z]{5}$/

First we make sure it matches the data we want it to:

"caius".validate(regex) # => true

Excellent, that validated. Now we'll try a shorter string, which we expect to fail.

"cai".validate(regex) # => false

Once more, it behaves how we expected it to. The shorter string was rejected as we wanted it to be. Now, what happens if we test a string with a newline character in it? We'll make sure the data before the \n is valid, and then add some more data after the newline.

"caius\nfoo".validate(regex) # => true

Uh oh! That validated and would've been saved as a username?!

Lets have a look at exactly what's happening there, the $ matches the \n character, so the regex is only matching the first 5 characters of the string, and just ignores anything after the \n. As it turns out, this is exactly what we've asked the regex to match, but we didn't want this behaviour.

Now you might be thinking, "So what? someone can have a username with a newline in it." For starters this will probably display weirdly anywhere you use their username, but more importantly it opens your application to an injection attack. Suppose they took advantage of this by setting their username to include some javascript on the page which stole your login cookie and sent it to them. You view their account in the admin section and oh no! They can login as your admin account and do what they want.

Simple example of this is just having it output an alert dialog. (This is actually the code I'll use to test an application as its not malicious, but blindingly obvious if the javascript is executed or not.)

"caius\n<script>alert('hello')</script>".validate(regex) # => true

Ok, so that was the result we were expecting this time, although it's still not the outcome we wanted. Anytime their username is viewed (providing you aren't escaping the data to HTML entities) you'll see the following:

javascript alert dialog

The Solution

Having realised from our testing above that ^$ matches the beginning/end of a line in ruby not the beginning and end of a string, I hear you cry, "How do we make sure we're matching the entire string?!"

The answer is pretty simple. Just swap out ^$ for \A\z. Lets go ahead and try this with the same data as we have above, but with the modified regular expression.

new_regex = /\A[a-z]{5}\z/
"caius".validate(new_regex) # => true

That's a good start, the valid string still matches.

"cai".validate(new_regex) # => false

Looks like it's going well, invalid string is invalid.

"caius\nfoo".validate(new_regex) # => false

Oh Excellent! It's validating this one correctly now.

And just for consistency, lets test it with a more likely attack string.

"caius\n<script>alert('hello')</script>".validate(new_regex) # => false

Fantastic! We've fixed the security hole in our validation of the user's username.


If you want to actually run the code above you'll need the following at the start of the ruby script to patch the validate method into String.

class String
  def validate regex
    !self[regex].nil?
  end
end

Update: I had \Z in the new_regex rather than the \z it should've been. Thanks Ciarán.

Install Mysql Gem on Leopard

So, I keep having to reinstall mysql5 and rubygems from time to time for various reasons. I always install mysql5 through MacPorts as a dependency for the php5 port (along with various other bits for the LA*P stack).

sudo port install php5 +mysql5 +pear +readline +sockets +apache2 +sqlite

Once this is installed then I have mysql and can setup my databases, etc.

Ignoring the rest of the LAMP stack, I then need to connect Ruby to the Mysql I just installed through MacPorts. Its quite simple to do, once you know the right argument to pass to it. The easiest way is to just tell it where the mysql5_conf file is and let it figure out the rest for itself.

sudo gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5

Hopefully this will save me 10 minutes of googling next time I need to do this!

Update 2009-01-21

I'm an idiot and typed the gem install command by hand, and ended up with --with-mysql-conf instead of --with-mysql-config. Updated now.

Update 2009-10-19

On Snow Leopard I needed to tell rubygems to install the gem as a 64-bit binary. Hattip to Philipp

sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- \
  --with-mysql-config=/opt/local/bin/mysql_config5

Fix Mail.app crashing after 10.5.6 upgrade

When you upgrade to Mac OS 10.5.6, Mail.app might start crashing a few seconds after starting due to the GPG Bundle.

The solution is to grab the updated version of the GPG bundleGPGMail_d55_Leopard.dmg

Installing Ubuntu on an iMac G3

I decided to install ubuntu onto my iMac G3450Mhz G3, 768mb ram, 20GB Hard Drive to play around with. Only problem was it would boot so far, then just stop at a black screen. In googling the fix, the blog post that contains the fix is slightly outdated and 100% 404.

Here is the fix, updated for Ubuntu 6.10 Desktop PPC:

  1. When the screen goes black, drop to the console

     Control - Option - F2
    

    (if you need to log in use the name ubuntu to log in.)

     $ sudo nano /etc/X11/xorg.conf
    
  2. Change the frequencies in monitor section as follows:

     Section “Monitor”
         Identifier “Generic Monitor”
         Option “DPMS”
         HorizSync 60-60
         VertRefresh 43-117
     EndSection
    
  3. After the changes then type control-o, return (to accept the filename), then control-x (save and exit nano)
  4. Restart X by running the following:

     sudo killall gdm && sudo /etc/init.d/gdm start
    

Removing non-existent source from rubygems

I just came to move some ruby scripts onto my mac mini, and to do so I needed to install a couple of gems. Now I realised I hadn't installed or updated rubygems on the machine for a while, so I figured it was best to update gem before installing the gems I wanted. Easier said than done.

At some point in the past I had added http://gems.datamapper.org as a source to rubygems. Since then the datamapper project has discontinued using this gem source to serve up gems, so I was getting the following output:

mm:daemons caius$ sudo gem update --system
Updating installed gems
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 404 fetching http://gems.datamapper.org/yaml

Eeek! I can't update because the source no longer exists. So I figured I'd remove the source before updating, that should work right? Wrong. It updates the sources before removing the source from the config it would appear.

mm:daemons caius$ sudo gem sources
** CURRENT SOURCES ***

http://gems.rubyforge.org
http://gems.datamapper.org

mm:daemons caius$ sudo gem sources -r http://gems.datamapper.org
Bulk updating Gem source index for: http://gems.rubyforge.org/
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 404 fetching http://gems.datamapper.org/yaml

Oh balls. So how do I remove the source without updating it first. I need to update it to remove it, but to remove it I need to update from it. Gotta love catch 22s!

I remembered that gem install has an option not to update sources, --no-update-sources. So I figured thats gotta work when removing a source as well, but it doesn't.

mm:daemons caius$ sudo gem sources -r http://gems.datamapper.org --no-update-sources
ERROR:  While executing gem ... (OptionParser::InvalidOption)
    invalid option: --no-update-sources

Oh crap. Now what do I do? Take my usual tactic and google for a hint of course! I'd considered trying to find where the gem config was and remove the source by hand, but I figured that wouldn't be that simple. After hitting a couple of sites that weren't relevant I ended up on the edge of complexity where he mentions the command nano ~/.gemrc. Which made me wonder if that file contains the sources.

mm:daemons caius$ cat ~/.gemrc
--- 
:update_sources: true
:verbose: true
:bulk_threshold: 1000
:sources: 
- http://gems.rubyforge.org
- http://gems.datamapper.org
:backtrace: false
:benchmark: false

All I needed to do was remove the - http://gems.datamapper.org line and poof, gem was working again. One quick gem update --system later and I was upgraded from gem 1.1.1 to 1.3.1 and installing the gems I needed.

Use datamapper sessions with merb & datamapper

Issue

Can't use merb sessions with datamapper & mysql, get back an error about needing an id on the text column or something (I had the error a couple of days ago.)

Solution

I suggest grabbing merb_datamapper svn source to fix this in. To do so make sure you have subversion installed on your machine (I'm assuming a Unix based machine here.)

  1. Checkout the source

     svn co http://svn.devjavu.com/merb/plugins/merb_datamapper
    
  2. Open up the affected file in your favourite editor (I use TextMate)

     cd merb_datamapper
     mate lib/merb/sessions/data_mapper_session.rb
    
  3. Find line 25 that contains

     `property :session_id, :text, :lazy => false, :key => true`
    

    and remove :text, :lazy => false to replace it with :string

     `property :session_id, :string, :key => true`
    

    Save and close the file, thats the editing done. Now to install the gem.

  4. Build the gem

     rake gem
    
  5. Install the gem

     sudo gem install pkg/merb_datamapper-0.5.gem
    

And you're away with the fix installed. Now just run merb to create your sessions table in the db. Hope this helped!