Caius Theory

Now with even more cowbell…

SmartOS Recovery mount /usbkey

Recently I managed to hose a box in a perfectly self-inflicted storm of idiocy. Imagine a SmartOS server with the following issues:

Needless to say, this caused a tiny issue in the server doing what it's supposed to. Luckily I had access to a KVM remote console for the box and the following worked.

I brought the machine up, choosing the second option for recovery at the grub menu. Waited for a login prompt, then logged in with root/root.

Realised quite quickly that /usbkey must be persisted on the zones zfs pool otherwise the configuration would be lost after shutdown, so imported the correct pool, created a directory to mount into and then mounted the zfs share.

zpool import zones
mkdir /usbkey
mount -F zfs zones/usbkey /usbkey

Setup DHCP interface in FreeBSD

Given a FreeBSD instance without a configured network interface that you'd like to configure, first check what the name of the interface you want to configure is with ifconfig. (Mine is em0 in this instance.)

Then we need to add the configuration telling services that we want to use DHCP for this interface, and setting up our default router (use your IP, not mine!) too:

cat >> rc.conf <<CONF
ifconfig_em0="DHCP"
default_router="192.168.1.1"
CONF

And then we need to start dhclient on the given interface:

service dhclient start em0

Hey presto, you should see dhclient finding a DHCP server and being handed an IP address for em0.

SoundCloud RSS Feeds

SoundCloud appears to have gained popularity in recent times for hosting podcasts on. As a consumer of their service they're pretty good at everything except having a visible RSS feed on a profile page for a show! If I want to listen to a show in my podcast app of choice, an RSS feed is the easiest way for me to achieve that.

Turns out SoundCloud do have RSS feeds, they're just well hidden and unfindable from the profile page itself. Thankfully, you can construct the URL for it from information on the profile page, and here's a bookmarklet that will do it for you:

javascript:var%20userURI;var%20metaTags=document.getElementsByTagName(
%22meta%22);for(var%20i=0;i<metaTags.length;i++){t=metaTags[i];if(
t.attributes[%22property%22]&&t.attributes[%22property%22].value==
%22al:ios:url%22){userURI=t.content;}}if(userURI){u=userURI.split(%22//%22)[1];
window.location=%22http://feeds.soundcloud.com/users/soundcloud:%22+u+
%22/sounds.rss%22;}

Or as a handy link to copy to your bookmarks bar. Simply click/run that when on a SoundCloud profile page and you'll be taken to the RSS Feed URL.

Compiling SmartOS for AMD processors

There's a few community-provided patches for SmartOS that enable KVM on AMD processors amongst other things, and given the HP Microserver has an AMD processor, that's quite useful for turning it into a better lab server. The main list of so called "eait" builds was hiccuping when I tried to download the latest, and all I could find was a 20140812T062241Z image here.

The source code for the eait builds is maintained at https://github.com/arekinath/smartos-live, and you can see the patches applied on top of the normal SmartOS master by going to https://github.com/arekinath/smartos-live/compare/joyent:master...eait.

So here's how to use SmartOS to compile a more up to date AMD-friendly Smartos!

  1. Grab the latest multiarch SmartOS image (which has to be used, or the compile will fail.) The latest at the time of writing was 4aec529c-55f9-11e3-868e-a37707fcbe86, so that's what I'll use.

     imgadm import 4aec529c-55f9-11e3-868e-a37707fcbe86
    
  2. Spin up a zone for us to build in (the Building SmartOS on SmartOS page has extra info about this):

     echo '{
       "alias": "platform-builder",
       "brand": "joyent",
       "dataset_uuid": "4aec529c-55f9-11e3-868e-a37707fcbe86",
       "max_physical_memory": 32768,
       "quota": 0,
       "tmpfs": 8192,
       "fs_allowed": "ufs,pcfs,tmpfs",
       "maintain_resolvers": true,
       "resolvers": [
         "8.8.8.8",
         "8.8.4.4"
       ],
       "nics": [
         {
           "nic_tag": "admin",
           "ip": "dhcp",
           "primary": true
         }
       ],
       "internal_metadata": {
         "root_pw": "password",
         "admin_pw": "password"
       }
     }' | vmadm create
    
  3. Login to the created zone:

     zlogin <uuid from `vmadm create` output>
    
  4. Update the image to the latest packages, etc:

     pkgin -y update && pkgin -y full-upgrade
    
  5. Install a few images we'll need to compile & package SmartOS:

     pkgin install scmgit cdrtools pbzip2
    
  6. Grab the source code of the fork containing the patches we want, from arekinath/smartos-live

     git clone https://github.com/arekinath/smartos-live
     cd smartos-live
    
  7. Optional: Edit src/Makefile.defs and change PARALLEL = -j$(MAX_JOBS) to PARALLEL = -j8 to do less at once. (Microserver only has a dual core CPU!)

  8. Copy the configure definition into the right place and start configuration:

     cp {sample.,}configure.smartos
     ./configure
    

    (You'll probably get asked to accept the java license during configuration, so keep half an eye on it)

  9. Once configure has completed (which doesn't take too long, 15 minutes or so), start building:

     gmake world && gmake live
    
  10. Once the build is successfully finished, time to package an iso & usb image:

    export LC_ALL=C
    tools/build_iso
    tools/build_usb
    

Hey presto, you've a freshly built AMD-friendly SmartOS build to flash to a USB key / put on your netboot server and boot your Microserver from!


References

Add to iCloud Reading List programmatically

One piece of a larger puzzle I'm trying to solve currently, was how to add a given URL to my Apple "Reading List" that is stored in iCloud and synced across all my OS X and iOS devices. More specifically, I wanted to add URLs to the list from my mac running Mavericks (10.9). I had a quick look at the Cocoa APIs and couldn't see anything in OS X to do this. (iOS has an API to do it from Cocoa-land it seems though.)

I figured Safari.app was the key to getting this done on OS X, given it has the ability itself to add the current page to the reading list, either via a keyboard command, a menu item, or a button in the address bar. One quick mental leap later, and I was wondering if the engineers at Apple had been nice enough to expose that via Applescript for me to take advantage of.

One quick stop in "Script Editor.app" later, and I had the Applescript dictionary open for Safari.app. Lo and behold, there is rather handily an Applescript command called "add reading list item", which does exactly what I want. It has a few different options you can call it with, depending on whether you want Safari to go populate the title & preview text, or if you want to specify it yourself at save-time.

As I want to be able to call this from multiple runtimes, I've chosen to save it as an executable, which leans on osascript to run the actual Applescript. And here it is:

#!/usr/bin/env osascript

on run argv
    if (count of argv) > 0
        tell app "Safari" to add reading list item (item 1 of argv as text)
    end if
end run

Save it as whatever you want (eg. add_to_reading_list), make it executable (chmod +x add_to_reading_list), and then run it with the URL you want saving as the first argument.

$ add_to_reading_list "http://caius.name/"
$ add_to_reading_list "http://google.com/"
# … etc …

(Adding support for specifying preview text and title is left as an exercise for the reader!)

Have fun reading later!

evil.rb

Here be hax. Don't ever do these. ;-)

Reduce local variables with instance_eval

Sometimes (usually in a one-liner) I want to do some work with a value without assigning it to a variable. Chucking an #instance_eval call in there will set self to the value, which saves having to assign it to a local value. Pretty much only used by me in one-off scripts or cli commands.

Good

start_date, end_date = ["24 Dec 2011", "23 Jan 2013"].map {|d| Date.parse(d) }
puts "#{start_date} to #{end_date} is #{(end_date - start_date).to_i} days"

Bad

puts ["24 Dec 2011", "23 Jan 2013"].map {|d| Date.parse(d) }
  .instance_eval { "#{first} to #{last} is #{(last - first).to_i} days" }

See, way less code! cough, cough

Bonus usage: Misdirection

I also dropped some instance_eval on our campfire bot at EmberAds to always blame one person, but without the code reading as such.

%w{Dom Mel Caius CBetta Baz}.sample.instance_eval do
  "(4V5A8F5T=&$`".unpack("u")[0]
end

That does not return one of the array elements as you might think it does from quickly scanning the code…

Set method-local variables in default arguments

You have a method and it takes one argument, which has a default value of nil specified. You then run into the situation where you need to know if nil was passed to the method, or if you're getting the default value of nil. You could change the default value to something you choose to be the "default value" and unlikely to be passed from elsewhere as the argument's value, and reset the parameter to nil after checking it, like this:

def output name=:default_value
  if name == :default_value
    name = "caius"
    default = true
  end

  "name: #{name.inspect} -- default: #{default.inspect}"
end

output() # => "name: \"caius\" -- default: true"
output("fred") # => "name: \"fred\" -- default: nil"

That's quite a lot of code added to the method just to find out if we passed a default value or not. And if we forget to reset the value when it's :default_value then we end up leaking that into whatever the method does with that value. We also have the problem that one day the program could possibly send that "default value" we've chosen as the actual parameter, and we'd blindly change it thinking it was set as the default value, not the passed argument.

Instead we could (ab)use the power of ruby, and have ruby decide to set default = true for us when, and only when, the variable is set to the default value.

def output name=((default=true); "caius")
  "name: #{name.inspect} -- default: #{default.inspect}"
end

output() # => "name: \"caius\" -- default: true"
output("fred") # => "name: \"fred\" -- default: nil"

As you can see, the output is identical. Yet we have no extra code inside the method to figure out if we were given the default value or not. And as a bonus to that, we no longer have to check for a specific value being passed and presume that is actually the default, and not one passed by the program elsewhere.

I posted this one in a gist a while back (to show Avdi it looks like), and people came up with some more insane things to do with it, including returning early, raising errors or even redefining the current method, all from the argument list! I'd suggest going to read them, it's a mixture of OMG HAHA and OMFG NO WAY WHYY?!?!.

Don't do this.

Don't do the above. No really, don't do them. Unless you're writing a one-off thing. But seriously, don't do them. :-D

Some Small Refactorings in Ruby

Here's a few things I refactor as I write code down initially. Not entirely convinced it's strictly refactoring, but it's how I amend from one pattern I see in a line or three of code into a different structure that I feel achieves the same result with cleaner or more concise code.

Multiple equality comparisons

Testing the equality of an object against another is fairly simple, just do foo == "bar". However, I usually try to test against multiple objects in a slightly different way. Your first thought might be that the easiest way is just to chain a series of == with the OR (||) operator.

foo == "bar" || foo == "baz" || foo == :sed || foo == 5

I much prefer to flip it around, think of the objects I'm testing against as a collection (Array), and then ask them if they contain the object I'm checking. And for that, I use Array#include?

["bar", "baz", :sed, 5].include?(foo)

(And if you're only testing against strings, you could use %w(bar baz) as a shortcut to create the array. Here's more ruby shortcuts.)

Assigning multiple items from a nested hash to variables

Occasionally I find myself needing to be given a hash of a hash of data (most recently, an omniauth auth hash) and assign some values from it to separate variables within my code. Given the following hash, containing a nested hash:

details = {
  uid: "12345",
  info: {
    name: "Caius Durling",
    nickname: "caius",
  },
}

Lets say we want to extract the name and nickname fields from details[:info] hash into their own local variables (or instance variables within a class, more likely.) We should probably handle the case of details[:info] not being a hash, and try not to read from it if that's the case - so we might end up with something like the following:

name = details[:info] && details[:info][:name]
nickname = details[:info] && details[:info][:nickname]

name # => "Caius Durling"
nickname # => "caius"

And then in the spirit of DRYing up our code, we see there's duplication in both lines in checking details[:info] exists (not actually that it's a hash, but hey ho, we rely on upstream to send us nil or a hash.) So we reduce it down using an if statement and give ourselves slightly less to type at the same time.

if (( info = details[:info] ))
  name = info[:name]
  nickname = info[:nickname]
end

name # => "Caius Durling"
nickname # => "caius"

Returning two values conditionally

Sometimes a method will end with a ternary, where depending on a condition it'll either return one or another value. If this conditional returns true, then the first value is returned. Otherwise it returns the second value. You could quite easily write it out as an if/else longer-form block too.

def my_method
  @blah == foo ? :foo_matches : :no_match
end

My brain finds picking the logic in this apart slightly harder mentally, than if I drop a return early bomb on the method. Then it reads more akin to how I'd think through the logic. Return the first value if this conditional returns true. Otherwise the method returns this second value. I think the second value being on a completely separate line helps me make this mental distinction quicker too.

So I'd write it this way:

def my_method
  return :foo_matches if @blah == foo
  :no_match
end

Returning nil or a value conditionally

Following on from the last snippet, but taking advantage of the ruby runtime a bit more, is when you're wanting to return a value if a conditional is true, or otherwise false. The easy way is to just write nil in the ternary:

def my_method
  @foo == :bar ? :foo_matches : nil
end

However, we know ruby returns the result of the last expression in the method. And that if a single line conditional isn't met, it returns nil from the expression. Combining that, we can rewrite the previous example into this:

def my_method
  :foo_matches if @foo == :bar
end

And it will still return nil in the case that @foo doesn't match :bar.

Returning a boolean

Sometimes you have a method that returns the result of a conditional, but it's written to return true/false in a conditional instead.

def my_method
  @foo == :bar ? true : false
end

The really easy refactor here is to just remove the ternary and leave the conditional.

def my_method
  @foo == :bar
end

And of course if you were returning false when the conditional evaluates to true, you can either negate the comparison (use != in that example), or negate the entire conditional result by prepending ! to the line.

Why I love DATA

In a ruby script, there's a keyword __END__ that for a long time I thought just marked anything after it as a comment. So I used to use it to store snippets and notes about the script that weren't really needed inline. Then one day I stumbled across the DATA constant, and wondered what flaming magic it was.

DATA is in fact an IO object, that you can read from (or anything else you'd do with an IO object). It contains all the content after __END__ in that ruby script file*. (It only exists when the file contains __END__, and for the first file ruby invokes though. See footnote for more details.)

How can we use this, and why indeed do I love this fickle constant? I mostly use it for quick scripts where I need to process text data, rather than piping to STDIN.

Given a list of URLs that I want to open in my web browser and look at, I could do the following for instance:

DATA.each_line.map(&:chomp).each do |url|
  `open "#{url}"`
end

__END__
http://google.com/
http://yahoo.com/

which upon running (on a mac) would open all the URLs listed in DATA in my default web browser. (For bonus points, use Launchy for cross-platform compatibility.) Really handy & quick/simple when you've got 500+ URLs to open at once to go through. (I once had a job that required me to do this daily. Fun.)

Or given a bunch of CSV data that you just want one column for, you could reach for cut or awk in the terminal, but ruby has a really good CSV library which I trust and know how to use already. Why not just use that & DATA to pull out the field you want?

require "csv"

CSV.parse(DATA, headers: true).each do |row|
  puts row["kName"]
end

__END__
kId,kName,kURL
1,Google UK,http://google.co.uk
2,"Yahoo, UK",http://yahoo.co.uk
# >> Google UK
# >> Yahoo, UK

I find when the data I want to munge is already in my clipboard, and I can run ruby scripts directly from text editors without having to save a file, it saves having to write the data out to a file, have ruby read it back in, etc just to do something with the data. I can just write the script reading from DATA, paste the data in and run it. Which also lets me run it iteratively and build up a slightly more complex script that I don't want to keep. Then do what I need with the output and close the file without saving it.

* technically DATA is an IO handler to read __FILE__, which has been wound forward to the start of the first line after __END__ in the file. And it only exists for the first ruby file to be invoked by the interpreter.

cat > tmp/data.rb <<RUBY
p DATA.read
__END__
data.rb
RUBY

ruby tmp/data.rb
# => "data.rb\n"

cat > tmp/data-require.rb <<RUBY
require "./tmp/data"
RUBY

ruby tmp/data-require.rb
# => /Users/caius/tmp/data.rb:1:in `<top (required)>': uninitialized constant DATA (NameError)

And because it's a file handle pointing at the current file, you can rewind it and read the entire ruby script into itself…

$ ruby tmp/readself.rb 
DATA.rewind
print DATA.read

__END__
something goes here

Geolocation in nginx

Sometimes you need to have a rough idea of where your website visitor is located. There's many ways to geolocate them, but if you just want to go to country level then MaxMind have free geo databases available to help you. When we needed to do this quickly on-the-fly at EmberAds, we came up with the trifle gem, which supports ipv4 and ipv6 lookups.

Recently I was searching for something else to do with nginx and ran across a mailing list thread about using the maxmind database with nginx's HTTP Geo module and do the lookup directly in nginx itself. Finally got a chance to sit down and work out the logistics of doing this. I've done this on an ubuntu 12.04 box, with the expected config file layouts that come with ubuntu.

Run the following on your server (as someone with write access to nginx config files):

# Generate the text file for nginx to import
perl <(curl -s https://raw.github.com/nginx/nginx/master/contrib/geo2nginx.pl) \
< <(zip=$(tempfile) && \
curl -so $zip http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip \
&& unzip -p $zip) > /etc/nginx/nginx_ip_country.txt

# Tell nginx to work out the IP country and store in variable
echo 'geo $IP_COUNTRY {
  default --;
  include /etc/nginx/nginx_ip_country.txt;
}' > /etc/nginx/conf.d/ip_country.conf

Now go find the http block for the vhost you want to have the header passed to, and assuming it's passenger, add the following:

# http {
  # server_name freddy.com;
  passenger_set_cgi_param HTTP_X_IP_COUNTRY $IP_COUNTRY;
# }

(If you don't use passenger, look at the docs for proxy_pass_header or fastcgi_pass_header to see which you'll require for your setup.)

Reload nginx, and behold, request.env["HTTP_X_IP_COUNTRY"] (assuming a rack app running under ruby) will be a two letter country code, or "--".

Unfortunately this is IPv4 only currently, there's a thread on the nginx mailing list from November 2012 saying IPv6 support should be coming on the v1.3 branch of nginx, but with no known ETA. So currently for IPv6 support, take a look at EmberAds' trifle gem instead.

Mounting Harman Kardon Soundsticks on the wall

Having recently moved my Soundstick III's into the front room, I've been thinking of a way to wall mount them safely to free up table room. Googling eventually turned up just one person who has previously documented mounting his soundsticks on the wall, using 22mm plumbing clips (intended for 22mm central heating pipes).

A quick scrounge round the local Homebase this afternoon yielded a pack of similar clips, 5x 22mm push clips for £1.99. Having just fitted the speakers to the wall, they're nice and secure (providing no-one hangs on them, which they shouldn't do), fairly neat and simple to fit.

I've left the subwoofer on the floor under the table, and only mounted the "sticks" (tweeters) on the wall, one each side of the mirror over our dining table. I can then conveniently run the cables to the "sticks" behind the mirror and keep it looking neater.

I affixed the clips to the wall, one either side of the mirror with enough space for the speaker to sit without overlapping the mirror. Mostly just held the stick up and guessed at where to put the clip, but it looks ok.

Speaker clip without speaker in it

Then I bent the speaker stand backwards behind the stick (don't worry, it's on a hinge!) and threaded the cable through the middle of the ring to make it sit flusher against the wall.

Speaker ring bent behind the unit

And then it was just a case of easing the speaker ring into the clip, with the clip at the top of the ring (picture below if you can't visualise that easily). I think the rings are more like 24-25mm so it takes a bit of easing to get them in there. Once it's in there's no play in the clip for it to wiggle out though, even though it's plastic. I tried not to fatigue the clip arms too much wiggling it in there as well, so as to minimise the chance of it failing over time. (Something to check periodically!)

Close up of speaker unit in clip

Lastly, I just zip tied the cables together behind the mirror, and ran them down vertically from the middle to the floor and plugged them in. Sounds fantastic, and due to being plugged into an Airport Express, anything compatible can stream audio through them wirelessly. Fabulous darling!

(Click any image for a larger version)