Caius Theory

Now with even more cowbell…

Bash script setup

Recently I've been writing a bunch of bash scripts for various things. As some up-front safety checks I've taken to opening every script with the following:

#!/usr/bin/env bash

[[ "$TRACE" ]] && set -o xtrace
set -o errexit
set -o nounset
set -o pipefail
set -o noclobber

Other things I'm also trying to be good about doing:

And some useful reading I ran across in my quest to level up bash-scripts:

Geolocation in nginx

Sometimes you need to have a rough idea of where your website visitor is located. There's many ways to geolocate them, but if you just want to go to country level then MaxMind have free geo databases available to help you. When we needed to do this quickly on-the-fly at EmberAds, we came up with the trifle gem, which supports ipv4 and ipv6 lookups.

Recently I was searching for something else to do with nginx and ran across a mailing list thread about using the maxmind database with nginx's HTTP Geo module and do the lookup directly in nginx itself. Finally got a chance to sit down and work out the logistics of doing this. I've done this on an ubuntu 12.04 box, with the expected config file layouts that come with ubuntu.

Run the following on your server (as someone with write access to nginx config files):

# Generate the text file for nginx to import
perl <(curl -s https://raw.github.com/nginx/nginx/master/contrib/geo2nginx.pl) \
< <(zip=$(tempfile) && \
curl -so $zip http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip \
&& unzip -p $zip) > /etc/nginx/nginx_ip_country.txt

# Tell nginx to work out the IP country and store in variable
echo 'geo $IP_COUNTRY {
  default --;
  include /etc/nginx/nginx_ip_country.txt;
}' > /etc/nginx/conf.d/ip_country.conf

Now go find the http block for the vhost you want to have the header passed to, and assuming it's passenger, add the following:

# http {
  # server_name freddy.com;
  passenger_set_cgi_param HTTP_X_IP_COUNTRY $IP_COUNTRY;
# }

(If you don't use passenger, look at the docs for proxy_pass_header or fastcgi_pass_header to see which you'll require for your setup.)

Reload nginx, and behold, request.env["HTTP_X_IP_COUNTRY"] (assuming a rack app running under ruby) will be a two letter country code, or "--".

Unfortunately this is IPv4 only currently, there's a thread on the nginx mailing list from November 2012 saying IPv6 support should be coming on the v1.3 branch of nginx, but with no known ETA. So currently for IPv6 support, take a look at EmberAds' trifle gem instead.

Install capybara-webkit gem on Ubuntu

Dear future Caius searching for this issue,

The apt package you need to install to use the capybara-webkit rubygem on ubuntu (tested on 10.04 and 11.10) is libqt4-dev. That is, to gem install capybara-webkit, you need to run aptitude install libqt4-dev.

Yours helpfully,
Past Caius

at(1) on OS X

I recently came across the at(1) command, and wondered why it wasn't executing jobs I gave it on my machine. Had a poke around the man pages, and discovered in atrun(8) that by default launchd(8) has the atrun entry disabled.

To enable it (and have at jobs fire) you simply need to run the following command once:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist

Personally I've taken to using this to sleep my machine after a custom amount of time, mainly because my alarm clock/sleep timer of choice (Awaken) can't handle playing Spotify for x minutes and then sleeping the machine. The following command puts the machine to sleep, which (quite effectively) silences spotify.

echo "osascript -e 'tell app \"Finder\" to sleep'" | at 1:00am

See the at(1) manpage for how to specify the time, but as I'm only ever scheduling it on the same day (usually 20 minutes or so in advance), just passing the time works fine.

Ignore .gitignore in Git

Recently I ran into an issue where I was working on a project which had files I wanted git to ignore, but I didn't want to commit a .gitignore file into the project. In case you don't know, any files matching a pattern in .gitignore in a git repository are ignored by git. (Unless the file(s) have already been committed, then they need removing from git before they are ignored.)

Initially I figured I could just throw the patterns I needed excluded into my global ~/.gitignore, but quickly realised that I needed files matching these patterns to show up in other git repos, so going the global route really wasn't an option. After some thought I wondered if you could make git ignore .gitignore, whilst still getting it to ignore files matching the other patterns in the .gitignore.

Lets create a new empty repo to test this crazy idea in:

$ mkdir foo
$ cd foo
$ git init
Initialized empty Git repository in /Volumes/Brutus/Users/caius/foo/.git/

And create a couple of files for us to play with:

$ touch bar
$ touch baz

Ignore one of the files so we can check other matches are still ignored later on:

$ echo "baz" >> .gitignore
$ git status
# On branch master
#
# Initial commit
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       .gitignore
#       bar
nothing added to commit but untracked files present (use "git add" to track)

Ok so far, but we can still see .gitignore in git, so now for the crazy shindig, ignore the ignore file:

$ echo ".gitignore" >> .gitignore 

Lets see if it worked, or if we can still see our .gitignore:

$ git status
# On branch master
#
# Initial commit
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       bar
nothing added to commit but untracked files present (use "git add" to track)

And lets just double-check that .gitignore and baz still exist on the filesystem:

$ ls -a
.  ..  .git  .gitignore  bar  baz

Fantastic! Turns out adding ".gitignore" to .gitignore works perfectly. The file is still parsed by git to ignore everything else too, so it does exactly what I needed in this instance.

Capitalise "ringer" on the iPhone Volume Bezel

Backstory: Got myself a first generation iPhone second hand and unlocked it to work on my existing T-Mobile (Official iPhone network in the UK is O2.) Noticed after a week or so of owning it that when you change the volume on the phone, the bezel that comes up says "ringer" across the top. But when you have headphones plugged in, it says "Headphones". (Note the capitalisation difference.)

Now I'm not usually bothered by stuff like this (honest!) but as soon as I'd noticed the "bug", I couldn't help but think of it everytime I changed the volume, whether I was looking at the screen or not. Seeing as I'm running a jailbroken phone, and therefore have SSH access to it, I figured the string would be defined in a .strings file somewhere in the /System folder. And I'd be able to change it!

Fast-forward a few months and I install the iPhone OS 3.0 update (jailbroken of course), and finally decide to turn the phone's SSH server on and go looking for the setting. To do so I figured I'd just need grep installed on the phone - I could copy the file itself to my mac and edit it there.

So I connect to the phone, have a poke around the filesystem and then start a search to find the correct file:

# On the iPhone
$ cd /System/Library/
$ grep -r "ringer" *
Binary file CoreServices/SpringBoard.app/English.lproj/SpringBoard.strings matches
Binary file CoreServices/SpringBoard.app/M68AP.plist matches
Binary file CoreServices/SpringBoard.app/SpringBoard matches
Binary file Frameworks/CFNetwork.framework/CFNetwork matches
Binary file Frameworks/CFNetwork.framework/da.lproj/Localizable.strings matches
Binary file Frameworks/CFNetwork.framework/no.lproj/Localizable.strings matches
Binary file Frameworks/Foundation.framework/da.lproj/URL.strings matches

At which point I stopped the grep search (^C) because I know the home screen of the iPhone is the SpringBoard.app, so I figured it would be in the file SpringBoard.app/English.lproj/SpringBoard.strings. Making sure to have SSH enabled on your mac, a simple scp CoreServices/SpringBoard.app/English.lproj/SpringBoard.strings user@your_mac.local: later and the file is sat in my home folder on my mac.

Switching to the mac, now I try and open the file with TextMate, only to realise its in binary format. I need it in the nice XML format to edit it, so a quick google later and I've found a hint on MacOSXHints telling me how to convert from binary to xml plist format.

# On the mac
$ plutil -convert xml1 SpringBoard.strings

Then opening the file in TextMate was a bit more successful! I can actually understand what its defining now. Search through the file for "ringer" and I found the following lines:

<key>RINGER_VOLUME</key>
<string>ringer</string>

Change the "ringer" to "Ringer" between the <string> and my editing work is complete! Yes, it really is that easy to edit an interface string that is defined in a .string. Now I just need to convert the file back to binary, and copy it back to the phone. Converting back to binary file is one line, just change the xml1 in the previous command to binary1.

# On the mac
$ plutil -convert binary1 SpringBoard.strings

And then scp it back to the phone, make a backup of the existing file, and overwrite the existing file with the new one I've edited:

# On the iPhone
$ cd ~
$ scp user@mac_name.local:SpringBoard.strings .
$ cd /System/Library/CoreServices/SpringBoard.app/English.lproj/
$ mv SpringBoard.strings SpringBoard.strings.bak
$ cp ~/SpringBoard.strings SpringBoard.strings

And then restart the phone, either in the usual manner or just run reboot on the phone via SSH. Lo and behold once its rebooted and I changed the volume, it read "Ringer"!

Screenshot of Volume bezel

Automatically Deploying Website From Remote Git Repository

Before I start, I'll just quickly run through where I put stuff on my server. Apache logs and config are in the ubuntu default folders: /var/log/apache2 and /etc/apache2/ respectively.

Websites:  /home/caius/vhosts/<domain name>/htdocs
Git Repos: /home/caius/git/<domain name>.git

So I have a git repo locally, ~/projects/somesite.com/, and want to deploy it to my webserver. I'll keep the git repo in ~/git/ and set it up so that when I push to the repo (over ssh) it will automatically checkout the new changes into the website's htdocs folder.

I'm assuming DNS is already setup (or I've used ghost to map it locally.) And that I've setup the virtualhost in apache pointing at /home/caius/vhosts/somesite.com/htdocs and reloaded apache so the config is in place.

Remote Machine

We create a bare git repo, then point the working tree at the docroot of our website. This means all the git stuff is kept in the somesite.git folder, but the files themselves are checked out to the website's folder. Then we setup a post-receive hook to update the worktree folder after new changes have been pushed to the repo.

$ cd git
$ mkdir somesite.git
$ cd somesite.git/
$ git init --bare
Initialized empty Git repository in /home/caius/git/somesite.git/
$ git --bare update-server-info
$ git config core.worktree /home/caius/vhosts/somesite.com/htdocs
$ git config core.bare false
$ git config receive.denycurrentbranch ignore
$ cat > hooks/post-receive
#!/bin/sh
git checkout -f
^D
$ chmod +x hooks/post-receive

Local Machine

And now on the client machine we add the remote repo as a git remote, and then push to it.

$ git remote add web ssh://myserver/home/caius/git/somesite.git
$ git push web +master:refs/heads/master
Counting objects: 3, done.
Writing objects: 100% (3/3), 229 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://myserver/home/caius/git/somesite.git
 * [new branch]      master -> master

All Done

And now if you go to somesite.com you'll see the contents of your git repo there. (somesite.com is just an example url though, I don't actually own it!)

Helpful URLs

Find shell commands with which

So I have this command in my $PATH, apachectl. Because I'm on a mac and I've installed apache2 through MacPorts, the command that gets found first is my macports install in /opt. Up until now I've always known that which apachectl will find that location, but to find any other locations of apachectl I'd usually use locate and egrep together.

Here's my original workflow, lets find the location of the apachectl being called when I don't specify a path.

Julius:~ caius$ which apachectl
/opt/local/apache2/bin/apachectl

Simple enough. Now lets figure out what other locations there's an apachectl installed at.

Julius:~ caius$ locate apachectl | egrep "\/apachectl$"
/opt/local/apache2/bin/apachectl
/opt/local/var/macports/software/apache2/2.2.11_0+darwin_9/opt/local/apache2/bin/apachectl
/usr/sbin/apachectl

Right, so now I know where else a command exists in the filesystem called apachectl, but I don't know if any of those is in my $PATH, or what order they come in when searching through my $PATH. In this (old) workflow I'd have compared them to my $PATH manually as there's so few of them.

So I noticed Ali googling for the which man page on IRC, and (quite stupidly) poked fun at him for doing so. I then swallowed my ego and actually followed the link to the man page, and boy was I glad I did. Just shows with even a fairly simple command like which, you sure don't know everything!

What I discovered was that which has a single flag you can pass it, -a. From the man page:

-a     print all matching pathnames of each argument

Right. So that locate | grep command plus manually figuring out what is in my $PATH is really hard work then. which -a should give us the same results, but a lot faster and with a lot less manual thought.

Julius:~ caius$ which -a apachectl
/opt/local/apache2/bin/apachectl
/usr/sbin/apachectl

And hey presto, yet another useful bit of bash knowledge for me, thanks to Ali not being afraid to RTFM!

Migrating Rubygems to Ruby 1.9.x

So I just installed ruby 1.9.1 through MacPorts and wanted to easily migrate my rubygems across from 1.8 to see which ones would fail to install.

Thought about it for a while, then came up with the following bash one-liner to do it:

gem list | grep "(" | awk '{ print $1 }' | xargs -L 1 gem1.9 install

NB: Installing Ruby 1.9.1 through macports sudo port install ruby19 means I get ruby1.9, gem1.9 and rake1.9 installed alongside my usual 1.8 ruby, gem and rake.

That grabs the list of installed gems from gem, searches for lines containing "(" so it only grabs the gem names, spits out the first section of the line, which is the name of the gem, and finally calls gem1.9 install for each line via xargs -L 1. Make sure to run it as root or prefix gem1.9 with sudo. (Or let it install in your home folder, but I hate that.)

From my quick run of the above snippet, 75% of my gems installed (73 out of 98) and the other few that failed to install were ones like Hpricot that require native extensions compiling. You can see the entire list of failures and successes of the gems in this pastie

Install Mysql Gem on Leopard

So, I keep having to reinstall mysql5 and rubygems from time to time for various reasons. I always install mysql5 through MacPorts as a dependency for the php5 port (along with various other bits for the LA*P stack).

sudo port install php5 +mysql5 +pear +readline +sockets +apache2 +sqlite

Once this is installed then I have mysql and can setup my databases, etc.

Ignoring the rest of the LAMP stack, I then need to connect Ruby to the Mysql I just installed through MacPorts. Its quite simple to do, once you know the right argument to pass to it. The easiest way is to just tell it where the mysql5_conf file is and let it figure out the rest for itself.

sudo gem install mysql -- --with-mysql-config=/opt/local/bin/mysql_config5

Hopefully this will save me 10 minutes of googling next time I need to do this!

Update 2009-01-21

I'm an idiot and typed the gem install command by hand, and ended up with --with-mysql-conf instead of --with-mysql-config. Updated now.

Update 2009-10-19

On Snow Leopard I needed to tell rubygems to install the gem as a 64-bit binary. Hattip to Philipp

sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- \
  --with-mysql-config=/opt/local/bin/mysql_config5