Texas outline

September 2009

Clusterfuck: A Subversive Job-Distribution Tool

I should start this off by making something clear: I am an extremely lazy programmer. I long ago adopted the Ruby mantra that programmer time is more precious than machine time, and tend to write most of my code in a way that is clear and comprehensible, with little regard for speed of execution.

When working with massive NLP datasets, this is perhaps not an entirely good idea.

Lately my experiments have begun to take up more and more days to complete. Part of this, of course, is Ruby’s fault — the language, or at least the current interpreter, is far from fast — and part of the blame lies with me, for writing fundamentally lazy code. I have, however, come up with something of an interesting fix for the problem by parallelizing almost all of my big experiments. Easily done, really — but I then realized that I don’t have access to any of the department’s clusters. But I do have a login that works on all of the public-access machines in the (huge) undergraduate labs.


Enter clusterfuck, a subversive job-distribution tool I’ve been working on to solve this niggly little dilemma. It’s basically a tool for automating the process of ssh-ing into multiple machines and starting jobs on each of them, writ large. It’s installable via GitHub (I’ll push a gem out to Gemcutter when I have a nice, polished version ready) and configured rake-style via clusterfiles:

Clusterfuck::Task.new do |task|
    task.hosts = %w{clark asimov}
    task.jobs = ["hostname","hostname","hostname","hostname"]
    task.temp = "fragments"
    task.username = "SSHUSERNAME"
    task.password = "SSHPASSWORD"
    #task.debug = true

You kick off a batch of cluster jobs via the command clusterfuck (which takes a clusterfile as an argument, or defaults to clusterfile if it exists). This example file will run the command hostname four times on two machines (two each, unless one goes down or is slow to respond) and save the output of each command into a directory fragments. A bit of a toy example to be sure, but replacing the uninteresting hostname job with something more time-consuming and extend the list of hosts to, say, several dozen idle machines can get large, decomposable jobs completed in a fraction of the time necessary to run ‘em on a single machine.

Why “clusterfuck”? The early versions of this tool were, to say the least, somewhat unreliable, and had a frustrating tendency to leave cpu-intensive processes floating around. Needless to say, the network admins were not happy about this state of affairs, and the entire thing turned into a giant, well, you get the idea.

Getting Started With Flixel on OS X

The other day I stumbled across Flixel, a brilliant collection of ActionScript3 classes for building Flash games. Flixel is a pure ActionScript library, meaning it’s intended for use without the Flash IDE. It’s mind-bogglingly new, but has a nascent community that’s growing larger by the day. Most tutorials and things I’ve found for Flixel tend to be focused on Windows development though, using either FlashDevelop or FlexBuilder. I couldn’t find a guide to getting started with Flixel development on OS X, so I thought I’d put on my tutoring hat and write one myself — this post (and posts to follow) should get you started developing Flixel-based games on OS X, using nothing but the free Flex SDK and a text editor.

Install Flex SDK:

  • Download the Flex SDK from http://www.adobe.com/cfusion/entitlement/index.cfm?e=flex3sdk
  • Unzip the file we just downloaded (flex_sdk_3.4.zip) and copy the resulting folder (flex_sdk_3.4) somewhere out of mind. I stuck mine in ~/Library (/Library).
  • Now we need to tell terminal where to find the Flex tools. Open up Terminal (Applications/Utilities/Terminal) and type open ~/.profile. This opens up your bash profile in TextEdit; add this line to the end of the file: export PATH=~/Library/flex_sdl_3.4/bin:PATH (if you put the flex_sdk_3.4 folder somewhere other than ~/Library, use that path here instead). Save the file and quit TextEdit
  • Close and re-open Terminal (or type source ~/.profile, either works), and make sure everything works by typing mxmlc -help. You should get a screenful of text, starting with the line “Adobe Flex Compiler (mxmlc).” Success!


  • Download Flixel from http://flixel.org/flixel_v1.25.zip
  • Unzip flixel_v1.25.zip
  • In your Terminal window, change into the newly-created flixel folder. If you downloaded and unzipped Flixel to your Downloads folder, you can do this by typing cd ~/Downloads/flixel_v1.25/.
  • Compile the Mode example with the Flex compiler by typing mxmlc Mode.as. You should see something like:
    Loading configuration file /Users/doches/Library/flex_sdk_3.4/frameworks/flex-config.xml
    /Users//Downloads/flixel_v1.25/Mode.swf (380608 bytes)
  • Alright! If you look in the flixel_v1.25 folder you should now see Mode.swf — double click on this to open it up in the Flash Player (or type open Mode.swf in Terminal).

Ok — that’s everything we need to build Flash games using Flixel. I'd write a bit more about actually using the library, but that's definitely been taken care of by other, more talented, folks than I...

Holy Crap! Wikirunner featured on IndieGames.com!

I am absolutely abuzz with joy. My entry for Mini-LD #12, Wikirunner, is on the front page of IndieGames.com!

From the post:

Wikirunner is an a chase game inspired by Jeremy Bushnell’s Wikipedian Tag. Created for Mini Ludum Dare #12 (the theme was ‘Wikipedia’), two players are given a random Wikipedia page each, and then ‘the chaser’ must try to navigate his or her way to the same page as their opponent, while ‘the runner’ must try to stay off the same page as the chaser for as long as possible.

Both players can only move to another Wiki page if it has some relevance to the page they are currently on - so for example, you could move from ‘London Olympics’ to ‘2012’, but not from ‘Russia’ to ‘kangaroo’. The chaser also gets two goes for every one of the runner’s goes, to give him a fair chance. The game can be played in a 2-player hotseat style, or 1 vs AI, or AI vs AI (if you want to watch how the program thinks).

By far the most fun mode is an AI chaser vs a player runner. It’s interesting to see the choices the AI makes - at one point I changed to ‘Canada’, so the AI jumped over to a page on ‘Dumbledore’s Army’. Escaping to the Wiki page for ‘Iraq’, the program then tried to follow me by bringing up the page for ‘The Manchester Evening News’, which just so happens to be my local newspaper strangely enough.