WoW Activity Feed – the tech behind it…

As mentioned previously – its live!

The app is built on nodejs, using coffeescript with a MongoDB database.

I chose coffeescript, as I prefer the Ruby-like syntax and really don’t like Javascript curly-brace’s. I also found NodeJS to be very fast and lightweight. Also, the callback model in node takes some getting used to.

Libraries like async – help make the callbacks more manageable and avoid the pyramid of death.

I used the rapidly growing expressjs framework – which was great at keeping out of your way and letting you just do what you need.

HTML was put together via Jade templates and Stylus stylesheet helper.

The core of the app accesses the Armory via a great little library – node-armory.

The data from the armory is saved directly as JSON to MongoDB and as further updates come in, the jsondiffpatch library – is used to determine whats changed.

There are some tests written with mocha and sinon for stubs/mocks.  My testing style is to use them when there is a problem – so they are probably broken at the moment.

I found a few features missing with the node-armory library and so forked the project to address them, such as support for using a proxy (for debug purposes) and using Mike Reinstein’s version of the http lib request that supports compressed request/replies.

See package.json for details of all the libraries used.

A couple of MongoDB’s features that came in handy, like:

  • Time to live (TTL) collections – records automatically removed when a date field is older than specified.  Useful to ensure only recent data is kept.
  • Capped collections – limit how big a table gets, automatically throws away old records when it gets full. Good for logs.
  • Aggregate queries – a map/reduce like facility for querying the DB.

Definitely the worst bit of the code is the RSS feed entry formatting – need to think of some ways to refactor it sensibly.

To make the search feature nice and responsive, backbone was used in a very basic way.

The main thing I missed from Rails was the asset pipeline, which lets you combine all the client side assets (javascript and css files) into just a few minified files.

One of the most interesting parts of the app (at least to me), is the scheduled job that checks the armory for updates.  The core of which uses an async queue to kick off many calls to the WoW API and collect the results.