WoW Activity Feed – the tech behind it…
As mentioned previously – its live!
Libraries like async – help make the callbacks more manageable and avoid the pyramid of death.
I used the rapidly growing expressjs framework – which was great at keeping out of your way and letting you just do what you need.
I found a few features missing with the node-armory library and so forked the project to address them, such as support for using a proxy (for debug purposes) and using Mike Reinstein’s version of the http lib request that supports compressed request/replies.
See package.json for details of all the libraries used.
A couple of MongoDB’s features that came in handy, like:
- Time to live (TTL) collections – records automatically removed when a date field is older than specified. Useful to ensure only recent data is kept.
- Capped collections – limit how big a table gets, automatically throws away old records when it gets full. Good for logs.
- Aggregate queries – a map/reduce like facility for querying the DB.
Definitely the worst bit of the code is the RSS feed entry formatting – need to think of some ways to refactor it sensibly.
To make the search feature nice and responsive, backbone was used in a very basic way.
One of the most interesting parts of the app (at least to me), is the scheduled job that checks the armory for updates. The core of which uses an async queue to kick off many calls to the WoW API and collect the results.