Ho ho hometer
December is always a busy month for us. You know what it is like. The rush to get client work out the door, Christmas parties, the usual distractions. With this in mind we decided what better time to set aside a day to work on a side project together?
Being December we had to do something festive so batted a few ideas around about data visualisation. We explored several ideas but kept coming back to scraping data from twitter and using this in some way to monitor the fluctuating Christmas Spirit. At first we weren’t sure whether to try and create our own hashtag to monitor - like Ben Marsh’s excellent #uksnow map - or simply use an existing hashtag or phrase. In the end we decided to capture tweets mentioning the word ‘Christmas’. This would let us tap into what people are already saying rather than only getting data from those within our immediate reach.
The initial focus was to try and get data to play with. This meant learning how the Twitter API and OAuth worked (leveraging the excellent TwitterAPIExchange) and working out how best to do the most with this data. Because of Twitter’s API rate limits (and potential performance issues) we took a snapshot of the data at 1 minute intervals to cache the results locally. The twitter API lets us do 18 requests every 15 minutes - limited to 100 tweets per request - so the most we could capture is every 50 seconds. We’re probably missing some tweets but I think this gives us enough data to be interesting.
To keep things focused we decided to only capture geo-referenced tweets and those sent within Britain. We were able to do this with the ‘geocode’ parameter which lets you specify a focal point and radius to restrict tweets retrieved.
Once we’ve captured the tweets we then need to work out whether that tweet is positive or negative. We looked into different ways of capturing this through sentiment analysis.
There are a number of APIs that can capture this information - such as AlchemyAPI - but it was quickly apparent that we would hit rate limits fast. So after a few tweets we were put onto the AFINN, a comprehensive list of English language words rated for valence with an integer between minus five (negative) and plus five (positive). With this list we can loop through the words contained in the tweets we are capturing and produce a sentiment score for each.
So that gave us our data - tweets within Britain storing longitude and latitude with an associated sentiment score ranging from -5 to +5.
So what now to do with this data? Inspired by the beautiful Listen to Wikipedia we wanted to look at ways we could visualise the sentiment of tweets and use sound to signify different sentiments.
The first thing we tackled was finding a simple way to plug into the Web Audio API. Howler.js from Goldfire Studios was the perfect choice as it comes with a nice set of methods which allowed us to quickly prototype the audio aspect of the project.
We wanted to add a little variation to the sounds played so we randomly switched between two instruments. Now that our project was quite literally humming along nicely it was time to integrate the location based data with the google maps api.
We decided that it would be a nice effect to hide the map entirely and just have some colour indicators appear on the page respective to the geo location data. The sounds and sizes correspond to the sentiment of the tweet. Red circles correspond to negative tweets and green positive. The larger the circle the more extreme the sentiment. Likewise the tone of the music corresponds to the mood of the tweet - lower tones are negative and higher tones are positive. If you look at the finished project for a short while you will start to notice the shape of the UK and Ireland starting to form.
Rather annoyingly we couldn’t find a way to directly target the map element so instead of simply hiding it we had to use a map style with all of the features (roads, rivers et al) set to visibility off.
The finish product can be seen at http://xmas.ournameismud.co.uk - it's been great to set aside a day from client work to build something together and we're really pleased with the outcome. And if this is something you'd like to do as part of your job then remember, we're currently recruiting a PHP developer!