Hi again all! After completing a week of work experience in February, I was eager to return to ScraperWiki as an intern as soon as possible and, a few emails to Francis later, we had organised for me to do a months internship this summer, which has now unfortunately come to an end. My first […]
GeoJSON into ScraperWiki will go!
Surely everyone likes things on maps? Driven by this thought we’re produced a new tool for the ScraperWiki Platform: an importer for GeoJSON. GeoJSON is a file format for encoding geographic information. It is based on JSON which is popular for web based APIs because it is light weight, flexible and easy to parse by […]
The story of getting Twitter data and its “missing middle”
We’ve tried hard, but sadly we are not able to bring back our Twitter data tools. Simply put, this is because Twitter have no route to market to sell low volume data for spreadsheet-style individual use. It’s happened to similar services in the past, and even to blog post instructions. There’s lots of confusion in the […]
NewsReader World Cup Hack Day
A long time ago*, in a galaxy far, far away** we ran the NewsReader World Cup Hack Day. *Actually it was on the 10th June . **It was in the Westminster Hub, London. NewsReader is a EU FP7 project aimed at developing natural language processing and Semantic Web technology to make sense of large streams […]
getting,stuff,done
In Berlin last week, a bunch of interoperability geeks gathered for the first csv,conf. Yes, that’s right, comma-separated value files. The conference was about getting stuff done. Data in, data out… With an ironic self-recognition that CSVs are weak in lots of ways, but still the best we’ve got. To give you a taste, here […]
The history of Pivot table
A pivot table is a spreadsheet feature that allows data tables to be rearranged in many ways for different views of the same data (pivot from one view to another). Pivot Tables have become ubiquitous amongst power users of Excel, even being listed as a skill in CVs and a “desirable” in job specifications – […]
Twitter tool update
Last week, our Twitter API use was suspended. We’re talking to various people at Twitter, DataSift and Gnip to try and resolve this. Unfortunately, we still can’t tell or predict when or if we’ll be able to bring the service back. To avoid making false promises, we’ve removed the tools from our website for now.
Which GOV.UK department is most mobile?
We recently made 37 dashboards for GOV.UK, full of stats about what people look at on the Government’s website. As you know, the best data opens itself, so I asked myself, what does the underlying data behind these new dashboards secretly reveal? Each dashboard shows the devices people used to access a department. If you mush that together across […]
No Twitter tools for now… Month’s refund
Twitter have, for now, suspended our API access. Don’t panic – this is normal for Twitter applications when they reach the popularity level of ScraperWiki. It can take a couple of weeks to sort out – here’s an example thread of TraffUp going through it a while ago. For that reason, I personally apologise to our customers, […]
And fast streaming CSV download…
We’re rolling out a series of performance improvements to ScraperWiki. Yesterday, we sped up the Tableau/OData connector. Today, it’s the turn of the humble CSV. When you go to “Download a spreadsheet” you’ll notice the CSV file is now always described as “live”. This means it is always up to date, and streams at full […]