I’m a coding pleb, or The Scraper’s Apprentice as I like to call myself. But I realise that’s no excuse, as many of the ScraperWiki users I talk to have not had formal coding lessons themselves. Indeed, some of our founders aren’t formally trained (we have a doctorate in Chemistry here!). I’ve been attempting to […]
Meet the User – Tim McNamara
You may have noticed a Kiwi driving our digger around of late (image is purely metaphorical). A New Zealander by the name of Tim McNamara has been unearthing earthquakes, government bills, historic places, clinical trials and even companies. I had to enquire after such a scraping wizard (trying to get a Lord of the Rings reference in […]
‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’
You may have noticed that the design of the ScraperWiki site has changed substantially. As part of that, we made a few improvements to the documentation. Lots of you told us we had to make our documentation easier to find, more reliable and complete. We’ve reorganised it all under one contents page, called Documentation throughout […]
Code for America getting all ScraperWikied over Government Data
Hackathons have been sprouting up all over the world. In fact, there was an Open Data Hackathon in Guatemala City just last week! But the loudest buzz coming for the data hack hive can be heard in the US where Code for America has been producing the sweetest honey. So here at ScraperWiki headquarters we are […]
All recipes 30 minutes to cook
The other week we quietly added two tutorials of a new kind to the site, snuck in behind a radical site redesign. They’re instructive recipes, which anyone with a modicum of programming knowledge should be able to easily follow. 1. Introductory tutorial For programmers new to ScraperWiki, to a get an idea of what it […]
It’s SQL. In a URL.
Squirrelled away amongst the other changes to ScraperWiki’s site redesign, we made substantial improvements to the external API explorer. We’re going to concentrate on the SQLite function here as it is most import, but as you can see on the right there are other functions for getting out scraper metadata. Zarino and Julian have made […]
There’s More Than One Way to Scrape a Site
A request came in to ScraperWiki to scrape information on the Members of the European Parliament. I put it out on Twitter and Facebook hoping a kind member of the ScraperWiki community will have spent so much time on the computer he/she has no life at all. I had to turn people away! Within minutes, two […]
Access government in a way that makes sense to you? Surely not!
alpha.gov.uk uses Scraperwiki, a cutting edge data-gathering tool, to deliver the results that citizens want. And radically for government, rather than tossing a finished product out onto the web with a team of defenders, this is an experiment in customer engagement. If you’re looking to renew your passport, find out about student loans or how […]
ScraperWiki: A story about two boys, web scraping and a worm
“It’s like a buddy movie.” she said. Not quite the kind of story lead I’m used to. But what do you expect if you employ journalists in a tech startup? “Tell them about that computer game of his that you bought with your pocket money.” She means the one with the risqué name. I think I’d […]
It’s all a matter of trust
According to the latest Ipsos MORI poll on trust in people, only 1 in 5 people think journalists tell the truth. They’re still more trustworthy than politicians generally and government ministers! Phew. But telling the truth and being trustworthy are not the same thing. There’s not believing what they say and then there’s knowing that […]