As I have been looking for your great guys to highlight your work, passion and scraping oddities, so your time has come to explore the craven habits of the new age data digging programmers! You can now search for people on ScraperWiki!!!! Just use our regular search box. So all you ScraperWikians out there, I […]
Diggers and Dinosaurs – Scraping at the Mozilla Festival
In a complete paradigm shift of the epic battle between Godzilla and Mothra we are turning our backs on the old claymation medium and embracing the digital age where dinosaurs and diggers (yes, I am aware we are a machine and not a moth) can roam free across the lawless plains of web 2.0. Both […]
Amazing Places Scrapers Go – The Big Clean
Earlier this month, there’s been some underground scraping action happening in Central Europe. We noticed this spark of activity and upon further investigation it was revealed to be a spill over from The Big Clean. At the beginning of this moth, there was Open Scraper Challenge happening in 3 hackerspaces in Czech Republic and Slovakia, in Prague, […]
Scraping Government Data for the Open Government Data Camp
Come one, come all and gather ye ’round the fantastical scraping table at the Open Government Data Camp at Warsaw. Here you will see such mythical beasts the Irish man with the gift of the gab and the German obsessed with numbers and efficiency he has become part database. So head to Soho Factory in Warsaw, […]
Scraping New Frontiers
Today is Columbus Day in the US (yes, I’m working regardless). So I’ve decided to write a post about discovery. This has been my first full week in America. I have toiled Heathrow Terminal 5, battled through the baffling New York subway and scaled the mountains of food to find, well, not the promised land. […]
Start Talking to Your Data – Literally!
Because ScraperWiki has a SQL database and an API with SQL extraction, I can SQL inject (haha!) straight into the API URL and use the JSON output. So what does all that mean? I scraped the CSV files of Special Advisers’ meetings gifts and hospitalities at Number 10. This is being updated as the data […]
Help Get Olympic Data off the Start Line
As part of Media2012 we’ll be running (no pun intended) a Hacks and Hackers Data Journalism workshop. It’s part of the Abandon Normal Devices Festival. It’ll be on 2nd October from 11:00-17:00 at FACT (Foundation for Art and Creative Technology) Medialab, 88 Wood Street, Liverpool, L1 4DQ. So if you’re interested in sports data and want […]
Conquering Copyright and Scaling Open Data Projects – How Chris Taggart is Counting Culture
Chris Taggart is a founder of OpenlyLocal and OpenCorporates. He says “When people ask what I do I say I open up data, sometimes whether people like it or not.” In the beginning he didn’t really expect much to come of his first scrapers “other than maybe being told off by the councils, because all […]
Constructing the Open Data Landscape
In an article in today’s Telegraph regarding Francis Maude’s Public Data Corporation, Michael Cross asks: “What makes the state think it can be at the cutting edge of the knowledge economy“. He writes in terms of market and business share, giving the example of the satnav market worth over $100bn a year yet it’s based […]
ScraperWiki goes on the Records at The Texas Tribune
Here at ScraperWiki, we’ve got a good eye for data. Not just structuring, formatting and quality, but also where data can tell stories (hence the addition of views to the site). The decision to put a scraper of the Texas Department of Criminal Justice by Noah Seger on the front page proved to be a bit […]