When programmers first hear about ScraperWiki, their initial reaction is often “what! you let anyone edit general purpose code and run it on your servers!”. The answer is that, yes, we do, but in an isolated environment. Your own “sandbox” if you like, where you can safely build castles without knocking others over. Or, as […]
Conquering Copyright and Scaling Open Data Projects – How Chris Taggart is Counting Culture
Chris Taggart is a founder of OpenlyLocal and OpenCorporates. He says “When people ask what I do I say I open up data, sometimes whether people like it or not.” In the beginning he didn’t really expect much to come of his first scrapers “other than maybe being told off by the councils, because all […]
Constructing the Open Data Landscape
In an article in today’s Telegraph regarding Francis Maude’s Public Data Corporation, Michael Cross asks: “What makes the state think it can be at the cutting edge of the knowledge economy“. He writes in terms of market and business share, giving the example of the satnav market worth over $100bn a year yet it’s based […]
ScraperWiki goes on the Records at The Texas Tribune
Here at ScraperWiki, we’ve got a good eye for data. Not just structuring, formatting and quality, but also where data can tell stories (hence the addition of views to the site). The decision to put a scraper of the Texas Department of Criminal Justice by Noah Seger on the front page proved to be a bit […]
Open Data Events – Meet like minded folk
Just a quick shout out to upcoming Open Data events crying out for participation from well minded coding citizens like yourselves: Open Australia and ScraperWiki Hackfest: In sunny down under (Sydney) on 10-11 September. No travel bursaries but it is free! Sign up here. It’s a hacking extravaganza for anyone interested in helping liberate data or […]
Scraping guides: Values, separated by commas
When we revamped our documentation a while ago, we promised guides to specific scraper libraries, such as lxml, Nokogiri and so on. We’re now staring to roll those out. The first one is simple, but a good one. Go to the documentation page and you’ll find a new section called “scraping guides”. The CSV scraping guide is available […]
Scheduling: A scrape a day keeps stale data away
We’ve just rolled out a change to the default frequency of new scrapers. They used to default to running once a day. Now they default to not running at all. We’ve made this change because people often make new scrapers that aren’t ready yet. These run every day and send annoying emails saying that they’re […]
ScraperWiki Digger Gets HTTPS Security System
During the last week of rioting across the UK 8 riot vans were called out to quel the unrest just around the corner from where I live. With scenes of chaos and destruction filling the airwaves and clogging up twitter I begun thinking: Are your scrapers safe from looters? We don’t stock trainers or flat-screen TVs, […]
ScraperWiki Tutorial Screencast for Non-Programmers
If you’ve been going through our first ambitious tutorial and taster session for non-coders then good for you! I hope you found it enlightening. For those of you yet to try it, here it is. It is a step-by-step guide so please give it a go and don’t just try and follow the answers as […]
This one goes out to the ScraperWikians we love!
From business licenses in Islington (London, UK) to Oakland (Calinfornia, USA), councillors in India, presidential engagements in Ireland, newspapers in America and planning applications in Westoxon; here at ScraperWiki we are saluting our users who are looting the web of data for those in need. The Robin Hood’s of the data digging world! We have […]