We’re rebuilding ScraperWiki.
For three years, we’ve been helping people get, clean and analyse data on the web. Our key insight was that you need to write code to do that, and we should make writing that code as easy as possible.
Earlier this year, we realised that that isn’t enough.
ScraperWiki Classic, as we now call the original ScraperWiki, falls into a gap. It’s neither flexible enough for most programmers, nor simple enough for most non-programmers.
There’s a bunch of people who love it, right in the middle. Our plan was to push that sideways – add more features to make it useful to programmers, and add tools to make it useful to anyone.
We listened and watched our users and found it wasn’t going to work. To make ScraperWiki useful to enough people now, we have to make it powerful and geeky enough for programmers, and, right at the other end, supply code-free tools that anyone can use straight away.
So that’s what we’re making. One unified platform that exposes the true power of a raw UNIX environment and a set of industry standards (like HTTP and SQL) for programmers; and that also, at the other end, builds on those foundations to give non-programmers access to a whole ecosystem of powerful, code-free, web-based tools for data science.
Everything is transparent, and every tool, no matter how simple, has a “view source” button so we can all get to, edit, and learn from the code underneath.
But more than transparency and code, ScraperWiki has always been about community. And that’s never going to change: By sharing one platform, the power-users can learn from the programmers, or use the tools they create, while the programmers can work with each other, and use the power-user tools to simplify the boring bits of coding.
If that sounds like your sort of data hub, stay tuned: We’ll be introducing the new ScraperWiki in the New Year.
 If you want to know more about the theory and practice of bringing “end-user computing” (letting people make their own custom apps) to everyone, I recommend Bonnie Nardi’s book A Small Matter of Programming.