He credits the internet revolution for transforming public sector data into ‘cashable proposition’. We, along with many other start-ups, foundations and civic coding groups, are part of this ‘geeky world’ of Open Data. So we’d like to add our piece concerning the Open Data movement.

Michael has the right to ask this question because there is this constant custodial battle being fought every day, every scrape and every script on the web for the rights to data. So let me tell you about the geeks’ take on Open Data.

[vimeo http://www.vimeo.com/21711338]

The idea(l) behind Open Data is to create sustainable Open Data projects with purpose. This has been championed in the last couple of years by civic data projects such as MySociety, Open Knowledge Foundation, Code for America, Open Australia. Open Development Cambodia is following me on twitter! Older, more established organizations are also being converted to the Open Data ethos. For instance, The World Bank is one major organization turning to Open Data in a big way.

However, much of the public sector data published so far has been pretty much useless. Governments, finally, are beginning to realize that data has little value unless people understand its context and provenance. They are beginning to see that opening up their data can reduce the cost and responsibility of getting it to the end point user, as the Open Declaration on European Public Services clearly says: “The needs of today’s society are too complex to be met by government alone”.

The key to a sustainable Open Data landscape lies not in the organisational heads of government bodies but in the provenance of the data they release and the ways in which it is released. The goal should be to gain the 5 stars of open linked data. For this to be achieved the data needs to be pared down to its raw ingredients. In a research paper entitled “Open Data, Open Society” (see end of post) Marco Fioretti explains:

Public data are really useful only when they are raw, really open and linked … only when data are published online in that way every citizen or organization will be able to automatically analyze and present them in easy to understand forms

This is where ScraperWiki really excels in terms of opening up data. Not only is our data open and accessible through various processes (csv, database, API), even the extraction process is open in the form of a code wiki. In terms of data, we are rawer than raw. If government ordered an open data steak they would order rare, data hubs would order raw, ours would be mooing!

We’re providing some of the heavy machinery needed to construct the Open Data landscape. What it will look like very much depends on the civic cyber-community getting involved. A leader in this community is Chris Taggart, creator of OpenlyLocal and OpenCorporates, and a prolific ScraperWiki user. So I Skyped him to see what he makes of the state thinking it can be at the cutting edge of the knowledge economy:

Speaking of the linked economy, do check out all the links in this post and all the media included here is under Creative Commons license.

If you are interested in getting more involved in the Open Data scene check out the Open Knowledge Foundation.

Open Data, Open Society

So here at ScraperWiki headquarters we are flattered to be their tool of choice. They’ve been using ScraperWiki as a project or session track for just about every hackathon, datacamp, or labs Friday. On the Stanford Hackathon blog, they write:

“ScraperWiki-ing” is a great activity for beginner coders. It is social and forgiving. Even better, your work goes towards making our governments more efficient, open, and transparent by making information more linkable and extendable.”

The closure of Data.gov has not dampened their spirits and with ScraperWiki, we think they’ll do an even better job of making data work for the government rather than the government working for the data.

Advantage of using ScraperWiki according to Tyler Stalder, fellow at Code for America:

• No technology setup or install-fest required to start working
• Small project with a clear deliverable product that can be completed in a couple of hours
• Directly focused on civic data
• It’s easy to pair on scrapers
• Python, Ruby, PHP support so it’s easy for us to organize pairs from a  diverse pool of devs