Henare here from the OpenAustralia Foundation – Australia’s open data, open government and civic hacking charity. You might have heard that we were planning to have a hackfest here in Sydney last weekend. We decided to focus on writing new scrapers to add councils to our PlanningAlerts project that allows you to find out what is being built or knocked down in your local community. During the two afternoons over the weekend seven of us were able to write nineteen new scrapers, which covers an additional 1,823,124 Australians – a huge result.

There are a number of reasons why we chose to work on new scrapers for PlanningAlerts. ScraperWiki lowers the barrier of entry for new contributors by allowing them to get up and running quickly with no setup – just visit a web page. New scrapers are also relatively quick to write which is perfect for a hackfest over the weekend. And finally, because we have a number of working examples and ScraperWiki’s documentation, it’s conceivable that someone with no programming experience can come along and get started.

It’s also easy to support people writing scrapers in different programming languages using ScraperWiki. PlanningAlerts has always allowed people to write scrapers in whatever language they choose by using an intermediate XML format. With ScraperWiki this is even simpler because as far as our application is concerned it’s just a ScraperWiki scraper – it doesn’t even know what language the original scraper was written in.

Once someone has written a new scraper and formatted the data according to our needs, it’s a simple process for us to add it to our site. All they need to do is let us know, we add it to our list of planning authorities and then we automatically start to ask for the data daily using the ScraperWiki API.

Another issue is maintenance of these scrapers after the hackfest is over. Lots of volunteers only have the time to write a single scraper, maybe to support their local community. What happens when there’s an issue with that scraper but they’ve moved on? With ScraperWiki anyone can now pick up where they left off and fix the scraper – all without us ever having to get involved.

It was a really fun weekend and hopefully we’ll be doing this again some time. If you’ve got friends or family in Australia, don’t forget to tell them to sign up for PlanningAlerts.

Cheers,

Henare
OpenAustralia Foundation volunteer

Open Australia and ScraperWiki Hackfest:

In sunny down under (Sydney) on 10-11 September. No travel bursaries but it is free! Sign up here. It’s a hacking extravaganza for anyone interested in helping liberate data or hacking on OpenAustralia projects. We look forward to following the tags. If you’re gonna go then sign up for an account now. If their mammals are anything to go by we’re hoping for some wacky uses of open data.

Open Government Data Camp:

This year held in Warsaw and taking place on 20-21 October. Run by the fabulous Open Knowledge Foundation it boasts two days of talks, code sprints and workshops. The week run up will have satellite events including a workshop from our fine selves so come along and tell us how we can make your scraping life better. Early bird tickets are still available so get them while they’re hot here. There are travel bursaries so check out those too. Get the full programme here.

Open Aid Data Conference and Hackday:

In Berlin, Germany on 28-29 September, this conference offers training on data sources of development cooperation and a hackday for programmers to develop application of aid data. Register here (PS: it’s in German but the event is in English). Check out the full programme here.

We hope you all can make it and help aid data, government data and Australia get ScraperWikied

PlanningAlerts.org.au started life as a port of the PlanningAlerts.com codebase, which scrapes development application information from councils and sends email alerts of planning applications nearby to the user. It has since significantly evolved and is now a Ruby on Rails application that has sent almost one million development applications to people throughout Australia.

“We’ve started to use ScraperWiki because it makes collaboratively fixing scraper problems much simpler”
– Henare Degan (openaustralia)

Open Australia were finding Councils update their development application pages quite regularly. As a result, almost 30 percent of their scrapers broke. By having the visibility of what scrapers are broken, OpenAustralia volunteers can quickly dive in and make the changes required to fix broken scrapers on ScraperWiki. This is where our email alerts come in to play.

“Because of the email alerts, I’ve been able to quickly correct [broken scrapers] meaning more development applications are flowing to PlanningAlerts. The real power for me is that ScraperWiki handles the regular runs of scrapers and the email alerts are a killer feature”
– Henare Degan (openaustralia)