(This is the second of two posts announcing ScraperWiki “views”. A new feature that Julian, Richard and Tom worked away and secretly launched a couple of months ago. Once you’ve scraped your data, how can you get it out again in just the form you want? See also: Views part 1 – Canadian weather stations.)

Lincoln Council committee updates

Sometimes you don’t want to output a visualisation, but instead some data in a specific form for use by another piece of software. You can think of this as using the ScraperWiki code editor to write the exact API you want on the server where the data is. This saves the person providing the data having to second guess every way someone might want to access it.

Andrew Beekan, who works at Lincoln City Council, has used this to make an RSS feed for their committee meetings. Their CMS software doesn’t have this facility built in, so he has to use a scraper to do it.

First he wrote a scraper in ScraperWiki for a “What’s new” search results page from Lincoln Council’s website. This creates a nice dataset containing the name, date and URL of each committee meeting. Next Andrew made a ScraperWiki view and wrote some Python to output exactly the XML that he wants.

Andrew then wraps the RSS feed in Feedburner for people who want email updates. This is all documented in the Council’s data directory. They used to use Yahoo pipes to do this, but Andrew is finding ScraperWiki easier maintain, even though some knowledge of programming is required.

Since then, Andrew has gone on to make a map for the Lincoln decent homes scheme, also using ScraperWiki views – he’s written a blog post about it.