Comments on: Why the Government scraped itself https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/ Extract tables from PDFs and scrape the web Thu, 14 Jul 2016 16:12:42 +0000 hourly 1 https://wordpress.org/?v=4.6 By: steve white https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-651 Thu, 09 Feb 2012 04:40:59 +0000 http://blog.scraperwiki.com/?p=758214953#comment-651 hmm is the theory the personell and details will change more often then the site structure, that justify the effort of writing scrapers of the gov sites

]]> By: ScraperWiki Tutorial Screencast for Non-Programmers | ScraperWiki Data Blog https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-650 Mon, 15 Aug 2011 17:25:05 +0000 http://blog.scraperwiki.com/?p=758214953#comment-650 […] nicola[at]scraperwiki.com. For full training sessions or scraping projects like OpenCorporates or AlphaGov contact aine[at]scraperwiki.com Like this:LikeBe the first to like this post. This entry was […]

]]> By: We Eat Data – ScraperWiki talk at Open Knowledge Conference 2011 | ScraperWiki Data Blog https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-649 Mon, 04 Jul 2011 18:49:14 +0000 http://blog.scraperwiki.com/?p=758214953#comment-649 […] of data needed for similar projects wherever we go. Tobias Escher wants to do something similar to AlphaGov for Germany called Meine Demokratie. A lot of very simple little scrapers can go a long way and if […]

]]> By: Peter C https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-648 Thu, 16 Jun 2011 22:27:43 +0000 http://blog.scraperwiki.com/?p=758214953#comment-648 Agreed, the alphagovuk approach is just rolling the wheel in slightly more webservicey mud. This is not too dreadful and to be expected as incremental innovation is the only way for bureaucratic beasts. There is only so much one can do with front-end eye candy, what needs to change are the organisational processes regarding information and communication flows, replacing the hordes of middlemen/data clerks with intelligent, adaptable & direct tools – the paper trails could get so much more efficient. Anyone else up for a processalphagov? 😉

]]> By: KevinJump https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-647 Thu, 16 Jun 2011 18:29:35 +0000 http://blog.scraperwiki.com/?p=758214953#comment-647 yes in almost every way alpha gov is reinventing the wheel, but I think it wasn’t a very good wheel to start with. All to often we are keen to move on to the next big thing, when the reality is we haven’t gotten the first thing right. rightly or wrongly direct.gov has become a bit of a beast, it has almost everything, but no one can find anything. Alpha gov is attempting to reverse that. not the easiest thing in the world.

In the same way we redeveloped http://liverpool.gov.uk – we did nothing new , we just concentrated on getting what was already there and available done right.

]]> By: Francis Irving https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-646 Thu, 16 Jun 2011 17:56:26 +0000 http://blog.scraperwiki.com/?p=758214953#comment-646 Thanks for your comment Claire!

I had no idea Directgov had a consultations page like that. Thank you for pointing it out.

It’s a great idea, but unfortunately it has been very badly executed

1. It is very hard to find – it doesn’t appear in Google. For some reason it is deliberately blocked (see http://consultations.direct.gov.uk/robots.txt). This means you have to click several times from a Google search for “government consultations” to get to it, and carefully read the text on two long Directgov pages and pick out the right links. I bet it has very low traffic.

2. It has no interface for browsing. This means if you don’t know anything about Government consultations and you just want to see what kind of thing there is, there is nothing you can do. You can’t even search for an empty string.

3. The searching interface is very inflexible, like an early web interface. These days it could have much clearer presentation, rather than being an upfront overwhelming form. I’m thinking of things like interactive sorting, and nice calendar widgets for choosing date ranges. Lots of details.

4. It has no alerting system – such as by RSS, Email or Twitter. Nor any easy way for anyone to make an alert on top of its content (because it doesn’t appear in Google). Consultations

Fixing all the issues above is exactly the point of Alphagov. I agree that their alpha prototype hasn’t solved everything there, but I at least have some hope that their final version will, if it is commissioned.

I could write a similar detailed criticism of the “Connect to your council” site.

To start you off, when I click on the “Enter details” I have to manually delete “e.g. SE1 or Lambeth” before I can type my own postcode in. It has been fairly standard on the web for more than 10 years to automatically remove such help text when the text box is clicked in.

So yes, Alphagov is in part “reinventing the wheel”. It is building a much better wheel, suited to how people use the web today, and focussing ruthlessly on the details of the experience of individual citizens trying to get something done.

This is quite hard thing to do for a web designer, and quite subtle. But for the citizen used to services from Google and Facebook it is as plain as day. Millions of people are frustrated every day by difficulty in using Government websites.

]]> By: Claire Simpson https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-645 Thu, 16 Jun 2011 09:59:28 +0000 http://blog.scraperwiki.com/?p=758214953#comment-645 Actually most of the functionality on alphagov already exists within Directgov. Public Consultation information across all those organisation has been available for over a year on Directgov (http://consultations.direct.gov.uk/).

Other functions that are on there, such as searching for services by Local Authority are also on there (http://mycouncil.direct.gov.uk/)

What they seem to have done is change/improve the navigation so that it is basically a search of government websites. I think it looks great, but isn’t it just reinventing the wheel?

]]> By: tom https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-644 Wed, 15 Jun 2011 12:10:26 +0000 http://blog.scraperwiki.com/?p=758214953#comment-644 If you are wondering why the government would want to scrape its own sites, I’d suggest high up on the list is the fact that many of the websites are run by third parties as part of an outsourcing agreement, and for a different branch of government to get hold of the underlying data that drives the site (in any kind of useful format) would require some kind of additional contractual agreement to get the outsourcer to give up the data.

More generally, information within government is very much held in silos, and these websites represent examples of that.

]]> By: Simon Dickson https://blog.scraperwiki.com/2011/06/why-the-government-scraped-itself/#comment-643 Mon, 13 Jun 2011 14:39:33 +0000 http://blog.scraperwiki.com/?p=758214953#comment-643 Tragically, those of us with experience of working on the inside won’t be at all surprised by the notion of government scraping its own websites.

]]>