It’s part of the Abandon Normal Devices Festival. It’ll be on 2nd October from 11:00-17:00 at FACT (Foundation for Art and Creative Technology) Medialab, 88 Wood Street, Liverpool, L1 4DQ.

So if you’re interested in sports data and want to see times, points and medal tables get off the line then come on down.

Register for this free event here.

Most importantly, beer and pizza will be provided!

So watch out London 2012, you’re being ScraperWikied!

We are therefore very pleased to announce that ScraperWiki has won the Knight News Challenge!

The Knight Foundation are spending $280,000 over 2 years for us to improve ScraperWiki as a platform for journalists, and to run events to bring together journalists and programmers across the United States.

America has trailblazing organisations that do data and journalism well already – for example, both ProPublica and the Chicago Tribune have excellent data centers to support their news content. Our aim is to lower the barrier to entry into data driven journalism and to create (an order of magnitude) more of this type of success. So come join our campaign for America: Yes We Can (Scrape).  PS: We are politically neutral but think open source when it comes to campaign strategy!

What are we going to do to the platform?

As well as polishing ScraperWiki to make it easier to use, and creating journalism focussed tutorials and screen casts, we’re adding four specific services for journalists:

• Data embargo, so journalists can keep their stories secret until going to print, but publish the data in a structured, reusable, public form with the story.
• Data on demand service. Often journalists need the right data ordered quickly, we’re going to create a smooth process so they can get that.
• News application hosting. We’ll make it scalable and easier.
• Data alerts. Automatically get leads from changing data. For example, watch bridge repair schedules, and email when one isn’t being maintained.

Here are two concrete examples of ScraperWiki being used already in similar ways:

• We manually embargoed the MP data that James Ball demanded off us for his Guardian front page.
• Edinburgh planning news application are embedded in the Greener Leith site, and now have alerts in the form of a Twitter bot.

Where in the US are we going to go?

What really matters about ScraperWiki is the people using it. Data is dead if it doesn’t have someone, a journalist or a citizen, analysing it, finding stories in it and making decisions from it.

We’re running Data Journalism Camps in each of a dozen states. These will be similar in format to our hacks and hackers hack days, which we’ve run across the UK and Ireland over the last year.

The camps will have two parts.

• Making something. In teams of journalists and coders, using data to dig into a story, or make or prototype a news app, all in one day.
• Scraping tutorials. For journalists who want to learn how to code, and programmers who want to know more about scraping and ScraperWiki.

This video of our event in Liverpool gives a flavour of what to expect.

Get in touch if you’d like us to stop near you, or are interested in helping or sponsoring the camps.

Finally…

The project is designed to be financially stable in the long term. While the public version of ScraperWiki will remain free, we will charge for extra services such as keeping data private, and data on demand. We’ll be working with B2B media, as well as consumer media.

As all Knight financed projects, the code behind ScraperWiki is open source, so newsrooms won’t be building a dependency on something they can’t control.

For more details you can read our original application (note that financial amounts have changed since then).

Finally, and most importantly, I’d like to congratulate and thank everyone who has worked on, used or supported ScraperWiki. The Knight News Challenge had 1,600 excellent applications, so this is a real validation of what we’re doing, both with data and with journalism.

We would like to say a particular thank you to the project’s editor, Sarah Hartley, for her generous help. We wish Sarah, Hannah, John and Michael the very best in their new ventures, whatever they may be.

As you can see from the comments under the Guardian post announcing the sites’ closure, the beatbloggers, led by Sarah, have done amazing work for their respective communities. It’s testament to their hard work and energy that they’ve built up such a loyal following in a short space of time.

Michael MacLeod from Guardian Edinburgh at our Glasgow event (right):

Earlier this month ScraperWiki put on its first ever student event, at Liverpool John Moores University in partnership with Open Labs for students from both LJMU’s School of Journalism and the School of Computing & Mathematical Sciences, as well as external participants. This fabulous video comes courtesy of the Hatch. Alison Gow, digital executive editor at the Liverpool Daily Post and the Liverpool Echo has kindly supplied us with the words (below the video).

Report: Hacks and Hackers Hack Day – student edition

By Alison Gow

At the annual conference of the Society of Editors, held in Glasgow in November, there was some debate about journalist training and whether journalism students currently learning their craft on college courses were a) of sufficient quality and b) likely to find work.

Plenty of opinions were presented as facts and there seemed to be no recognition that today’s students might not actually want to work for mainstream media once they graduated – with their varied (and relevant) skill sets they may have very different (and far more entrepreneurial) career plans in mind.

Anyway, that was last month. Scroll forward to December 8 and a rather more optimistic picture of the future emerges. I got to spend the day with a group of Liverpool John Moores University student journalists, programmers and lecturers, local innovators and programming experts, and it seemed to me that the students were going to do just fine in whatever field they eventually chose.

This was Hacks Meet Hackers (Students) – the first event that ScraperWiki (Liverpool’s own scraping and data-mining phenomenon that has done so much to facilitate collaborative learning projects between journalists and coders) had held for students. I was one of four Trinity Mirror journalists lucky enough to be asked along too.

Brought into being through assistance from the excellent LJMU Open Labs team, backed by LJMU journalism lecturer Steve Harrison, #hhhlivS as it was hashtagged was a real eye-opener. It wasn’t the largest group to attend a ScraperWiki hackday I suspect, but I’m willing to bet it was one of the most productive; relevant, viable projects were crafted over the course of the day and I’d be surprised if they didn’t find their way onto the LJMU Journalism news website in the near future.

The projects brought to the presentation room at the end of the day were:

• The Class Divide: Investigating the educational background of Britain’s MPs
• Are Police Helicopters Effective in Merseyside?
• Football League Attendances 1980-2010
• Sick of School: The link between ill health and unpopular schools

The prize for Idea With The Most Potential went to the Police Helicopters project. This group had used a sample page from Merseyside Police helicopter movements report, which showed time of flight, geography, outcome and duration. They also determined that of the 33% of solved crimes, 0.03% involved the helicopter. Using the data scraped for helicopter flights, and comparing it to crimes and policing costs data, the group extrapolated it cost £1,675 per hour to fly the helicopter (amounting to more than £100,000 a month), and by comparing it to average officer salaries projected this could fund recruitment of 30 extra police officers. The team also suggested potential spin-off ideas around the data.

The Best Use of Data went to the Football League Figures team an all-male bunch of journos and student journos aided by hacker Paul Freeman who scraped data of every Football League club and brought it together into a database that could be used to show attendance trends. These included the dramatic drop in Liverpool FC attendances during the Thatcher years and the rises that coincided with exciting new signings, plunging attendances for Manchester City and subsequent spikes during takeovers, and the affects of promotion and relegation Premier League teams. The team suggested such data could be used for any number of stories, and would prove compelling information for statistics-hungry fans.

The Most Topical project went to the Class Divide group – LJMU students who worked with ScraperWiki’s Julian Todd to scrape data from the Telegraph’s politics web section and investigate the educational backgrounds of MPs. The group set out to investigate whether parliament consisted mainly of privately-educated elected members. The group said the data led them to discover most Lib Dem MPs were state educated, and that there was no slant of figures between state and privately educated MPs, contrary to what might have been expected. They added the data they had uncovered would prove particularly interesting once the MPs’ vote was held on University tuition fees.

The Best Presentation and the Overall Winner of the hackday went to Sick of Schools by Scraping The Barrel – a team of TM journos and students, hacker Brett and student nurse Claire Sutton – who used Office for National Statistics, Census, council information, and scraped data from school prospectuses and wards to investigate illness data and low demand for school places in Sefton borough. By overlaying health data with school places demand they were able to highlight various outcomes which they believed would be valuable for a range of readers, from parents seeking school places to potential house buyers.

Paul Freeman, described in one tweet as the “the Johan Cruyff of football data scraping” was presented with a Scraperwiki mug as the Hacker of the Day, for his sterling work on the Football League data.

Judges Andy Goodwin, of Open Labs, and Chris Frost, head of the Journalism department, praised everyone for their efforts and Aine McGuire, of ScraperWiki, highlighted the great quality of the ideas, and subsequent projects.  It was a long day but it passed incredibly quickly – I was really impressed not only by the ideas that came out but by the collaborative efforts between the students on their projects.

From my experience of the first Hacks Meet Hackers Day (held, again with support from Open Labs, in Liverpool last summer) there was quite a competitive atmosphere not just between the teams but even within teams as members – usually the journalists – pitched their ideas as the ones to run with. Yesterday was markedly less so, with each group working first to determine whether the data supported their ideas, and adapting those projects depending on what the information produced, rather than having a complete end in sight before they started. Maybe that’s why the projects that emerged were so good.

The Liverpool digital community is full of extraordinary people doing important, innovative work (and who don’t always get the credit they deserve). I first bumped into Julian and Aidan as they prepared to give a talk at a Liver and Mash libraries event earlier this year – I’d never heard of ScraperWiki and I was bowled over by the possibilities they talked about (once I got my brain around how it worked). Since then team has done so much to promote the cause of open data, data journalism, the opportunities it can create, and the worth and value it can have for audiences; Scraperwiki hackdays are attended by journalists from all media across the UK, eager to learn more about data-scraping and collaborative projects with hackers.

With the Hacks Meet Hackers Students day, these ideas are being brought into the classroom, and the outcome can only benefit the colleges, students and journalism in the future. It was a great day, and the prospects for the future are exciting.

Watch this space for more ScraperWiki events in 2011!

The concept behind Hacks and Hackers, a global phenomenon, is simple: bring a bunch of hacks (journalists) and hackers (coders) together to build something really cool that other journalists and industry people can use. We were in nerd heaven.

The day kicked off with a talk from the lovely Francis Irving (@frabcus), Scraperwiki’s CEO. Francis talked about Scraperwiki’s main use-scraping data, stats & facts from large datasets – and the company’s background, from being built by coder Julian Todd to
getting funded by 4IP.

After that, the gathered geeks split off into groups, all with the same
goal: scrape data and find an explosive, exclusive story. First, second and third prizes would be awarded at the end of the day.

You don’t realize how similar coding and reporting are until you
watch a hack and a technologist work together to create something.
Both vocations have the same core purpose: creating something
useful that others can use (or in the hack’s case, unearthing
information that is useful to the public).

The headlines that emerged out of the day were amazing. ‘Mr No Vote’ won first prize. When citizen hacks Ivor Whitten, Matt Johnston and coder Robert Moore of e-learning company Learning
Pool used Scraperwiki to scrape electoral data from local government websites, they found that over 60% of voters in every constituency in Northern Ireland (save one) abstained from voting in the last election, raising questions about just how democratically MPs and MLAs have been elected.

What was really significant about the story was that the guys were able to uncover it within a number of hours. One member of Team Qluso, an ex investigative journalist, was astounded, calling Scraperwiki a “gamechanger” for the industry. It was an almost historical event, seeing technology transform a small but significant part of the industry: the process of finding and analyzing data. (A process that, according to said gobsmacked Team Qluso member, used to take days, weeks, even months).

If you get a chance to chat with the Scraperwiki team, take it
with both hands: these guys are building some cracking tools for
hacks’n’hackers alike.

Finally, from the Scraperwiki town, a big thank you to the McKeown family for all their hospitality in Belfast!

By Philip John, Journal Local. This has been cross-posted on the Journal Local blog.

It may be a tiny city but Lichfield has shown that it has some great talent at the Hacks and Hackers Hack Day.

Sponsored by Lichfield District Council and Lichfield-based Journal Local, the day was held at the George Hotel and attended by a good selection of local developers and journalists – some coming from much further afield.

Once the introductions were done and we’d all contributed a few ideas the work got started and five teams quickly formed around those initial thoughts.

The first two teams decided to look into Private Finance Initiatives (PFIs) and Information Asset Registers (IARs). The first of these scraped information from 470 councils to show which of these published information about PFIs. The results showed that only 10% of councils actually put out any details of PFIs, highlighting a lack of openness in that area.

Also focused on PFIs was the ‘PFI wiki’ project which scraped the Partnerships UK database of PFIs and re-purposed it to allow deeper interrogation, such as by region and companies. It clearly paves the way for an OpenCharities style site for PFIs.

Future care needs was the focus of the third team who mapped care homes along with information on ownership, public vs private status and location. The next step, they said, is to add the number of beds and match that to the needs of the population based on demographic data, giving a clearer view of whether the facilities exist to cater for the future care needs in the area.

A Lichfield-related project was the focus of the fourth group who aimed to create a comprehensive guide to events going on in Lichfield District. Using about four or five scrapers, they produced a site that collated all the events listing sites serving Lichfield into one central site with a search facility. The group also spawned a new Hacks/Hackers group to continue their work.

Last but not least, the fifth group worked on flood warning information. By scraping the Environment Agency web site they were able to display on a map, the river level gauges and the flood warning level so that at a glance it’s possible to see the water level in relation to the flood warning limit.

So after a long day Lizzie Thatcher and Rita Wilson from Lichfield District Council joined us to judge the projects. They came up with a clever matrix of key points to rate the projects by and decided to choose the ‘what’s on’ and ‘flood warning’ projects as joint winners, who each share a prize of £75 in Amazon vouchers.

The coveted ScraperWiki mug also went to the ‘what’s on’ project for their proper use of ScraperWiki to create good quality scrapers.

Pictures from the event by Nick Brickett. Click through to view slideshow…

• Reserve your *free* place here

B2B journalists, developers and designers are invited to attend the one-day ‘Hacks and Hackers’ event hosted and sponsored by RBI, B2B publisher of titles including FlightGlobal, Farmers Weekly and New Scientist.

The idea is that business journalists and bloggers (‘Hacks’) pair up with computer programmers and designers (‘Hackers’) to produce innovative data projects in the space of one day. Food and drink will be provided throughout the event. Prizes for the best projects will be awarded in the evening.

Any journalist from a B2B background, or developer/designer with an interest in business journalism is welcome to attend. We’re especially keen to work with people who are interested in producing data visualisations.

“Data journalism is an important area of development for our editorial teams in RBI,” said Karl Schneider, RBI editorial development director:

“It’s a hot topic for all journalists, but it’s particularly relevant in the B2B sector. B2B journalism is focused on delivering information that it’s audience can act on, supporting important business decisions.

“Often a well-thought-out visualisation of data can be the most effective way of delivering critical information and helping users to understand key trends.

“We’re already having some successes with this kind of journalism, and we think we can do a lot more. So building up the skills of our editorial teams in this area is very important.”

The event is the first in-house hack day that Scraperwiki has organised as part of its UK and Ireland Hacks & Hackers tour.

5o places are available in total: half for RBI staff; half for external attendees. People wishing to attend should select the relevant ticket at this link.

Past hacks and hackers days have run in London, Liverpool, Birmingham and Manchester. For a flavour of the projects please see this blog post.

If you have any questions please contact Aine McGuire via Aine [at]scraperwiki.com.

• BOOK TICKETS HERE

“Lichfield District Council have been publishing open data for a while now, and it seems a good fit to put on a day where we can showcase the data we have published, as well as encourage people to do something with it,” said council webmaster Stuart Harrison.

“We’re not precious though, and if something is built using other public data, we’ll be just as happy!”

The details:

What? Scraperwiki, the award-winning new screen scraper and data mining tool, funded by 4iP and Lichfield District Council are putting on a one day practical hack day* in Lichfield, Staffordshire at which web developers and designers (hackers) will pair up with journalists and bloggers or anyone with an interest in media and communications (hacks) to produce a number of projects and stories based on public data. It’s all part of the ScraperWiki UK & Ireland Hacks and Hackers tour.

Who’s it for? We hope to attract ‘hacks’ and ‘hackers’ from all different types of backgrounds – across programming, media and communications.

What will I get out of it? The aim is to show journalists how to use programming and design techniques to create online news stories and features; and vice versa, to show programmers how to find, develop, and polish stories and features. To see what happened at our past events in Liverpool and Birmingham visit the ScraperWiki blog.

How much? NOTHING! It’s free, thanks to our sponsors, Lichfield District Council and Journal Local.

What should I bring? We would encourage people to come along with ideas for local ‘datasets’ that are of interest. In addition we will create a list of suggested data sets at the introduction on the morning of the event but flexibility is key for this event. If you have a laptop, please bring this too.

So what exactly will happen on the day? Armed with their laptops and WIFI, journalists and developers will be put
into teams of around four to develop their ideas, with the aim of
finishing final projects that can be published and shared publicly. Each team will then present their project to the whole group. Overall winners will receive a prize at the end of the day.

*Not sure what a hack day is? Let’s go with the Wikipedia definition: It “an event where developers, designers and people with ideas gather to build ‘cool stuff'”…

With thanks to our sponsors:

• Belfast tickets at this link

It’s a chance for web developers and designers to pair up with journalists and bloggers to produce a number of projects and stories based on public data. We’re using the tag ‘#hhhBel‘ to talk about this on Twitter.

It’s the latest confirmed date on the ScraperWiki tour: we’ll be going to Dublin on the 16th November, Manchester on Friday October 15th and Leeds on Thursday October 28th. We’re currently negotiating venues and dates in Cardiff and Glasgow.

What? Hacks and Hackers Hack Day Belfast

When? Saturday November 13th 2010 from 9.30 am to 7.30 pm

Where? At  the University of Ulster – room TBC (on Google Maps)

Who is it for? We hope to attract ‘hacks’ and ‘hackers’ from all different types of backgrounds: people from big media organisations, as well as individual online publishers and freelancers.

How much? Absolutely free, thanks to our sponsors, the School of Media, Film and Journalism at the University of Ulster and Guardian Open Platform.

Refreshments, lunch and evening snacks will be all be provided.

NB: We’re still seeking sponsors to get involved. If you think your organisation might be interested please contact aine [at] scraperwiki.com. Or get in touch with any other questions!

What happens on the day? The aim is to show journalists how to use programming and design techniques to create online news stories and features; and vice versa, to show programmers how to find, develop, and polish stories and features. All sorts of data was scraped and played with at our past events: in Liverpool, projects included mashes of police, libraries and courts data. Birmingham saw lots of health-related projects, as well as scraping of political party donor and leisure centre information.

This four minute video from Liverpool gives you a flavour of the kind of projects that can be developed.

Armed with their laptops and WIFI, journalists and developers will be put into teams of around four to develop their ideas, with the aim of finishing final projects that can be published and shared publicly. Each team will then present their project to the whole group. Overall winners will receive a prize at the end of the day.

• Book your place now, via this page: http://belfasthackday.eventbrite.com/

The *free* hack day is for both developers and journalists. For further sponsorship opportunities please contact aine [at] scraperwiki.com.

• You can read more about the format and book tickets here

Armed with their laptops and WIFI, journalists and developers will be put into teams of around four to develop their ideas, with the aim of finishing final projects that can be published and shared publicly. Each team will then present their project to the whole group.

As previously announced, we will be running an event in Liverpool on July 16; more on that here.