Our experience is that right now an attempt at a completely automated tool would always end up frustrating. There are quite a few products that try to do it, and they can be very useful – but they can’t magically scrape anything that is scrapable. You’ll end up programming, or doing programming like things, within them anyway (as happens in, say, Refine).

We are, however, doing several things with the grant money that are in the direction of an automatic tool. One of these is a set of tools that write the first pass of code for you – and in simple cases that will be all you need to do. (Others are journalist specific tutorials, and general usability improvements)

Would love to visit Portland!

But nearly all of the journalists I know are dead-set against the kind of “coding” that’s required to acquire data via ScraperWiki. They have deadlines, and, let’s face it, HTML parsing, regular expressions and the mechanics of interrogating sites is hard programming. So the uptake of data journalism here in Portland has been slow, even with available hackers like myself. Right now I know of only one journalist here who’s attempted to use ScraperWiki (with the help of two hackers) and it’s been a frustrating experience.

What I think I’m looking for is some kind of “wireframing tool” – an interactive “drag and drop, point and click, WYSIWIG” user interface so that *journalists* can build scrapers. They have this now for the visualization / storytelling part of storytelling with tools like Tableau. They (or at least I) have this now for the data exploration part with tools like R, GGobi and Mondrian (and Excel, of course). But there’s not a tool I know of for rapidly building a scraper. Is there a chance you can use some of the grant money on a more journalist-friendly user interface?