Recently at ScraperWiki we replaced the old datastore, which was creaking under the load, with a new, lighter and faster solution – all your data is now stored in Sqlite tables as part of the move towards pluggable datastores. In addition to the new increase in performance, using Sqlite also provides some other benefits such […]
Hacks & Hackers Cardiff: the video
Gavin Owen, a recent postgraduate student at the Skillset Media Academy Wales, has produced this excellent video from the Hacks & Hackers Hack Day in Cardiff last month. Read about the projects here, and watch what the journalists and programmers got up to, below:
#newschallenge: ScraperWiki honoured to be invited to the Knight’s roundtable!
Last Thursday we were very excited to find out that we have been shortlisted by the Knight Foundation for its prestigious #newschallenge competition. This is a very big deal for us as it is such a highly respected organisation internationally. Over 1600 applications were entered in the first round. This was whittled down in the […]
OpenCorporates partners with ScraperWiki & offers bounties for open data scrapers
This is a guest post by Chris Taggart, co-founder of OpenCorporates When we started OpenCorporates it was to solve a real need that we and a number of other people in the open data community had: whether it’s Government spending, subsidy info or court cases, we needed a database of corporate entities to match against, and […]
See a scraper you like? Fork it!
We’ve just pushed a new feature to let you “fork” scrapers and views. That is, you can quickly and easily make a copy so you can make radical changes without interfering with the original, or to make customisations for your own use. We’ve used a red pitchfork for the icon, because amongst computer programmers forking […]
New Ruby scraping tutorials – PDFs and Mechanize
Got a PDF you want to get data from?Try our easy web interface over at pdftables.com! Mark Chapman has made us two new Ruby tutorials. Advanced Scraping: Pages Behind Forms shows you how to get data that is buried behind search boxes and drop down query lists. It uses the Mechanize library, which is a class […]
Sometimes you only need HTML / Javascript
Sometimes Julian adds simple things to ScraperWiki, and hardly finds it worth telling anybody about them. For a while now, you’ve been able to create views, to get your data out of ScraperWiki just how you want, in Python, Ruby and PHP. Julian has added a new option for HTML. This is useful because you can […]
Libraries that are ready and waiting
One of the nice things about ScraperWiki is that it gives you access to all sorts of useful data manipulation libraries, all from your web browser. Everything from PDF extractors, to statistical analysis. We’ve now documented what is available – in Python, in Ruby and in PHP. Here’s a screenshot of a few for a […]
New event! Hacks & Hackers Glasgow (#hhhglas)
Calling journalists, bloggers, programmers and designers in Scotland! Scraperwiki is pleased to announce another hacks & hackers hack day: in Glasgow. BBC Scotland is hosting and sponsoring the one day event, with support from BBC College of Journalism. As with our other UK hack days, Guardian Open Platform is providing the prizes. Web developers and […]
Who changed my scraper? History upgrade
We’ve upgraded the history window, to make it easier to see who’s changed your scraper, what they changed, and when it broke. This is particularly useful if you’ve just received an email alert. You can get to it from the “History” tab at the top right of each scraper or view’s page. Here’s an example from the Work4MP […]