Sometimes Julian adds simple things to ScraperWiki, and hardly finds it worth telling anybody about them. For a while now, you’ve been able to create views, to get your data out of ScraperWiki just how you want, in Python, Ruby and PHP. Julian has added a new option for HTML. This is useful because you can […]
Libraries that are ready and waiting
One of the nice things about ScraperWiki is that it gives you access to all sorts of useful data manipulation libraries, all from your web browser. Everything from PDF extractors, to statistical analysis. We’ve now documented what is available – in Python, in Ruby and in PHP. Here’s a screenshot of a few for a […]
Who changed my scraper? History upgrade
We’ve upgraded the history window, to make it easier to see who’s changed your scraper, what they changed, and when it broke. This is particularly useful if you’ve just received an email alert. You can get to it from the “History” tab at the top right of each scraper or view’s page. Here’s an example from the Work4MP […]
Job advert: Product / UX lover
ScraperWiki is a Silicon Valley style startup, but based in the UK. We’re changing the world of open data, and how programming is done together on the Internet. We’re looking for a web product designer who is… Able to make design decisions to launch features by themselves. Capable of writing CSS and HTML, and some […]
Be alert! Your scrapers need alerts
It’s important to know when your scrapers have stopped working, so you can fix them. And if someone else makes a change to one of your scrapers, you need to know, so you can check it’s OK and thank them. Over the next day or two, if you have made or contributed to a scraper […]
Ruby screen scraping tutorials
Mark Chapman has been busy translating our Python web scraping tutorials into Ruby. They now cover three tutorials on how to write basic screen scrapers, plus extra ones on using .ASPX pages, Excel files and CSV files. We’ve also installed some extra Ruby modules – spreadsheet and FastCSV – to make them possible. These Ruby scraping […]
Job advert: Web designer/programmer
Care about oil spills, newspapers or lost cats? ScraperWiki is a Silicon Valley style startup, but in the North West of England, in Liverpool. We’re changing the world of open data, and how programming is done together on the Internet. We’re looking for a web designer/programmer who is… Capable of writing standards compliant CSS and […]
Views part 2 – Lincoln Council committees
(This is the second of two posts announcing ScraperWiki “views”. A new feature that Julian, Richard and Tom worked away and secretly launched a couple of months ago. Once you’ve scraped your data, how can you get it out again in just the form you want? See also: Views part 1 – Canadian weather stations.) Lincoln […]
Views part 1 – Canadian weather stations
(This is the first of two posts announcing ScraperWiki “views”. A new feature that Julian, Richard and Tom worked away and secretly launched a couple of months ago. Once you’ve scraped your data, how can you get it out again in just the form you want? See also: Views part 2 – Lincoln Council committees) Canadian […]
Get the source to ScraperWiki
We’ve always intended that ScraperWiki itself should be open source – philosophically it doesn’t make sense for a collaborative code wiki for working with public datasets not to be. More practically, letting anyone see our developer tools means you can also access our bug tracker, so you know what we’re working on and what issues […]