Comments on: How to get along with an ASP webpage Extract tables from PDFs and scrape the web Thu, 14 Jul 2016 16:12:42 +0000 hourly 1 By: Why does this ScraperWiki for an ASPX site return only the same page of search results? - Python Questions - Developers Q & A Mon, 22 Jul 2013 10:31:08 +0000 […] blog post helped me get started. I thought the following code would grab the final page of BBS listings for […]

By: Jonathan Wed, 17 Apr 2013 17:44:09 +0000 I am trying this method but there doesn’t seem to be a submit control on my page. Any ideas?

By: Big fat aspx pages for thin data | ScraperWiki Data Blog Tue, 07 Feb 2012 06:06:15 +0000 […] this point it’s worth a recap on how to get along with an asp webpage, because that is what this […]

By: Henare Degan Thu, 10 Nov 2011 06:50:11 +0000 It wasn’t clear in my original comment but the bunch of scrapers are scraping an ASP.NET-based system.

By: Henare Degan Thu, 10 Nov 2011 04:04:31 +0000 Epic article Julian! I’ve wanted to write up something similar in the past but by the time I’ve got the scraper worked out I’m so over dealing with it that I can’t bring myself to write about it :-/

We’ve got a bunch of scrapers that scrape the same planning system albeit with tiny differences (“customisations” that councils no doubt pay shit loads for). What can we do to help?

By: P Thu, 10 Nov 2011 01:58:11 +0000 Ruby mechanize is a little bit smarter than python as there is no need to disable that button.
What the scraper might look like in ruby:

agent =
doc = agent.get ‘’

while next_link =‘form a#lnkNext’)
break unless next_link[‘href’]
form = doc.forms[0]
form[“__EVENTTARGET”],form[“__EVENTARGUMENT”] = $1,$2 if next_link[‘href’] =~ /__doPostBack(‘(.*)’,'(.*)’/
doc = form.submit

By: mazadillon Wed, 09 Nov 2011 12:30:02 +0000 I’ve wasted many hours of my life struggling with scraping of ASP pages in the past, it probably would have helped if I’d blogged about it and explained some of my work arounds! A support group sounds great but what would be even better would be if Microsoft didn’t use such an insane way of doing things!