Comments on: How to get along with an ASP webpage https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/ Extract tables from PDFs and scrape the web Thu, 14 Jul 2016 16:12:42 +0000 hourly 1 https://wordpress.org/?v=4.6 By: Why does this ScraperWiki for an ASPX site return only the same page of search results? - Python Questions - Developers Q & A https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/#comment-725 Mon, 22 Jul 2013 10:31:08 +0000 http://blog.scraperwiki.com/?p=758215794#comment-725 […] blog post helped me get started. I thought the following code would grab the final page of BBS listings for […]

]]> By: Jonathan https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/#comment-724 Wed, 17 Apr 2013 17:44:09 +0000 http://blog.scraperwiki.com/?p=758215794#comment-724 I am trying this method but there doesn’t seem to be a submit control on my page. Any ideas?

]]> By: Big fat aspx pages for thin data | ScraperWiki Data Blog https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/#comment-723 Tue, 07 Feb 2012 06:06:15 +0000 http://blog.scraperwiki.com/?p=758215794#comment-723 […] this point it’s worth a recap on how to get along with an asp webpage, because that is what this […]

]]> By: Henare Degan https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/#comment-722 Thu, 10 Nov 2011 06:50:11 +0000 http://blog.scraperwiki.com/?p=758215794#comment-722 It wasn’t clear in my original comment but the bunch of scrapers are scraping an ASP.NET-based system.

]]> By: Henare Degan https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/#comment-721 Thu, 10 Nov 2011 04:04:31 +0000 http://blog.scraperwiki.com/?p=758215794#comment-721 Epic article Julian! I’ve wanted to write up something similar in the past but by the time I’ve got the scraper worked out I’m so over dealing with it that I can’t bring myself to write about it :-/

We’ve got a bunch of scrapers that scrape the same planning system albeit with tiny differences (“customisations” that councils no doubt pay shit loads for). What can we do to help?

]]> By: P https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/#comment-720 Thu, 10 Nov 2011 01:58:11 +0000 http://blog.scraperwiki.com/?p=758215794#comment-720 Ruby mechanize is a little bit smarter than python as there is no need to disable that button.
What the scraper might look like in ruby:

agent = Mechanize.new
doc = agent.get ‘http://data.fingal.ie/ViewDataSets/’

while next_link = doc.parser.at(‘form a#lnkNext’)
break unless next_link[‘href’]
form = doc.forms[0]
form[“__EVENTTARGET”],form[“__EVENTARGUMENT”] = $1,$2 if next_link[‘href’] =~ /__doPostBack(‘(.*)’,'(.*)’/
doc = form.submit
end

]]> By: mazadillon https://blog.scraperwiki.com/2011/11/how-to-get-along-with-an-asp-webpage/#comment-719 Wed, 09 Nov 2011 12:30:02 +0000 http://blog.scraperwiki.com/?p=758215794#comment-719 I’ve wasted many hours of my life struggling with scraping of ASP pages in the past, it probably would have helped if I’d blogged about it and explained some of my work arounds! A support group sounds great but what would be even better would be if Microsoft didn’t use such an insane way of doing things!

]]>