Transfer to Morph.io
ShareScraperWiki Classic is now read-only.
But don’t worry! You can transfer this scraper to Morph.io if you want to continue editing it.
Transfer to Morph.io
Current statusIdle – not scheduled to run Last runScraped 526 pages from www.marketnews.usda.gov |
About this scraper
Scrapes commodities categorized under “Browse by Commodity” at this USDA Market News site: http://www.marketnews.usda.gov/portal/fv.
Each day, the database stores category, commodity, lowPriceMin, highPriceMax, cityName, originName, itemSize, date and a pointer to the XML source.
Currently it parses the XML source into the data store, but this is too slow (the connection frequently times out). I'd like to change it to parse the HTML table on the commodity page instead. (Feel free to do so!)
Timeout errors create holes in the data.
This scraper is publicAnyone can see this scraper, and any registered user can edit it. Users who have contributed to this scraper:
|
Public
Kevin Curry
Download SQLite databaseThis scraper in context
Transfer to Morph.io
Copied to:
December 2012
Run succeeded:
- ran 2 times, most recently for 0 seconds
(526 scraped pages, 11267 records)
15:29, 31 December 2012
Loading...
Run failed:
#<SqliteException: Query timeout: > - ran for 0 seconds
(6 scraped pages, 1312 records)
00:23, 28 December 2012
Loading...
Run failed:
#<HTTPClient::ReceiveTimeoutError: execution expired> - ran for 0 seconds
(0 scraped pages, 0 records)
19:42, 22 December 2012
Loading...
Run failed:
#<SqliteException: Query timeout: > - ran for 0 seconds
(4 scraped pages, 831 records)
16:39, 21 December 2012
Loading...
Run failed:
#<SqliteException: dataproxy.process: database is locked> - ran for 0 seconds
(2 scraped pages, 0 records)
15:42, 19 December 2012
Loading...
Run failed:
#<SqliteException: Query timeout: > - ran for 0 seconds
(4 scraped pages, 487 records)
15:41, 19 December 2012
Loading...
Run failed:
#<SqliteException: dataproxy.process: database is locked> - ran for 0 seconds
(2 scraped pages, 0 records)
13:26, 18 December 2012
Loading...
Run failed:
#<SqliteException: Query timeout: > - ran for 0 seconds
(4 scraped pages, 21 records)
13:26, 18 December 2012
Loading...
June 2011
Edited by
Kevin Curry
4 times
04:07, 7 June 2011
Edited by
Kevin Curry
2 times
03:55, 7 June 2011
Edited by
Kevin Curry
2 times
03:50, 7 June 2011
Edited by
Kevin Curry
03:10, 7 June 2011
Edited by
Kevin Curry
4 times
04:07, 7 June 2011
Loading...
Difference between, revision and previous revision
Edited by
Kevin Curry
2 times
03:55, 7 June 2011
Loading...
Difference between, revision and previous revision
Edited by
Kevin Curry
2 times
03:50, 7 June 2011
Loading...
Difference between, revision and previous revision
Edited by
Kevin Curry
03:10, 7 June 2011
Loading...
Difference between, revision and previous revision
May 2011
Edited by
Kevin Curry
21:40, 18 May 2011
Edited by
Kevin Curry
21:40, 18 May 2011