flights – ScraperWiki https://blog.scraperwiki.com Extract tables from PDFs and scrape the web Tue, 09 Aug 2016 06:10:13 +0000 en-US hourly 1 https://wordpress.org/?v=4.6 58264007 Up in the Air with ScraperWiki and Tropo https://blog.scraperwiki.com/2011/12/up-in-the-air-with-scraperwiki-and-tropo/ Fri, 23 Dec 2011 11:55:24 +0000 http://blog.scraperwiki.com/?p=758216004 We came across this blog post a few days ago from these cool guys at Tropo in Florida, and thought you’d be interested in how they’ve used ScraperWiki. Tropo is a simple API for adding voice and other goodies to your apps and, as Mark Headd explains, it can be really powerful when combined with data ScraperWiki…

ScraperWiki is a powerful cloud-based service that lets you scrape data from online documents and websites.

When you write a scraper – a script to pull information from a web resource and then parse out the bits you want – it will execute inside the ScraperWiki environment.

SMS flight information

You can store the data that is scraped inside a data store and then access the data from outside the ScraperWiki environment using their API. Scrapers can be written in one of several different languages – Ruby, PHP and Python.

ScraperWiki and Tropo operate in a very similar way. The Tropo scripting environment allows you to write scripts in one of several different languages, including Ruby, PHP and Python (Groovy and JavaScript are also supported).

Your script executes inside the Tropo environment, which means you can make direct connections to external resources – like the ScraperWiki API – from within your executing script. There is no need for extra HTTP overhead, and the additional step of posting to a back end server to connect to other APIs or resources.

In the following screencast, I demonstrate how to use Tropo and ScraperWiki to quickly and easily build an airport information system for the Philadelphia International Airport.

All of the code for this example can be found here. If you’d like to view the actual scraper I wrote on ScraperWiki, you can find it here.

This is still a work in progress – I’d like to run this script multiple times per day (ideally, maybe once an hour) to get updates to flight information and ensure that the app has the most up to date flight status. The voice dialog could also use a little tweaking, and I’d like to offer the option of repeating the information.

But even with these refinements aside, it is evident how combining these two powerful cloud resources can generate a pretty useful application in a very short time.

Tropo and ScraperWiki are a powerful combination. Happy flying!

]]>
758216004