The first step is to collect some data using the ScraperWiki Platform, the demonstration here uses the our new US Stock Market data tool but could equally well have used the Twitter Follower or Twitter Search tools.

To connect Qlikview to ScraperWiki you should install the OData tool on the dataset on the ScraperWiki Platform and paste the link provided in File > Edit Script > Web files (in the Data pane at the bottom). For each table you have in your dataset there will be a different link.

On the next dialog page just choose “feed” and then “Finish”.

You will end up with and apparently complicated piece of code in your browser, but with this you only need to eliminate all rows which do not contain content/properties and also those who end in m:type.

To get just the data you have scraped into your table, the end result should be like this:

LOAD [content/properties/Volume] as Volume,
[content/properties/AdjClose] as AdjClose,
[content/properties/High] as High,
[content/properties/Low] as Low,
[content/properties/rowid] as rowid,
[content/properties/Date] as Date,
[content/properties/Close] as Close,
[content/properties/Open] as Open
FROM [https://free-ec2.scraperwiki.com/tb89xnn/7ynpetp3oczdtbt/cgi-bin/odata/stockdata] (XmlSimple, Table is [feed/entry]);

After saving and reloading, you have all your data ready for action in Qlikview.

Happy Qliking…

All you have to do is enter some comma-separated stocks, for example: AAPL,FB,MSFT,TWTR and then press the Get Stocks button to collect all the data that’s available. Once you’ve set the tool running, the data continues to automatically update with the latest data daily. Just as with any other ScraperWiki dataset, you can view in a table, query with SQL or download the data as a spreadsheet for use elsewhere. With our new OData connector, you can also import the data directly into Tableau.

You can see Ian demonstrating the use of the US stock market tool, and using the OData tool to connect to Tableau in this YouTube video:

If you use Tableau or QlikView, it lets you easily get and refresh data from ScraperWiki.

From today, the OData tool is now available on our new 30 day trial accounts.

Which means anyone can try it out for free!

Instructions here (particularly for Tableau).

In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get “live” stock market data from Yahoo!Finance into Tableau. Python is a well-established programming language with a rich ecosystem of software libraries which can provide access to a wide range of data.

This isn’t a route to doing high frequency trading but it is a demonstrates the principles of using ScraperWiki as an adaptor to data on the web. Although Tableau supports a wide range of data connections it can’t handle everything. As well as ready-made tools to collect data and serve it up in different formats, ScraperWiki allows users to write their own tools. The simplest method is to use the “Code in a browser” tool.

I wrote about the Pandas library a few weeks ago, its designed to provide some of the statistical and data processing functionality R to users of Python. It grew out of the work of a financial analyst, Wes McKinney, so naturally he added a little piece of functionality to pull in stock market data from Yahoo!Finance. The code required to do this is literally a single line.

To make data we collect using the pandas library available to all of ScraperWiki tools, like the OData connector or the View in a Table tool, we need to write the data into a local database.

You can see the code to get Yahoo!Finance data and make it available in the screenshot below, and you can get a copy directly from this GitHub gist.

Once you’ve entered the code, then you can run it immediately or schedule it to run regularly.

In less than 10 lines of code we’ve added a new data source to Tableau!

The most complicated part of the process is getting the pandas library to recognise the dates properly. This is by no means a polished tool but it is fully functioning and can easily be modified to collect different stock data. Obvious extensions would be to collect a list of stocks, and to provide a user interface.

Once we have the data then we can access it over OData, I followed Andrew Watson’s instructions for making a “candlestick” plot (here). And the resulting plot is shown below and can be found on Tableau Public.

On a desktop installation of Tableau you can refresh the data at the click of a button.

What data can you get in less than 10 lines of code?

There must be a simpler way!

And so there is. Today we’re excited to announce our new “Connect with OData” tool: the hassle-free way to get ScraperWiki data into analysis tools like Tableau, QlikView and Excel Power Query.

To get a dataset into Tableau, click the “More tools…” button and select the “Connect with OData” tool. You’ll be presented with a list of URLs (one for each table in your dataset).

Copy the URL for the table of interest. Then nip over to Tableau, select “Data” > “Connect to Data” > “OData”, and paste in the URL. Simple as that.

The OData connection is fast and robust – so far we’ve tried it on datasets with up to a million rows, and after a few minutes, the whole lot was downloaded and ready to visualise in Tableau. The best bit is that dates and Null values come through just fine, with zero configuration.

The “Connect with OData” tool is available to all paying ScraperWiki users, as well as journalists on our free 20-dataset journalist plan.

If you’re a Tableau user, try it out, and let us know what you think. It’ll work with all versions of Tableau, including Tableau Public.