Last week, our Twitter API use was suspended. We’re talking to various people at Twitter, DataSift and Gnip to try and resolve this. Unfortunately, we still can’t tell or predict when or if we’ll be able to bring the service back. To avoid making false promises, we’ve removed the tools from our website for now.
Which GOV.UK department is most mobile?
We recently made 37 dashboards for GOV.UK, full of stats about what people look at on the Government’s website. As you know, the best data opens itself, so I asked myself, what does the underlying data behind these new dashboards secretly reveal? Each dashboard shows the devices people used to access a department. If you mush that together across […]
No Twitter tools for now… Month’s refund
Twitter have, for now, suspended our API access. Don’t panic – this is normal for Twitter applications when they reach the popularity level of ScraperWiki. It can take a couple of weeks to sort out – here’s an example thread of TraffUp going through it a while ago. For that reason, I personally apologise to our customers, […]
And fast streaming CSV download…
We’re rolling out a series of performance improvements to ScraperWiki. Yesterday, we sped up the Tableau/OData connector. Today, it’s the turn of the humble CSV. When you go to “Download a spreadsheet” you’ll notice the CSV file is now always described as “live”. This means it is always up to date, and streams at full […]
Super-faster Tableau integration
We’ve just rolled out a change to make our OData endpoint much faster. For example, it is down from 20 minute to 6 minutes to import 150,000 Tweets. If you’ve Tableau, or other software that can read OData, please try it out! If you’ve already got a connection set up, you need to go and get the […]
Getting all the hash tags, user mentions…
We’ve rolled out a change so you get more data when you use the Twitter search tool! We’ve changed four columns. They used to all just randomly return one thing. Now they return all the things, separated by a space. The columns are: hashtags now returns all of them with the hashes, e.g. #opendata #opendevelopment user_mention has […]
Two jobs at ScraperWiki to ponder over the bank holiday
It’s busy busy at ScraperWiki at the moment. We’re growing our award winning data hub (where people scrape Twitter). We’ve lots of interesting consultancy, for clients like the Cabinet Office (GDS), United Nations (OCHA), and Autotrader. So we’re hiring two people. 1) A Digital Marketer. This is an unusual opportunity to market a marketing data product! […]
Favorite Tweets!
How often was each Tweet favorited? Now you can tell, with a new column we’ve just added to our Twitter search tool – thanks to ScraperWiki user Alden Golab for suggesting this. You can sort by that column to find the most liked Tweet on a subject. For example, ScraperWiki’s right now is about our London Underground visualization. More […]
Try out the Tableau and QlikView connector
In March, we launched an OData tool. If you use Tableau or QlikView, it lets you easily get and refresh data from ScraperWiki. From today, the OData tool is now available on our new 30 day trial accounts. Which means anyone can try it out for free! Instructions here (particularly for Tableau).
Verified Twitter users
We’ve added a “verified accounts” column to our Twitter friends tool – thanks to ScraperWiki user Delfin Paris for suggesting this. The 1 means it is a Twitter verified account, 0 means it isn’t. You can sort by that column to find all the most notable accounts that are following someone. For example, ScraperWiki has 44 followers who […]