where data wizards learn magic

As a business person, if I want insight into my business needs, I can ask a data scientist for answers, so I can make better decisions!

Urgh: the sound I make reading that.

I am starting to wonder if Data Science is seen as magic, and insight as arcane wisdom distilled from eyes of newts – lots of them, cause it’s a Big Data cauldron, of course!

Many online business publications cover Data Science as a buzzword, and a quick search adding “data science” to a business function like, say: “marketing,” returns many posts extolling the power of data to inform decisions. But examples are scarce.

This awareness seems to be reflected from well-known examples of analysis actually turning up business ideas. One of the best examples is America’s everything-shop Target inferring that some of its shoppers were pregnant before it became public knowledge (sometimes, even before they had told close family). This came about, it seems, because people asked a simple business question:

‘“Specifically, the marketers said they wanted to send specially designed ads to women in their second trimester … Can you give us a list?” the marketers asked.’ (NYT)

Target distills a lot of information into its shoppers’ profiles: shopping habits (obviously), bank used, car driven, websites visited (oh hai cookies!), and the rest. Using all this data, Target’s stat wonks were able to mashup user profiles with wider habit-patterns, and find people matching a predicted pattern of behaviours.

The NYT article discusses a particularly interesting area of data science, where internal data is augmented by external information. So, by acquiring population stats (the NYT guesses at things like demographics, zip codes, birth records, thought Target didn’t say exactly what external data), they were able to make better guesses about what kind of people were in their big stacks of user profiles.

This is interesting (if somewhat creepy), because the people asking the question had a pressing business need: tell pregnant women that Target sells newborn baby stuff, before they give birth. The marketers asked that question of the analysts, instead of asking for just the metrics. Avinash Kaushik talks about that phenomenon particular to web analytics, and I think it’s a similar story when business people ask report questions instead of business questions of their data scientists.

All of that stems from the simple business question: “Can we have a list of people who are probably pregnant, so we can send them a specific message?” And, let’s face it, Target had budget to spend on huge data. So, what are the lessons for smaller businesses beyond seeing data science as magic?

OK, so there must be some wins closer to home: where else are business questions being asked of data? So, I am very interested in other examples of marketers, flaks, hacks, managers and CIO’s asking such questions, and would love to talk to some of you!

I put a similar question up on Quora, and there is a comment box down there. You can also drop me a line at zach@scraperwiki.com, if you have good examples. I’d like to write them up, tell the stories to the ScraperWiki community, and get beyond the “magic” and into cases, facts, and workable ideas!

image credit: “Arches upon Arches” by Zach Beauvais, CC BY-SA 2.0 via flickr