As social technologies and applications took off, the ethos of collective intelligence permeated our behavior as individuals, consumers, employees and citizens. Everyone within reach of an internet connection or mobile phone has the ability to contribute and consume information, which can be augmented or acted upon by other individuals, local or afar, in a variety of pursuits.
The typical connection between social technologies and collective intelligence is the reams of data shared by individuals via venues such as Facebook, Twitter, Google+ and Wikipedia. Collective intelligence as source of big data.
Recently though, I've noticed the emergence of companies that are applying collective intelligence to solve big data problems. And not their big data problems, but yours.
First up, featured in a good series on the WSJ's Venture Capital Dispatch blog is 1010data. According to the dispatch [emphasis is mine]:
"1010data set out to build a scalable, easy-to-use data warehouse, enabling customers to replace on-premise database software, machines and staff with a subscription fee. But what it’s built is much more than a hosted data service — it has pioneered a new type of data marketplace. Customers now come to 1010data not only to outsource the management of their own data, but to mix it together with data from third parties, such as credit bureaus and retailers, and analyze it all in a single place."
The dispatch describes Dollar General Corp's use case, which includes sharing (for a fee) its information with suppliers:
"How it’s been used: Dollar General Corp. became a customer about three years ago when it began using 1010data to analyze basket data to determine what items were commonly sold together. It gained insights such as that people who bought Gatorade most frequently bought laxatives as well.
Dollar shortly moved its entire data warehouse onto 1010data, including all point-of-sale and inventory data, making up 17 billion and 35 billion rows of data, respectively. The retailer provides consumer goods companies with access to the data about its own products and, for a higher price, products of its competitors, with some limitations. For example, companies can’t see inventory data on competing products."
A startup going even deeper on the collective intelligence aspect is Kaggle. According to an All Things D post, Kaggle is applying the Netflix Prize model to all sorts of big data problems:
"Kaggle has facilitated breakthroughs in NASA’s analysis of dark matter, improved Allstate’s actuarial methods, predicted many of the top finishers of the Eurovision Song Contest, and is currently hosting a $3 million prize to devise ways to reduce unnecessary hospitalizations.
...Kaggle founder and CEO Anthony Goldbloom, a former statistician for the Australian treasury, says his company addresses “a serious market failure.” That is: Companies have data and can’t analyze it as well as they’d like, and academia is desperate for real-world data sets."
Interesting, Kaggle has found a way around the data scientist shortage:
"Today, Kaggle’s 17,000 participating data scientists — which happen to include former Netflix Prize winners — participate mostly for the challenge and the chance to prove themselves. But Goldbloom and Howard want to make it worth their while.
“In five years’ time, I want to do 10,000 competitions per year,” Goldbloom said. “I’m hoping competitors can earn a full-time income from Kaggle.”
And yes, you can submit a private problem, under NDA:
"On that front, Kaggle will now start paying small groups of data scientists — selected based on their past performance on Kaggle — to analyze sensitive data sets for which companies require NDAs."
Big data, innovating business and inspiring innovative business models. Opportunity abounds.
