Rarely does a day pass when the team at the TechChange office misses an opportunity to say something like “hey data nerd, look at this and tell us what you think.”  Often these requests are made unceremoniously on Twitter, forcing me to respond or risk what little confidence the public has vested in my intellectual abilities.  Basically I’m the dancing analytics monkey, and Nick Martin et al is the organ grinder.

 So I was tweeted a question by @techchange this morning about something called PopTip on TechCrunch.  It’s basically a system that uses the vast crowd of the Twitterverse and the Twitter software’s semantic architecture to crowdsource answers to questions.  In the article Jordan Crook proposes a marketing question that PopTip could answer (best mobile phone platform), and the video that PopTip put together below focuses on MVPs of basketball games:

But what is the meaning of all of this?  While I think this is a great tool for selecting the best player during a basketball game, I’m not convinced that “real time” equals better or more reliable data.  Can this tool be useful for say, stock trading, or more germane to my research interests conflict management and political stabilization?  Indeed it could, so my reservations lie in the caveats; or as they say, the devil is in the details.

The biggest detail is temporal.  In quantitative social science volume of data over time is more valuable than a massive snap shot of one moment.  This isn’t limited to quantitative social science either; this extends to economics, the natural sciences, and policy making.  What a researcher, stock trader or policy maker should value is trends over time.  This is the root of empiricism; we make predictions about future outcomes based on the amount of data that already exists and the consistency of that data over time.  Or at least we should.

The problem is that these kinds of crowdsourced data programs are snapshots. They’re excellent mechanisms for selecting the MVP of an basketball game, which is a discrete event that isn’t meant to provide any kind of predictive value to the client.  The selection of Lebron James as the MVP one night will not affect the way the Heat play in the NBA finals, or the roster selection of the Oklahoma City Thunder.  But when the conversation shifts to something like stock trading, where empirical history should matter, PopTip’s value-added should decrease.  It should be disturbing that traders might make buy-sell decisions on a Twitter-based poll about whether “…#Apple, #RIM…or #Nokia” is better.  This would be hugely problematic since it would ignore entire histories of company behavior, and also leads to the second problem: does the public have the knowledge to make an informed decision?

The “crowd” is a funny thing; the people in it have intuition, and there’s no doubt that the aggregate of that intuition can be right on a number of things including market trends and their preferences for public policy.  At the same time, this intuition is inherently general and thus the results of the data that you get from a crowd using something like PopTip is only generally descriptive.  Jordan Crook points out another tool similar to PopTip called Estimize which allows a crowd of 6,500 users to estimate stock trajectories; he notes that 67% of the time the crowd is more accurate then Wall Street.  But again, the devil is in the details.

This is a big crowd and they’re given a fair amount of data; I’m more likely to credit their accuracy to the volume of contributors than to the idea that the “wisdom of the crowd” is more accurate than the analysts at Thomson Reuters or the Economist Intelligence Unit.  A way to test this would be to take random samples of Estimize contributors equal to the size of the Thomson Reuters or EIU teams, give them matching data, and have them make predictions on stock outlooks simultaneously.  Do this 20 or 30 times, and that would tell us a great deal more about the relative wisdom of the crowd.

Even if sample sizes were similar, there are non-quantifiable factors that affect results of stock predictions; traders and analysts have real money at stake, while the Estimize community may or may not.  Perceptions of risk affect decision making, and Estimize users might benefit from being more aggressive since they may not face the risk of real losses.  Another factor is what are they are making estimates on; are we talking about blue chip stocks or frontier investments in Africa?  Blue chips are pretty predictable, while higher risk investment that nets bigger returns are harder to predict without fairly in-depth knowledge and expertise.  Comparing the ‘crowd’ to the analysts across a spectrum of investment instruments would also tell us a great deal about the relative wisdom of the crowd.

To take this up to a more meta level, these kinds of questions are fair game when we talk about policy analysis or conflict prevention.  Snapshots of what’s happening in real time might be extremely valuable to peacekeepers or first responders, but this kind of data isn’t a predictor of events to come.  The ‘crowd’ certainly has wisdom, but it’s wisdom that still needs to be checked against larger datasets and historical evidence.  Along with checking, those of us involved in crowdsourcing need to understand that while we may know our data is useful, our clients (first responders, diplomats, pilots, etc.) have a different set of risks and responsibilities and the better we are about recognizing those needs the more valuable the growing trove of crowdsourced data will be.