All You Never Knew About Denmark’s Fat Tax
October 1, 2017
The Best Places For Paying Business Taxes
October 3, 2017
The Twitter Predictor might have some problems.
Where are we going? To research replication issues.
But first, the Twitter story…
The Twitter Predictor
Initially, the Twitter Predictor developer expected to see a positive relationship between the direction of the Dow and a subsequent Twitter response. However, the connection was not sad tweets on down days and happy ones after the Dow went up.
Instead, the Tweets came first. A slew of calm tweets meant the Dow would probably rise. Anxious tweets and it fell. Their accuracy? The data had an 86.7% correlation between Tweet sentiment and the Dow. It appeared to be yet another big data conquest.
Not necessarily.
In the real world, a British hedge fund said it used the Twitter Predictor. Claiming to have fared well, they closed down in less than a year.
Then, other researchers tried to replicate the Predictor’s results and the problems really started. Yes, they could get a similar correlation for the sample time period in the original paper. But that was just 2 1/2 weeks–December 1 to December 19, 2008. Looking beyond, they had an “out-of-sample-failure” with replication. They found no predictive power for Twitter.
However, our story does need a postscript. In a more recent paper, the same researcher says he can now use the words “bullish” and “bearish” to do his predicting.
Our Bottom Line: Replication
A 2015 paper from the Federal Reserve tells us that only 22 out of 67 economics papers from reputable journals had replicable results. Concerned, they believe one solution would be to require data submission. Then, the economic community can more easily confirm research. And, journal authors can learn from their mistakes.
Our takeaway? A process, replication attempts might or might not be accurate. But they can move us forward because of the thinking they generate.
My sources and more: If you just read one paper on replication, this is it from fivethirtyeight. The next possibility is the Federal Reserve description of economic research replication attempts. And after that, this econtalk podcast is also a possibility. As for the Twitter predictor, you might want to see the original study, this ECB paper, and recent replication research.
Please note that several sentences on the Twitter Predictor were posted at econlife during 2010.
