Predicting Experimental Results: Who Knows What? / Stefano DellaVigna, Devin Pope.

Format:

Book

Author/Creator:

DellaVigna, Stefano.

Contributor:

National Bureau of Economic Research.

Pope, Devin.

Series:

Working Paper Series (National Bureau of Economic Research) no. w22566.

NBER working paper series no. w22566

Language:

English

Physical Description:

1 online resource: illustrations (black and white);

Other Title:

Predicting Experimental Results

Place of Publication:

Cambridge, Mass. National Bureau of Economic Research 2016.

Summary:

Academic experts frequently recommend policies and treatments. But how well do they anticipate the impact of different treatments? And how do their predictions compare to the predictions of non-experts? We analyze how 208 experts forecast the results of 15 treatments involving monetary and non-monetary motivators in a real-effort task. We compare these forecasts to those made by PhD students and non-experts: undergraduates, MBAs, and an online sample. We document seven main results. First, the average forecast of experts predicts quite well the experimental results. Second, there is a strong wisdom-of-crowds effect: the average forecast outperforms 96 percent of individual forecasts. Third, correlates of expertise---citations, academic rank, field, and contextual experience--do not improve forecasting accuracy. Fourth, experts as a group do better than non-experts, but not if accuracy is defined as rank ordering treatments. Fifth, measures of effort, confidence, and revealed ability are predictive of forecast accuracy to some extent, especially for non-experts. Sixth, using these measures we identify `superforecasters' among the non-experts who outperform the experts out of sample. Seventh, we document that these results on forecasting accuracy surprise the forecasters themselves. We present a simple model that organizes several of these results and we stress the implications for the collection of forecasts of future experimental results.

Notes:

Print version record

August 2016.