Stability of Experimental Results: Forecasts and Evidence / Stefano DellaVigna, Devin Pope.

Format:

Book

Author/Creator:

DellaVigna, Stefano.

Contributor:

National Bureau of Economic Research.

Pope, Devin.

Series:

Working Paper Series (National Bureau of Economic Research) no. w25858.

NBER working paper series no. w25858

Language:

English

Physical Description:

1 online resource: illustrations (black and white);

Other Title:

Stability of Experimental Results

Place of Publication:

Cambridge, Mass. National Bureau of Economic Research 2019.

Summary:

How robust are experimental results to changes in design? And can researchers anticipate which changes matter most? We consider a specific context, a real-effort task with multiple behavioral treatments, and examine the stability along six dimensions: (i) pure replication; (ii) demographics; (iii) geography and culture; (iv) the task; (v) the output measure; (vi) the presence of a consent form. We use rank-order correlation across the treatments as measure of stability, and compare the observed correlation to the one under a benchmark of full stability (which allows for noise), and to expert forecasts. The academic experts expect that the pure replication will be close to perfect, that the results will differ sizably across demographic groups (age/gender/education), and that changes to the task and output will make a further impact. We find near perfect replication of the experimental results, and full stability of the results across demographics, significantly higher than the experts expected. The results are quite different across task and output change, mostly because the task change adds noise to the findings. The results are also stable to the lack of consent. Overall, the full stability benchmark is an excellent predictor of the observed stability, while expert forecasts are not that informative. This suggests that researchers' predictions about external validity may not be as informative as they expect. We discuss the implications of both the methods and the results for conceptual replication.

Notes:

Print version record

May 2019.