Split-Sample Strategies for Avoiding False Discoveries / Michael L. Anderson, Jeremy Magruder.

Format:

Book

Author/Creator:

Anderson, Michael L.

Contributor:

National Bureau of Economic Research.

Magruder, Jeremy.

Series:

Working Paper Series (National Bureau of Economic Research) no. w23544.

NBER working paper series no. w23544

Language:

English

Physical Description:

1 online resource: illustrations (black and white);

Place of Publication:

Cambridge, Mass. National Bureau of Economic Research 2017.

Summary:

Preanalysis plans (PAPs) have become an important tool for limiting false discoveries in field experiments. We evaluate the properties of an alternate approach which splits the data into two samples: An exploratory sample and a confirmation sample. When hypotheses are homogeneous, we describe an improved split-sample approach that achieves 90% of the rejections of the optimal PAP without requiring preregistration or constraints on specification search in the exploratory sample. When hypotheses are heterogeneous in priors or intrinsic interest, we find that a hybrid approach which prespecifies hypotheses with high weights and priors and uses a split-sample approach to test additional hypotheses can have power gains over any pure PAP. We assess this approach using the community-driven development (CDD) application from Casey et al. (2012) and find that the use of a hybrid split-sample approach would have generated qualitatively different conclusions.

Notes:

Print version record

June 2017.