Variation in Performance of Commonly Used Statistical Methods for Estimating Effectiveness of State-Level Opioid Policies on Opioid-Related Mortality / Beth Ann Griffin, Megan S. Schuler, Elizabeth A. Stuart, Stephen Patrick, Elizabeth McNeer, Rosanna Smart, David Powell, Bradley Stein, Terry Schell, Rosalie Liccardo Pacula.

Format:

Book

Author/Creator:

Griffin, Beth Ann.

Contributor:

National Bureau of Economic Research.

Schuler, Megan S.

Stuart, Elizabeth A.

Patrick, Stephen.

McNeer, Elizabeth.

Smart, Rosanna.

Powell, David.

Stein, Bradley.

Schell, Terry.

Pacula, Rosalie Liccardo.

Series:

Working Paper Series (National Bureau of Economic Research) no. w27029.

NBER working paper series no. w27029

Language:

English

Physical Description:

1 online resource: illustrations (black and white);

Place of Publication:

Cambridge, Mass. National Bureau of Economic Research 2020.

Summary:

Over the last two decades, there has been a surge of opioid-related overdose deaths resulting in a myriad of state policy responses. Researchers have evaluated the effectiveness of such policies using a wide-range of statistical models, each of which requires multiple design choices that can influence the accuracy and precision of the estimated policy effects. This simulation study used real-world data to compare model performance across a range of important statistical constructs to better understand which methods are appropriate for measuring the impacts of state-level opioid policies on opioid-related mortality. Our findings show that many commonly-used methods have very low statistical power to detect a significant policy effect (< 10%) when the policy effect size is small yet impactful (e.g., 5% reduction in opioid mortality). Many methods yielded high rates of Type I error, raising concerns of spurious conclusions about policy effectiveness. Finally, model performance was reduced when policy effectiveness had incremental, rather than instantaneous, onset. These findings highlight the limitations of existing statistical methods under scenarios that are likely to affect real-world policy studies. Given the necessity of identifying and implementing effective opioid-related policies, researchers and policymakers should be mindful of evaluation study statistical design.

Notes:

Print version record

April 2020.