1 option
The Utility of AI Tools in Auditing Adherence to Pre-Analysis Plans / Jeffrey Clemens, Anwita Mahajan.
- Format:
- Book
- Author/Creator:
- Clemens, Jeffrey.
- Series:
- Working Paper Series (National Bureau of Economic Research) no. w35719.
- NBER working paper series no. w35719
- Language:
- English
- Physical Description:
- 1 online resource: illustrations (black and white);
- Place of Publication:
- Cambridge, Mass. National Bureau of Economic Research 2026.
- Summary:
- Pre-analysis plans (PAPs) can improve research reproducibility by reducing researchers' degrees of freedom, but their value depends on adherence. We argue that large language models (LLMs) can provide an efficient, scalable, and systematic way for authors and reviewers to assess adherence to PAPs. In an application to our own research, an LLM systematically identifies precommitted design choices, evaluates deviations, and diagnoses gaps in pre-specification, substantially reducing the human labor required for these tasks. However, variability in audit output across LLMs underscores the continued importance of human judgment. We discuss implications for best practices in AI-assisted PAP auditing.
- Notes:
- September 2026.
- Print version record
The Penn Libraries is committed to describing library materials using current, accurate, and responsible language. If you discover outdated or inaccurate language, please fill out this feedback form to report it and suggest alternative language.