Mismeasuring impact : how randomized controlled trials threaten the nonprofit sector / Nicole Marwell and Jennifer Mosley.

Format:

Book

Author/Creator:

Marwell, Nicole, author.

Mosley, Jennifer, author.

Series:

Stanford social innovation review books.

Stanford Social Innovation Review Bks

Language:

English

Subjects (All):

Nonprofit organizations--United States--Evaluation.

Nonprofit organizations.

Organizational effectiveness--United States--Evaluation.

Organizational effectiveness.

Randomized controlled trials (Social sciences)--United States.

Randomized controlled trials (Social sciences).

Physical Description:

1 online resource (208 pages)

Edition:

First edition.

Place of Publication:

Stanford, California : Stanford University Press, [2025]

System Details:

Mode of access: World Wide Web.

Summary:

"Demonstrating the effectiveness of nonprofit social programs is a challenge. This has led to a rapid rise in nonprofit organizations' use of randomized controlled trials (RCTs), for evaluation. As a result, most nonprofit sector professionals today can tell you why nonprofits should do an RCT. This book tells you why they probably shouldn't. Mismeasuring Impact explores why RCTs are being embraced as the "gold standard" for nonprofit evaluation, despite the high cost and time investment required, and the problems with using RCTs in a nonprofit context. The book describes what happens inside nonprofits when they take part in RCTs, the unintended equity issues that arise, and why nonprofits decide to participate in RCTs despite the problems. University of Chicago professors Marwell and Mosley's research is based on extensive interviews with key players: nonprofit managers, professional program evaluators, and program officers in philanthropic foundations. The book explains why most RCTs conducted in nonprofits fail to meet required standards for rigor, undercutting their accuracy. Utlimately, RCTs are used to create nonprofit legitimacy, not to foster nonprofit improvement. RCTs also privilege program and organizational standardization over the key strengths of nonprofit organizations: innovation and responsiveness to community needs. Nonprofits and funders need forms of evaluation that lift up these strengths, but RCTs fall short. Mismeasuring Impact offers alternatives that build strong organizations, not just standardized programs. The book concludes by suggesting evaluation approaches that funders and nonprofits of all sizes can support"-- Provided by publisher.

Contents:

Front Cover

Praise for Mismeasuring Impact

Half-title

Title Page

Copyright

Contents

Acknowledgments

1. Five Problems With RCTs, and Why They Threaten the Nonprofit Sector

The Case for RCTs

Are RCTs Really the "Gold Standard" for Nonprofits?

Five Problems with Using RCTs in Nonprofit Organizations

The "False Certainty" Problem

The "Programs Need Organizations" Problem

The "Communities Need Organizations" Problem

The "Rich Get Richer" Problem

The "Agility" Problem

Goals of This Book

Brief Answers to Some Common Objections to Our Argument

2. The Siren Song of RCTs: Building the Gold Standard Movement

How Did We Get Here? Understanding the Growth of RCTs for Social Programs

The Evidence Battle

The Funding Battle

Spreading RCTs to U.S. Nonprofits: The Social Innovation Fund

Changing the Conversation

3. Null Effects and Dumb Effects: Do RCTs Really Tell Us "What Works?"

The Ubiquity of Null Effects

Underpowered: The Importance of Sample Size

Randomization Failure

Control Group Contamination

The Tested Intervention Is Not Delivered Consistently

RCT Focuses on Easy-to-Measure, but Sometimes Inappropriate Outcomes

Dumb Effects: When Positive Effects Are Not Very Meaningful

Context Is Important, or What Works in St. Paul May Not Work in St. Louis

What Works in Typical Study Conditions Can Fail in Typical Practice Conditions

Overly Narrow Samples or Questions Lead to Unactionable Findings

RCTs Take a Long Time, While Social, Political, and Economic Conditions Have Changed

Moving Beyond RCTs

4. "This Was Harder Than We Thought": Implementing RCTs in Nonprofits

Twice the People, Twice the Work: RCT Recruiting Challenges Staff Capacity and Organizational Mission.

"It Sounded a Lot Easier Than It Actually Was": What It's Really Like for Nonprofits Involved in RCTs

Delays and (Financial) Drains: How RCTs Make Running an Organization Harder

RCTs Don't Just Evaluate, They Can Change Both Programs and Organizations

Putting Organizational Well-Being First

5. Looking for Legitimacy: Why We Keep Doing RCTs

RCTs Can Build Organizational Legitimacy

Does an RCT Always Enhance Nonprofit Legitimacy?

Evaluators See Limits on RCTs

The View from Philanthropy: How Foundation Program Officers Think About RCTs and Nonprofit Legitimacy

RCTs May Increase Legitimacy, but Not Without Consequences

6. What Kind of Work Should We Value?: How RCTs Undermine the Equity Goals of Nonprofits and the Communities They Serve

Is Pursuing Effectiveness the Same as Pursuing Equity?

Do RCTs Privilege Evidence-Building Over People?

The Rich Get Richer

Practicing Equity: A Key Goal for Philanthropy

Evaluation, Effectiveness, and Equity

7. A Bad Fit: How RCTs Hinder Nonprofit Responsiveness and Innovation

Long Time Horizon: The Information Generated by RCTs Comes After Organizations Have Already Moved On

The "Hold Still" Challenge: RCTs Limit Ongoing Program Responsiveness and Innovation

"You Get One Shot": RCTs Are Too Expensive and Complex to Be Used for Continual Improvement

A Black Box: RCTs Often Don't Identify Why a Program Works, Which Hinders Learning

The Need for Alternatives to the RCT . . . and More Evaluation Capacity Overall

Evaluation Should Better Support Responsiveness and Innovation

8. How Might We Think about Evaluation Differently?: Three Principles for Nonprofit Improvement

Principle 1: Tailor Evaluation to Specific Organizational Strategies and Community Needs

Principle 2: Centering Participant Perspectives Is Key to Nonprofit Performance.

Principle 3: Focus on Improvement Through Iteratively Addressing Problems Instead of Standardizing Solutions

An Improvement Orientation

9. Moving Beyond RCTs: Lessons for Nonprofit Leaders, Donors, Evaluators, and Policymakers

Evidence Matters

Five Problems with RCTs

When Is the Right Time to Do an RCT?

Three Principles for Using Evaluation to Foster Nonprofit Improvement

Recommendations for Building Evidence Outside the Evidence Hierarchy

Nonprofit Leaders and Managers

Evaluators

Foundation Program Officers and Trustees

Policymakers and Government Agency Staff

The Way Forward

Appendix: Research Methodology

Notes

Bibliography

Index

Back Cover.

Notes:

Includes bibliographical references and index.

Description based on publisher supplied metadata and other sources.

Description based on print version record.

ISBN:

9781503643123

1-5036-4312-3