Evaluating Program Effectiveness : Validity and Decision-Making in Outcome Evaluation.

Format:

Book

Author/Creator:

Braverman, Marc T.

Series:

Evaluation in Practice

Language:

English

Subjects (All):

Evaluation research (Social action programs).

Physical Description:

1 online resource (281 pages)

Edition:

1st ed.

Place of Publication:

Thousand Oaks : SAGE Publications, Incorporated, 2022.

Summary:

The book shows how to apply validity concepts when planning and conducting an evaluation, thereby making sure that the evaluation stays true to the purposes for which it was initiated. Marc T. Braverman demonstrates that evaluating with validity means being able to answer the evaluation questions in a way that is useful, accurate, and reflective of the information needed from the evaluation. His approach is practical, with the goal of helping evaluators to conduct high-quality outcome evaluations, and he illustrates concepts with case study examples drawn from studies of intervention effectiveness over many years.

Contents:

Cover

Half Title Page

Title Page

Copyright Page

Brief Contents

Detailed Contents

Acknowledgments

Preface

About the Author

Part I. An Overview of Validity Concepts

Chapter 1. Introduction

What Is Accuracy in Evaluation?

Evaluation Questions and the Evaluation Plan

Evaluations Sometimes Lose Their Way

Evaluation as Argument

The Role and Significance of Validity

Constructs and Construct Validity

Construct Validity's Utility in Assessing Program Theory and Evaluation Plans

Operationalizing in Evaluation

Chapter Summary

Chapter 2. Validity in Testing and Psychometrics

Validity in the Psychometric Tradition

Evolution of the Conceptualization of Validity

Introduction of the Standards

Tests and Their Constructs

The Traditional, Tripartite View of Validity

Criterion-Related Validity

Content Validity

Construct Validity

Limitations of the Tripartite View

The Current View: Validity as a Unitary Phenomenon

Approaches to the Process of Test Validation

The Argument Perspective on Validity in Measurement

Chapter 3. Interpretations of Validity in the Theory and Practice of Evaluation

Validity Concepts Applied to Research Designs

Donald Campbell and the Expansion of the Validity Domain: Internal and External Validity

Cook and Campbell (1979) and Shadish et al. (2002): The Validity Full Monty

Ernest House: The Validity of Evaluations

Cultural Influences in Evaluation and Validity

Karen Kirkhart: Multicultural Validity

Part II. Validity Applied to Phases of the Evaluation Process

Chapter 4. Conceptualizing Your Intervention

What Exactly Is Your Intervention? The Intervention as Construct

Distinguishing Between the Intervention as Planned and as Delivered

Construct Considerations Applied to Policies.

The Interaction of the Intervention With Its Context

Understanding the Intervention as Planned: Program Theory and Logic Models

Program Theory

The Logic Model and the Logistics of Implementation

Other Information About the Intervention as Planned

Understanding the Intervention as Implemented

Program Monitoring

Intervention Fidelity

Evaluation Activities Aimed at Understanding the Intervention's Operation

Multisite Interventions

Evaluating Independently Functioning Sites

Local Adaptations

The Evaluand as Label

Building the Knowledge Base: Comparing Interventions Across Evaluations

Chapter 5. Translating Your Constructs Into Variables

Identifying and Specifying the Variables to Be Investigated

A General Model of Variable Selection and Measurement Specification

Step 1: Specifying the Constructs for the Evaluation

Step 2: Identifying Variables

Step 3: Choosing the Measurement Strategies for the Variables

Step 4: Selecting and/or Developing Our Instruments

Levels of Specificity

Types of Variables to Account for in Your Program Theory

The Use of Proxy Variables to Represent the Primary Construct

The (Sometimes) Perilous Journey From Construct to Variable

Convenience

Organizational and Administrative Decisions

Changes in Conceptualization in the Translation From Research to Intervention Practice

The Evolution of Concepts over Time

The Understanding and Use of Constructs in the Political Domain: An Example From California in the 1980s

Nailing Down the Construct

The Task Force's Treatment of the Construct of Self-Esteem

The Legacy of the Task Force: A Failure to Clarify

Recommendations on Selecting Your Variables

Know the Field and the Literature

Identify the Alternatives

Consult With Your Primary Stakeholders.

Make Sure Your Choices Are Logically Defensible

Chapter 6. Measurement Strategies and Measurement Instruments

Measurement and Validity

Measurement and Campbell's Validity Typology

Auxiliary Measurement Theories

The Reliability of Measures

Identifying Your Measurement Strategies

Considerations in Selecting a Strategy

Using Multiple Strategies for the Same Construct

Identifying Your Specific Measures

Single Item or Scale?

Consistency Across Studies When Doing Replications

Using Extant Measures

Developing Your Own Measures

Measures Not Based on Human Response

Anticipating Potential Sources of Bias and Error

Asking About Sensitive Subject Matter

Overly Restricted Response Range: Ceiling and Floor Effects

Memory Demands

Incomplete Understanding of the Question

Question Wording, Context, and Framing Effects

Effects of Question Order and Placement

Chapter 7. Evaluation Design

The Function of Evaluation Design

Estimating the Counterfactual Condition

Significant Elements of Impact Evaluation Design

Study Participants (or Other Units): Who and How Many?

Choice of Study Conditions

Assignment of Participants to Conditions

Number and Timing of Observation Points

Critical Comparisons Between the Conditions

Evaluation Designs and Validity Theory

Internal Validity

External Validity

Statistical Conclusion Validity

What Makes a Design Strong?

Alignment of the Design With the Evaluation Questions

Answering the Evaluation Questions With High Confidence: Minimizing Threats to Validity

Meeting Assumptions

Adequate Statistical Power

Transparency and Built-in Protections Against the Possibility of Deliberate Bias

Optimal Use of Available Resources.

Taking Account of Limitations and Weaknesses in the Evaluation Study Design

Chapter 8. Data Analysis

Data Analysis and Evaluation Validity

The Core Analysis

Mediators and Moderators

Null Hypothesis Significance Testing and p-Values

The Critique of Statistical Significance Testing

More Advanced Data Analysis Issues

Post Hoc and Exploratory Analyses

Would It Be Helpful to Conduct the Analysis in Different Ways?

Ad hoc Variables

The Ethical Limits of Fishing

Analysis Decisions and Adjustments

Analysis Decisions Related to the Evaluation Measures and Scores

Analysis Decisions Related to the Evaluation Participants

Analysis Decisions Related to Defining and Understanding the Intervention

The Creation of New Evaluation Questions

Chapter 9. Evaluation Conclusions and Recommendations

Synthesizing the Evaluation's Results to Arrive at Conclusions

Providing Recommendations

Framing the Problem

The Validity of Conclusions From Campbell's Validity Perspective

The Validity of Conclusions From House's Validity Perspective

Truth

Coherence

Justice

References

Index.

Notes:

Description based on publisher supplied metadata and other sources.

ISBN:

1-5063-5157-3

1-5063-5160-3

1-5063-5158-1

1-5063-5161-1

9781506351575

OCLC:

1350442664

Publisher Number:

252253