Evaluating AIDS prevention programs / Susan L. Coyle, Robert F. Boruch, and Charles F. Turner, editors ; Panel on the Evaluation of AIDS Interventions, Committee on AIDS Research and the Behavioral, Social, and Statistical Sciences, Commission on the Behavioral and Social Sciences and Education, National Research Council.

Format:

Book

Author/Creator:

National Research Council (U.S.). Panel on the Evaluation of AIDS Interventions.

Contributor:

Coyle, Susan L.

Boruch, Robert F.

Turner, Charles F.

Language:

English

Subjects (All):

AIDS (Disease)--Prevention--Evaluation--Methodology.

AIDS (Disease).

Physical Description:

xii, 376 p. : ill.

Edition:

Expanded ed.

Place of Publication:

Washington, D.C. : National Academy Press, 1991.

Language Note:

English

Summary:

With insightful discussion of program evaluation and the efforts of the Centers for Disease Control, this book presents a set of clear-cut recommendations to help ensure that the substantial resources devoted to the fight against AIDS will be used most effectively. This expanded edition of Evaluating AIDS Prevention Programs covers evaluation strategies and outcome measurements, including a realistic review of the factors that make evaluation of AIDS programs particularly difficult. Randomized field experiments are examined, focusing on the use of alternative treatments rather than placebo controls. The book also reviews nonexperimental techniques, including a critical examination of evaluation methods that are observational rather than experimental--a necessity when randomized experiments are infeasible.

Contents:

Evaluating AIDS Prevention Programs

Copyright

Preface

Acknowledgments

NOTE ON CONTRIBUTIONS

Contents

Summary

BACKGROUND

EVALUATION: NEEDS AND IMPLEMENTATION

OUTCOMES

THE MEDIA CAMPAIGN

COMMUNITY-BASED ORGANIZATIONS

HIV TESTING AND COUNSELING

RANDOMIZED AND OBSERVATIONAL APPROACHES TO EVALUATION

SPECIFIC RECOMMENDATIONS

All AIDS Intervention Programs

National AIDS Media Campaign

Community-Based Organizations

Testing and Counseling

1 Design and Implementation of Evaluation Research

TYPES OF EVALUATION

EVALUATION RESEARCH DESIGN

Process Evaluation Designs

Outcome Evaluation Designs

Nonexperimental and Quasi-Experimental Designs

Randomized Experiments

Rationale

Pitfalls

Unit of Assignment

Choice of Methods

THE MANAGEMENT OF EVALUATION

Project Selection

Research Administration

Conducting the Research

Independent Oversight

Agency In-House Team

Interagency Collaboration

Costs of Evaluation

REFERENCES

2 Measurement of Outcomes

PROGRAM OBJECTIVES

OUTCOMES FOR EVALUATIONS OF HIV PREVENTION PROGRAMS

Biological Outcomes

Strengths and Weaknesses of Biological Outcomes

Behavioral Outcomes

Primary Prevention Behaviors

Risk Reduction

Protective Behaviors

Complementary Prevention Behaviors

Strengths and Weaknesses of Behavioral Outcomes

Psychological Outcomes

Strengths and Weaknesses of Psychological Outcomes

EVALUATION MEASURES

Timing of Measurement

Quality of Measures

3 Evaluating Media Campaigns

BACKGROUND AND OBJECTIVES

FORMATIVE EVALUATION: WHAT WORKS BETTER?

Step 1: Idea Generation

Step 2: Concept Testing

Step 3: The Positioning Statement

Step 4: Copy Testing

Methodological Issues

Resources and Aspirations.

EFFICACY TRIALS: CAN THE CAMPAIGN MAKE A DIFFERENCE?

Resources and Aspirations

PROCESS EVALUATION: WHAT IS ACTUALLY DELIVERED?

OUTCOME EVALUATION: DOES THE CAMPAIGN MAKE A DIFFERENCE?

The National Health Interview Survey

Hotline Calls

Other Archival Sources

Effectiveness Evaluation of Current Activities

Effectiveness Evaluation of Future Activities

Problems with Sources of Data

The NHIS and Other Surveys

Hotline Calls and Other Archival Data

4 Evaluating Health Education and Risk Reduction Projects

WHAT SERVICES ARE DELIVERED?

Case Studies of a Sample of Projects

Sample

Data Collection

Analysis

Standardized Administrative Reporting

A Census or Sample Survey

Recommended Combination of Strategies

DO THE PROJECTS MAKE A DIFFERENCE?

Before-and-After Evaluation Designs

Randomized Field Studies

WHAT WORKS BETTER?

5 Evaluating HIV Testing and Counseling Projects

HOW WELL ARE SERVICES DELIVERED?

A Site Services Inventory

Client Surveys

Population Surveys

Case Studies Using Direct Observation

OPTIONS FOR EVALUATING WHETHER HIV TESTING AND COUNSELING SERVICES MAKE A DIFFERENCE

Randomized Experiments of Alternative Treatments

Experimental Regimens

Service Delivery Setting

Content, Duration, and Intensity of Counseling

Additional Services

Methodological Issues.

Resources and Aspirations

6 Randomized and Observational Approaches to Evaluating the Effectiveness of AIDS Prevention Programs

OVERVIEW

Choosing Among Strategies

RANDOMIZED EXPERIMENTATION

The Power of Experiments: An Example

Compromised Randomization

Attrition

Compliance

Spillover

Compensatory Behavior

Salvaging Compromised Experiments

When Should Randomized Experiments Be Considered?

Is a Randomized Experiment Appropriate?

What Is Being Asked?

Timeliness

Is It Feasible?

Affordability

Resources for Evaluation

Acceptability

Logistics of Randomized Assignment

DESIGNING COMPARABILITY INTO NONRANDOMIZED STUDIES

Quasi-Experiments

Interrupted Time Series

Assumptions

Data Needs

Inferences

Regression Discontinuity or Regression Displacement

Assumptions.

Existing Data Sources for Use in Quasi-Experimental Designs

The Neonatal Screening Survey.

Natural Experiments

Identifying Natural Experiments

Assumptions of Natural Experiments

Data Needs of Natural Experiments

Matching Without Randomization

Prospective Nonrandomized Matching

Retrospective Nonrandomized Matching

Existing Data Sources for Matching Without Randomization

Cohorts of Gay Men

Cohorts of Intravenous Drug Users

MODELING AND STATISTICAL ADJUSTMENTS FOR BIAS

Analysis of Covariance

Structural Equation and Selection Models

Structural Equation Models

Selection Models

Selection Models and Natural Experiments

Selection Modeling and Historical Controls

Assumptions of Modeling

Data Needs of Models

Inferences from Modeling

The Role of Models.

WHEN SHOULD NONRANDOMIZED APPROACHES BE CONSIDERED?

INTERPRETING EVALUATION RESULTS

Nonrandomized Methods

Accessibility of Assumptions

Interpretation

Appendixes

A Collaborative Contracting Strategy

References

B Oversight and Coordination Strategy

The Project Review Team

Operations

Summary and Discussion

C Methodological Issues in AIDS Surveys

Introduction

Fallibility of Measurement in Other Sciences

Recruitment of Respondents in Sex and Seroprevalence Surveys

Scope of the Review

Participation in Sex Surveys

Data Collection Procedures and Response Rates

Survey Configurations Associated with High Response Rates

Use of Telephone Surveys

Participation in Seroprevalence Surveys

Nonresponse Bias in Sex and Seroprevalence Surveys

Nonsampling Issues in Aids Surveys

Terms and Concepts

Survey Measurement of Sexual Behaviors

Overview

Inference in the Presence of Bias

Assumption of Constant Bias in Measurements

Approaches to Validation

Validation Using STD Rates

Psychometric Approaches to Validity

Empirical Studies of Sexual Behaviors

Validation

Partner Reports

Other Validation Techniques

Replication of Surveys on Samples of the Same Population

Proportion of Teenagers Who Are Sexually Active

Number of Sexual Partners Reported by Adults

Replication of Measurements Using Same Respondents

Empirical Studies of Drug-Using Behaviors

Accuracy of Self-Reports of Drug Use Behaviors

Measurement Bias

Summary of Findings

Feasibility

Replicability

Validity

Reliability

Improving Validity and Reliability

Literacy

Alternatives to Self-Reports

Physical Evidence

Skills Demonstrations

Other Safeguards for Surveys

Randomized Response Techniques

Pilot Studies.

Pretests

Cognitive Research Strategies

Ethnographic Studies

Examples of Studies Related to HIV Transmission

Male-Male Sexual Contacts

Variation in Drug Use Patterns

Ethnographic Methods

Ethnographic Methods in AIDS Research

Findings of Ethnographic Research on AIDS

Gaps and Deficiencies in Current Ethnographic Research

Recommendations

D Sampling and Randomization: Technical Questions about Evaluating CDC's Three Major AIDS Prevention Programs

Sampling Issues

Number of Case Studies

Estimating Sample Sizes

Controlling Attrition

Confidentiality Guarantees

Compensation

Stabilization Funds

Cultivating and Tracking Respondents

Personnel for Tracking Respondents

Modeling Attrition

Convenience and Probability Sampling

Sample Studies of Gay and Bisexual Men

Sample Studies of Intravenous Drug Users

Randomization

Examples of Randomized Experiments

The Ethics of No-treatment Controls

E Ancillary, Emerging, and Related Projects

Evaluating Clearinghouse Materials

Background and Objectives

Does It Work?

What Works Better?

Evaluating Referral Services of the Counseling and Testing Program

Monitoring Services at Non-CDC Testing Sites

F The Use of Selection Modeling to Evaluate AIDS Interventions with Observational Data

I. Introduction

II. Historical Development of Econometric Methods for Program Evaluation

III. The Statistics of Program Evaluation with Observational Data

The Problem

Solutions

Solution 1: Identifying Variables ("Z's")

Solution 2: Parametric Distributional Assumptions on

Solution 3: Availability of Cohort Data

The Relationship between Data Availability and Testing of Assumptions

IV. Application to Aids Interventions

The Search for Z's

Collection of Histories.

V. Summary and Conclusions.

Notes:

Bibliographic Level Mode of Issuance: Monograph

Includes bibliographical references and index.

ISBN:

1-280-21236-5

9786610212361

0-309-55567-1

0-585-15561-5

OCLC:

923267668