Technical issues / Alexandra K. Wigdor and Bert F. Green, Jr., editors ; Committee on the Performance of Military Personnel, Commission on Behavioral and Social Sciences and Education, National Research Council.

Format:

Book

Contributor:

Green, Bert F.

Wigdor, Alexandra K.

National Research Council (U.S.). Committee on the Performance of Military Personnel.

Series:

Performance assessment for the workplace ; v.2

Language:

English

Subjects (All):

Soldiers--United States--Classification.

Soldiers.

Soldiers--United States--Examinations.

United States--Armed Forces--Personnel management.

United States.

Physical Description:

1 online resource (344 pages)

Edition:

1st ed.

Place of Publication:

Washington, D.C. : National Academy Press, 1991.

Summary:

Volume II covers a number of measurement and analytical issues in greater technical detail, including: range restriction adjustments, methods for evaluating multiple sources of error in measurement, comparing alternative measures of performance, and strategies for clustering military occupations.

Contents:

Performance Assessment for the Workplace

Copyright

Preface

Contents

Implications of Cognitive Psychology for Measuring Job Performance

INTRODUCTION

COGNITIVE COMPONENTS OF SKILL

Procedural Content

Goal Structure

Basic and Prerequisite Abilities (Operators)

Procedures (Methods)

Selection Rules

Conceptual Knowledge

Task Structure

Background Knowledge

Context of Use

Critical Mental Models

Knowledge Engineering is One Aspect of Cognitive Task Analysis

Levels of Acquisition

Declarative Knowledge

Skill Automation

Skill Refinement

Summary

METHODS FOR COGNITIVE TASK ANALYSIS MEASUREMENT

Procedure Ordering Tasks

Sorting Tasks

Realistic Troubleshooting Tasks

Connection Specification Tasks

What-How-Why Tasks

AREAS OF THE MILITARY WHERE COGNITIVE TECHNIQUES HAVE PROMISE

REFERENCES

Work Samples as Measures of Performance

Theoretical Bases for Work Sample Testing

Previous Reviews of Work Sample Testing

Meta-analytic Reviews of Validity Studies Involving Work Samples

WORK SAMPLES AS PREDICTORS

Assessing Trainability Using Work Samples

WORK SAMPLES AS CRITERIA

Physical Ability as a Predictor of Work Sample Performance

Paper-and-Pencil Tests as Predictors of Work Sample Performance

Performance Ratings Validated Against Work Samples

Behavior Modeling Measured by Work Sample Performance

ASSESSMENT CENTERS

MEASUREMENT ISSUES

Reliability

Response Formats for Work Sample Evaluations

ADVERSE IMPACT

CONCLUSION

Measuring Job Competency

THE RECOMMENDATION TO MEASURE COMPETENCY

Performance-Based Selection Standards

Interpreting Performance Scores

Domain-Referenced Testing

Advantages of a Domain-Referenced Scale

Validity

Entrance Standards

Allocation

Quality Needs.

Ancillary Uses of the Job Performance Measures

OPERATIONALIZING THE COMPETENCY IDEA

Representing the Job Domain

Test Scoring Strategies

Scoring a Task

Combining Task Scores to Obtain Test Scores

Interpretive Scale Anchors

Taking Account of Experience

Scale Comparability

CONCLUSIONS

APPENDIX

Inferring a Scoring Procedure from Expert Judges

The Evaluation of Alternative Measures of Job Performance

The Criterion Problem in Personnel Research

Evolution of the Criterion Problem

THE NATURE OF CRITERION EQUIVALENCE

Five Major Facets of Equivalence

Susceptibility to Compromise

Financial Cost

Acceptability to Interested Parties

Weighting Facets of Nonequivalence by Importance

MAJOR ISSUES IN THE VALIDATION OF CRITERION MEASURES

The Nature of Validation for Criterion Measures

Construct Validity and Relevance

The Role of Content-Oriented Test Development

Criterion Bias against Subgroups

STRATEGIES FOR ASSESSING NONEQUIVALENCIES IN CRITERION VALIDITY

Assessing Nonequivalencies in Construct Validity: Correlational Methods

Assessing Nonequivalencies in Construct Validity: Other Methods

Assessing Nonequivalencies in Criterion Relevance

The Impact of Measurement Limitations on Validation

Sampling Error

Unreliability

Differential Restriction in Range across Criterion Measures

Criterion Measures Not All Available in Same Sample

SUMMARY

Outline of a Strategy for Assessing Criterion Equivalence

Range Restriction Adjustments in the Prediction of Military Job Performance

SELECTION EFFECTS IN CORRELATION AND REGRESSION

Coping With the Effects of Selection

Corrections for Sample Selection Bias

Technical Considerations and the Accuracy of Pearson-Lawley Corrections.

Alternative Adjustment Procedures

IMPLICATIONS FOR PREDICTIVE VALIDITY IN A JOINT-SERVICE CONTEXT

Analytical Procedure

Reference Population

Selection Variables

CONCLUDING REMARKS

Alternatives to the Validity Coefficient for Reporting the Test-Criterion Relationship

Generalizability Theory and Military Performance Measurements: I. Individual Performance

APPLICATION OF GENERALIZABILITY THEORY TO THE MEASUREMENT OF MILITARY PERFORMANCE

Background

Performance Measurement: Operate and Maintain Caliber .38 Revolver

Classical Theory Approach

Generalizability Theory Approach

Variance Components

Improvement of Performance Measurement

Generalizability of the Performance Measurement

Summary: Revolver Test With Accuracy Scores

Design of the Revolver Test Using Accuracy Scores

Variance Components and G Coefficients

Modifications for Future Decision Studies

SKETCH OF GENERALIZABILITY THEORY

Generalizability and Decision Studies

Relative and Absolute Decisions

Measurement Error for Relative Decisions

Measurement Error for Absolute Decisions

Generalizability Coefficients for Relative and Absolute Decisions

Random and Fixed Facets

G and D Studies With Crossed and Nested Facets

Multivariate Generalizability Theory

Symmetry in Behavioral Measurements

ILLUSTRATIVE APPLICATIONS OF GENERALIZABILITY THEORY

Generalizability of General Educational Development Ratings

G-Study Design

Univariate G Analysis

Multivariate G Analysis

Unbalanced Designs

Generalizability of Unit-Performance Measurements

Design of the G Study

Classical Reliability Approach

CONCLUDING COMMENTS ON GENERALIZABILITY THEORY: ISSUES AND LIMITATIONS.

Small Samples and Negative Estimated Variance Components

Small Samples and Variability of Estimated Variance Components

Negative Estimates of Variance Components

Assumption of Constant Universe Scores

Dichotomous Data

Concluding Comment

Procedures for Eliciting and Using Judgments of the Value of Observed Behaviors on Military Job Performance Tests

THE PROBLEMS ADDRESSED

Establishing Test Standards

Eliciting and Combining Judgments of the Worth of Job Performance Test Behaviors

Using Predicted Test Performances and Value Judgments in Personnel Classification

ESTABLISHING MINIMUM STANDARDS OF PERFORMANCE

Consequences of Setting Standards

Conventional Standard-Setting Procedures

Procedures That Require Judgments About Test Items

Procedures That Require Judgments About Examinees

Prospects for Applying Conventional Standard-Setting Procedures

Operational Questions and Issues

Types of Judges to be Used

Numbers of Judges to be Used

Stimulus Materials to be Used in Setting Standards

Training of Judges

Information to be Provided to Judges

Measurement Error

DEFINITION AND CONSTRUCTION OF VALUE FUNCTIONS

Defining Task Value Functions

Structuring Cluster and Military Occupational Specialty Value Functions

Eliciting Value Judgments

Average Value Function Procedure

Method of Successive Lotteries

CLASSIFICATION OF NEW ENLISTEES

Individual Classification Strategies

An Institutional Classification Strategy

Example 1

Example 2

Exploring Strategies for Clustering Military Occupations

CLUSTERING MILITARY OCCUPATIONS

JOB ANALYSIS METHODS: THE CHOICE OF THE JOB DESCRIPTOR.

EXTENDING ASVAB VALIDITY TO THE UNIVERSE OF MOS

Validity Generalization/Meta-Analysis

Linking Hands-On Performance Measures and Training Criteria

Synthetic Validity

McCormick's Job Component Model

J-coefficient

Judgmental Estimates of Validity

Paired Comparison Judgments of Validity

Determination of Minimum Standards

REFERENCES.

Notes:

Includes bibliographical references and index.

ISBN:

9786610203697

9781280203695

1280203692

9780309595193

0309595193

9780585170923

0585170924

OCLC:

923264523