U.S. flag

An official website of the United States government, Department of Justice.

NCJRS Virtual Library

The Virtual Library houses over 235,000 criminal justice resources, including all known OJP works.
Click here to search the NCJRS Virtual Library

SENSITIVITY, BIAS, AND THEORY IN IMPACT EVALUATION (FROM EVALUATION STUDIES REVIEW ANNUAL, VOL 4, 1979, BY LEE SECHREST ET AL - SEE NCJ-65332)

NCJ Number
65334
Author(s)
R F BORUCH; H GOMEZ
Date Published
1979
Length
26 pages
Annotation
TO DEVELOP MORE RIGOROUS AND SOCIALLY BENEFICIAL PROGRAM EVALUATION, THIS ARTICLE IDENTIFIES AND ATTEMPTS TO SOLVE MEASUREMENT PROBLEMS WHICH, IN USUAL EXPERIMENTAL DESIGN, INVITE UNDERPOWERED (INSENSITIVE) EXPERIMENTS.
Abstract
MEASUREMENT SHOULD BE CONCERNED NOT ONLY WITH THE RELIABILITY OF THE DEPENDENT VARIABLE BUT, MORE IMPORTANTLY, WITH VALIDITY OF MEASUREMENT AND WITH MEASUREMENT, THROUGH SYSTEMATIC OBSERVATION, OF THE TREATMENT VARIABLE. IN AN IMPACT EVALUATION, WHICH ENTAILS THE MEASUREMENT OF THE RELATIVE EFFECT OF A PROGRAM ON ITS TARGET GROUP, EVALUATORS SHOULD BE ABLE TO ANTICIPATE THE POWER OF AN EVALUATION DESIGN, ENHANCE THAT POWER, AND UNDERSTAND THE EVALUATION RESULTS. WHEN A PROGRAM IS TRANSFERRED FROM THE LABORATORY TO THE FIELD, THE PRECIPITOUS DROP IN STATISTICAL POWER RESULTS FROM THE MULTIPLICATIVE EFFECTS OF INVALIDITY OF RESPONSE AND INFIDELITY OF TREATMENT. TO DEAL WITH THIS PROBLEM, ORTHODOX MEASUREMENT THEORY SHOULD BE AUGMENTED BY INPUT FROM PROGRAM MANAGERS, DEVELOPERS, AND THE SUBSTANTIVE THEORIST. THE RESPONSE VARIABLE MUST BE MEASURED USING CRITERIA RELEVANT TO THE TREATMENT VARIABLE. CRITERIA FOR DETERMINING RELEVANCE INCLUDE SUBSTANTIVE THEORY, PRIOR DATA, PILOT TESTS, SIDE THEORIES, AND OTHER MODELS FOR UNDERSTANDING THE RESPONSE VARIABLES' RELATIONSHIP TO THE ACTUAL STATUS OF THE INDIVIDUAL AND TO PROGRAM CONTENT. THE TREATMENT VARIABLE MUST ALSO BE EXAMINED, RATHER THAN ASSUMING THAT THE PRESCRIBED LEVEL OF TREATMENT IS EITHER PRESENT OR ABSENT. THE THREE MAIN PROBLEMS IN IDENTIFYING AND MEASURING TREATMENT IN FIELD STUDIES ARE (1) POLICY PACKAGED AS PROGRAM, (2) THE STRUCTURALLY INCOMPLETE PROGRAM, AND (3) INCOMPLETE PROGRAM RECEPTION OR DELIVERY AT THE INDIVIDUAL LEVEL. PILOT TESTS OR SIDE STUDIES, MOUNTED PRIOR TO A FULL-SCALE EVALUATION, SEEM WELL JUSTIFIED HERE. IT IS CONCLUDED THAT ORTHODOX EXPERIMENTAL DESIGN AND QUASI-EXPERIMENTAL APPROACHES NEED TO BE AUGMENTED FOR EVALUATING SOCIAL PROGRAMS AND THAT AN INTERDISCIPLINARY TEAM APPROACH SHOULD BE USED IN EVALUATION. REFERENCE NOTES, A REFERENCE LIST, AND AN APPENDIX PRESENTING THE ALGEBRAIC FORMULATION OF THE MEASUREMENT THEORY, ARE INCLUDED. (CFW)