skip navigation

PUBLICATIONS

Register for Latest Research

Stay Informed
Register with NCJRS to receive NCJRS's biweekly e-newsletter JUSTINFO and additional periodic emails from NCJRS and the NCJRS federal sponsors that highlight the latest research published or sponsored by the Office of Justice Programs.

NCJRS Abstract

The document referenced below is part of the NCJRS Virtual Library collection. To conduct further searches of the collection, visit the Virtual Library. See the Obtain Documents page for direction on how to access resources online, via mail, through interlibrary loans, or in a local library.

 

NCJ Number: 64397 Add to Shopping cart Find in a Library
Title: MEASURING IMPACT - POWER THEORY IN SOCIAL PROGRAM EVALUATION (FROM IMPROVING EVALUATIONS, 1979, BY LOIS-ELLIN DATTA AND ROBERT PERLOFF - SEE NCJ-64392)
Author(s): R F BORUCH; H GOMEZ
Corporate Author: Sage Publications, Inc
United States of America
Date Published: 1979
Page Count: 32
Sponsoring Agency: Sage Publications, Inc
Thousand Oaks, CA 91320
US Dept of Health, Education, and Welfare
Washington, DC 20203
Publication Number: C-74-0115
Type: Program/Project Evaluation
Format: Document
Language: English
Country: United States of America
Annotation: THE MEASUREMENT OF TREATMENT AND RESPONSE IN SOCIAL PROGRAMS ARE EXAMINED IN ORDER TO BETTER ESTABLISH THE STATISTICAL POWER OF AN EVALUATION DESIGN, ENHANCE THAT POWER, AND UNDERSTAND THE EVALUATION RESULTS.
Abstract: THE DISCUSSION IS CONFINED TO OUTCOME EVALUATION: ESTIMATING IN THE LEAST EQUIVOCAL AND LEAST BIASED WAY POSSIBLE THE RELATIVE EFFECT OF A PROGRAM ON ITS TARGET GROUP. WEAK EVALUATION DESIGN OFTEN CAUSES THE FINDING OF 'NO SIGNIFICANT DIFFERENCES' IN PROGRAM EVALUATIONS. STATISTICAL POWER THEORY SHOULD THEREFORE BE AUGMENTED TO BETTER REFLECT THE REALITIES OF SOCIAL EXPERIMENTATION. THE PROBLEM OF ENSURING STATISTICAL POWER IN EVALUATION DESIGN RESULTS FROM LOSS OF STATISTICAL POWER WHEN A PROGRAM IS TRANSFERRED FROM THE LABORATORY TO THE FIELD. THIS DROP OCCURS BECAUSE OF THE PARTIAL IRRELEVANCE OF RESPONSE VARIABLES AND THE DEGRADATION OF TREATMENT VARIABLES. INVALIDITY OF RESPONSE AND INFIDELITY OF TREATMENT HAVE A MULTIPLICATIVE, RATHER THAN ADDITIVE, EFFECT ON STATISTICAL POWER. BY AUGMENTING THE USUAL ANALYSIS OF VARIANCE MODEL WITH SIMPLE REGRESSION MODELS WHICH RECOGNIZE VARIATION IN THE FIELD, AND BY COMPUTING THE STATISTICAL POWER FOR BOTH THE SIMPLE AND AUGMENTED MODELS, IT IS EASY TO SHOW THE DROP IN POWER. RESPONSE VARIABLES WHICH ARE RELEVANT TO THE PROGRAM MUST BE USED. SUBSTANTIVE THEORY, PRIOR DATA, PILOT TESTS, AND SIDE THEORIES ARE ALL CRITERIA FOR DETERMINING RELEVEANCE TO TREATMENT OF A RESPONSE VARIABLE. IDENTIFYING AND MEASURING TREATMENT IMPLEMENTATION IN FIELD STUDIES INVOLVES THREE PROBLEMS: POLICY PACKAGED AS PROGRAM, STRUCTURALLY INCOMPLETE PROGRAMS, AND INCOMPLETE PROGRAM RECEPTION OR DELIVERY. PILOT TESTS OR SIDE STUDIES SHOULD BE USED TO PROVIDE INFORMATION ON THE EXTENT OF THESE PROBLEMS. THE PROBLEM MUST ALSO BE CHARACTERIZED ALGEBRAICALLY AS WELL AS LITERALLY. IT IS CONCLUDED THAT (1) ORTHODOX EXPERIMENTAL DESIGN AND QUASI-EXPERIMENTAL APPROACHES NEED TO BE AUGMENTED FOR EVALUATING SOCIAL PROGRAMS; (2) KNOWLEDGE FROM MANY DISCIPLINES SHOULD BE USED; (3) PROGRAM MANAGERS, EVALUATION DESIGNERS, MEASUREMENT SPECIALISTS, AND ACADEMIC THEORISTS SHOULD ALL BE INVOLVED IN EVALUATION DESIGN; AND (4) THE POWER OF DESIGNS SHOULD BE ROUTINELY TESTED AND PUBLISHED WITH EVALUATION RESULTS, BECAUSE SOCIAL PROGRAMS OFTEN PRODUCE SMALL EFFECTS. RESEARCH ON BOTH SUBSTANTIVE THEORY AND STATISTICAL MODELS IS RECOMMENDED. EXAMPLES FROM DIVERSE FIELDS, A REFERENCE LIST, AND AN APPENDIX PRESENTING ALGEBRAICALLY A THEORY OF MEASUREMENT IN FIELD EVALUATIONS, ARE INCLUDED. (CFW)
Index Term(s): Behavioral and Social Sciences; Evaluation; Evaluation measures; Evaluation techniques; Mathematical modeling; Program evaluation; Statistical analysis; Statistical bias; Testing and measurement
To cite this abstract, use the following link:
http://www.ncjrs.gov/App/publications/abstract.aspx?ID=64397

*A link to the full-text document is provided whenever possible. For documents not available online, a link to the publisher's website is provided. Tell us how you use the NCJRS Library and Abstracts Database - send us your feedback.