Selection Criteria
The success of a community’s violence prevention efforts depends, in large part, on the interventions used. That is why it is imperative to identify approaches that have been proven effective. Although a program model can rarely, if ever, be proved superior to all others, a particular model elicits greater confidence after its theoretical rationale, goals and objectives, and outcome evaluation data have been carefully reviewed. Although various scholarly reviews have identified exemplary programs, the methodological standards used in evaluating program effectiveness can vary greatly. A few of these scholarly reviews actually score each assessed program evaluation on its methodological rigor, but for most, the standards are variable and seldom made explicit. Evaluations of many of the programs did not employ true experimental designs and did not demonstrate statistically significant results. In addition, very few of these recommended programs have been replicated at multiple sites or have demonstrated a sustained deterrent effect after the program ended. In essence, the standard for the claims of program effectiveness in most of these reviews is very low.
In contrast, Blueprints programs meet the most rigorous tests of effectiveness in the field. The most important criterion to be considered when reviewing program effectiveness is evidence of deterrent effect when using a strong research design. Other important criteria are sustained effects and multiple site replication. Blueprints model programs must meet all three of these criteria, while promising programs must meet only the first criterion. Although additional factors are considered in the selection process, the three criteria discussed below are given greater weight.
Evidence of Deterrent Effect When Using a Strong Research Design
As noted above, this is the most important of the selection criteria. Relatively few programs have demonstrated effectiveness in reducing the onset, prevalence, or individual offending rates of violent behavior for juveniles. The Blueprints advisory board accepts evidence of deterrent effects for three key indicators—violence (including childhood aggression and conduct disorder), drug use, and/or delinquency—as evidence of program effectiveness.
Providing sufficient quantitative data to document effectiveness requires the use of evaluative designs that afford reasonable confidence in the findings (e.g., experimental designs with random assignment or quasi-experimental designs with matched control groups).
Most researchers recognize random assignment studies (randomized trials) executed faithfully as providing the highest standard of program evaluation. Random assignments offer the most compelling evidence that study results are due to the intervention rather than to preexisting differences between experimental and control groups and other threats to internal validity.3 In these studies, assignment to experimental or control conditions is determined solely by chance, and the likelihood of differences being attributed to the assignment process can be assessed.
|
||||
When random assignment cannot be used, the advisory board considers studies that use control groups matched as closely as possible to experimental groups on relevant characteristics (e.g., gender, race, age, socioeconomic status, income) and studies with control groups that use statistical techniques to control for initial differences on key variables. This is the most common method to be used when testing programs in a school environment, for example. As carefully as experimental and control groups are matched, however, it is impossible to determine if the groups will vary on some characteristics that have not been matched or controlled for and that are related to program outcome. Random assignment, therefore, is believed to be the most rigorous of methodological approaches.
Research designs vary greatly in quality, particularly with respect to several key aspects: sample size, attrition (loss of study participants over time), and measurement issues. At a minimum, the following issues need to be addressed:
- Sample sizes must be large enough to provide statistical power to detect effects. It is more difficult to detect statistically significant differences between groups when small sample sizes are used.
- Attrition may be indicative of problems in program implementation or may be a failure to locate subjects during a followup period. Attrition is dangerous, particularly because it can compromise the integrity of the original randomization or matching process. It reduces confidence that the original and final samples are comparable and that the final experimental and control comparisons reflect only treatment effects.
- Tests to measure outcomes must be administered fairly, accurately, and consistently to all study participants. For example, the use of inconsistent measures over time may produce less reliable test scores. The instruments used to measure outcomes should be demonstrated to be reliable and valid.
All Blueprints programs have used strong research designs and demonstrated significant effectiveness in reducing or preventing problem behaviors.
Sustained Effects
Although one criterion of program effectiveness is that the program demonstrate success by the end of the treatment phase, it is also important to demonstrate that these program effects endure beyond treatment and from one developmental period to the next. Designation as a Blueprints program requires a sustained effect at least 1 year beyond treatment, with no subsequent evidence that this effect is lost. Unfortunately, many programs that demonstrated initial success have failed to show long-term maintenance of the effects after the intervention has ended.
Depending on whether effects are immediate or delayed, the full impact of an intervention or treatment may not be realized at the end of treatment. Significant improvement may be realized over time, or a decay or decline may result. For example, if a program designed to prevent drug use (e.g., Life Skills Training) demonstrates its effectiveness in middle/junior high school when youth first begin encountering peer offers to use drugs, it is also important to demonstrate that these effects are sustained over a longer period of time. Unless this protective effect is sustained through high school, the program is unlikely to have an impact when problem behavior is at its peak. Although programs that have specifically failed to produce a sustained effect do not qualify as Blueprints model programs, those programs that have not yet demonstrated long-term effects may be considered as promising programs.
Multiple Site Replication
Replication is an important element in establishing program effectiveness and understanding what works best, in which situations, and with whom. Some programs are successful because of unique characteristics in the original site that may be difficult to duplicate in another site (e.g., having a charismatic leader or extensive community support and involvement). Replication establishes the strength of a program and its prevention effects and demonstrates that it can be successfully implemented in other sites.
Programs that have demonstrated success in diverse settings (e.g., urban, suburban, and rural areas) and with diverse populations (e.g., different socioeconomic, racial, and cultural groups) create greater confidence that such programs can be transferred to new settings. As communities prepare to tackle the problems of violence, delinquency, substance abuse, school dropout, and teenage pregnancy, knowledge that a specific program has had success in various settings with similar populations adds to its credibility.
Some projects may initially be implemented as a multisite single design (i.e., several sites are included in the evaluation design). When this occurs, it is preferable to review evaluation results from each site rather than pool the results to assess the project’s transportability to other sites. Single site designs with no replication are the least acceptable. Becoming a Blueprints model program requires at least one replication with demonstrated effects.
Additional Factors
In the selection of Blueprints programs, two additional factors are considered: whether a program conducted an analysis of mediating factors and whether a program is cost effective.
Analysis of mediating factors. The Blueprints advisory board looks for evidence that change in the targeted risk or protective factor(s) mediates the change in violent behavior. This evidence clearly strengthens the claim that program participation is responsible for changes in violent behavior, and it contributes to the theoretical understanding of the causal processes involved. In its reviews of different programs, the advisory board has discovered that many programs reporting significant deterrent “main effects” have not collected the data necessary to complete an analysis of mediating factors.
Costs versus benefits. Program costs should be reasonable and should be less or no greater than the program’s expected benefits. High-price-tag programs are difficult to sustain when competition is high and funding resources are low. Implementing expensive programs that will, at best, have small effects on violence is counterproductive. Although outcome evaluation research established that Blueprints programs were effective in reducing violence, delinquency, and drug use, very few data were available initially regarding the costs associated with replicating these violence prevention programs. A meta-analysis of mean effect sizes4 of “best” programs had shown that such programs (e.g., behavioral programs, interpersonal skills training, multiple services programs) could reduce recidivism rates between 20 and 40 percent (Lipsey, 1992, 1999).5 What was unclear was whether the benefits of providing these programs outweighed the costs. Two recent cost-benefit studies involving Blueprints programs—the RAND Corporation study (Greenwood et al., 1996) and a study of the Washington State Institute for Public Policy (Aos et al., 1999)—suggest that they are cost effective.
The RAND Corporation study (Greenwood et al., 1996) compared four different crime prevention approaches with California’s “Three Strikes and You’re Out” law (Cal. Penal Code § 667, 1994), which guarantees extended sentences for repeat offenders, to determine the benefits of programs that divert at-risk youth from crime. The four approaches were high school graduation incentives (e.g., Quantum Opportunities Program), parent training (e.g., Functional Family Therapy), delinquent supervision (e.g., programs using intensive supervision), and home visitation/daycare (e.g., Prenatal and Infancy Home Visitation by Nurses, Perry Preschool Program).
Three of these approaches (graduation incentives, parent training, and delinquent supervision) compared favorably with the Three Strikes law in terms of serious crime prevented per dollar expended; the first two were dramatically more cost effective. Graduation incentives were the most cost effective, preventing 250 serious crimes per $1 million program dollars (a cost of $4,000 per serious crime prevented). Parent training interventions were the next most cost effective, preventing 160 crimes per $1 million spent ($6,500 per serious crime prevented).
These two approaches were much more cost effective than the Three Strikes law, which prevented 62 crimes per $1 million spent ($16,000 per serious crime prevented). Although the comparison was less dramatic for the delinquent supervision interventions, which prevented about 70 crimes per $1 million spent ($14,000 per serious crime prevented), these interventions also were more cost effective than the Three Strikes law.
Only the home visitation/daycare intervention was less cost effective than the Three Strikes law, preventing 11 crimes per $1 million spent ($90,000 per serious crime prevented). Because home visitation/daycare occurs during the first 5 years of childhood, up to 15 years pass before the intervention can begin to affect serious street crimes, which typically occur as youth enter puberty. Child abuse, however, is affected immediately and is typically not included in cost-benefit analyses focusing solely on criminal justice cost savings. Other substantial favorable results found in areas such as child health and development, educational achievement, and economic well-being, which generate government savings that exceed program costs, are not included in these analyses.
The Washington State Institute for Public Policy (Aos et al., 1999) completed a cost-benefit study of programs dealing with age groups from early childhood through adulthood. The Institute examined the costs of crime to taxpayers (i.e., criminal justice costs) and to victims who suffer personal and property losses. Several programs had benefits that exceeded costs. The table below presents the costs versus the benefits of Blueprints programs examined in this study.
Programs designed for juvenile offenders had the largest and most consistent economic returns (e.g., Functional Family Therapy, Multisystemic Therapy, Multidimensional Treatment Foster Care). Programs that target younger juveniles who are not already involved in the criminal justice system have smaller returns when considering savings in criminal justice costs. Their benefits, however, can be calculated in other ways, such as savings to the health and welfare system. Many of these programs, such as the Prenatal and Infancy Home Visitation by Nurses and Quantum Opportunities programs, have reduced the amount of Temporary Assistance to Needy Families (formerly called Aid to Families with Dependent Children), the number of babies born with low birth weight, the incidence of school dropout, and the unemployment rate.
Cost-Benefit Analysis of Selected Blueprints Programs
† Taxpayer cost savings per participant divided by the cost per participant. Values greater than $1.00 indicate that the program’s crime-reducing benefits are greater than its costs.
‡ Taxpayer and crime victim cost savings per participant divided by the cost per participant.
Source: Adapted from Aos et al.,1999.
3 An evaluation is valid internally if it is able to eliminate all alternative explanations for a program’s results. Threats to internal validity include maturation, selection bias, and testing effects. 4 Effect size is the change in outcome as a result of an intervention. 5 The classification of “best” programs was determined by this meta-analysis, which is defined as “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings” (Glass, 1976:3).
| Previous | Contents | Next |
 |
|
| Blueprints for Violence Prevention | |