Chapter Ten



by Lawrence W. Sherman

The effectiveness of most crime prevention strategies will remain unknown until the nation invests more in evaluating them. That is the central conclusion of this report. The inadequacy of that investment to date prevents a judgment for or against the effectiveness of the $3 Billion in federal crime funds, at least to a reasonable degree of scientific certainty. Using "rigorous and scientifically recognized standards and methodologies"--the mission Congress set out for this report--the review of over 500 impact evaluations reveals only a handful of conclusions that can be generalized from those studies to similar programs around the nation. By scientific standards, there are very few "programs of proven effectiveness."

This lack of evidence is not a reflection on the DOJ programs themselves. Nor does it reflect the quality of their administration. It is a direct consequence of the legislative plan for federal funding of local crime prevention. That consequence was not by Congressional intent. For over a quarter century, the Congress has asked the Department of Justice to evaluate the effectiveness of local crime prevention funding. But as the preceding Chapters show, the Congress has never provided the tools needed to get the job done. Those tools include adequate funding for program evaluation, and a structure of federal program funding that permits controlled testing of crime prevention effectiveness.

In order for Congress to learn whether DOJ programs are effective, it must provide a more balanced approach to program funding and evaluation. The latter cannot be accomplished without some small compromise in the principle of State and local control over how most of the federal funding is spent. While this principle may have many merits for the federal support of State and local operations, it is a roadblock to federal production of sound guidance on what works to prevent crime. Exclusively local control of funding conflicts with the scientific principles of controlled field testing, preventing DOJ-funded program evaluations from using "rigorous and scientifically recognized standards and methodologies." A statutory evaluation plan can remove this obstacle by setting aside just ten percent of operational funding for federal-local partnerships to accomplish scientifically controlled field tests. This would allow DOJ to control program funding in ways that will help insure reliable evaluation results.

A secondary legislative obstacle to evaluating crime prevention is insufficient funding for employing scientifically recognized standards and methodologies. There are substantial costs of using such scientific evaluation techniques as victimization surveys, systematic observations of program implementation, field interviews of offenders and case screening for randomized controlled trials. These techniques can make evaluations cost as much as or more than the programs being evaluated. Under-funding of DOJ evaluations is ultimately wasteful, spending substantial amounts for descriptive evaluations but not enough to answer the primary questions about program effectiveness. Congress can solve this problem by limiting the scope of required evaluations, but requiring that evaluations that are funded receive sufficient funding levels to answer the important questions. In order for this approach to be effective, Congress must also match the ten percent of program funding earmarked for field-tested programs with another ten percent to pay for the evaluations.

The third legislative obstacle to evaluating crime prevention is the structural separation of research and program funding. From an evaluation perspective, this has often put the cart before the horse, forcing evaluators to play "catch-up" or "patch-up" in evaluating programs that are implemented too quickly or not at all. Evaluator control of operational funding for field testing programs is the standard approach in private industry and has good precedents within DOJ. Congress can remove the obstacle to this approach by appropriating the ten percent for field-tested operations and ten percent for evaluating those operations to the direct control of a central evaluation office within the Office of Justice Programs. Such an office can then be held directly accountable for the amount and strength of scientific evidence it produces.

This chapter summarizes what is known about the effectiveness of local operations supported by various DOJ funding mechanisms. The specific findings are reported in detail at the end of each of the seven preceding chapters. This chapter integrates those findings into a more critical assessment of the effectiveness of DOJ programs in addressing the known risk factors for crime and delinquency, especially youth violence, in each of those settings. This analysis centers on two key questions:

1. Using Knowledge. How well do DOJ-funded programs correspond to what is known about causes and prevention of crime?

2. Creating Knowledge. How well do DOJ-funded programs help to increase what is known about the effectiveness of crime prevention?

In general, DOJ-funded programs do a better job of using than creating knowledge. The answers to the first question show increasing responsiveness to accumulating research evidence on the risk factors associated with crime, and the systematic use of evaluation results in designing prevention programs. The exceptions to that conclusion are duly noted in each section below.

The answers to the second question are much less encouraging. Despite substantial improvements in DOJ's institutional capacity for crime prevention evaluation science, the legislative obstacles to using scientifically recognized methodologies have prevented DOJ from measuring the impact of most of its programs on crime. The chapter extends this conclusion into an analysis of the scientific requirements for improving the answers to the second question, concluding with the recommendation for the statutory evaluation plan summarized at the Chapter's outset.


The national debate over DOJ funding for crime prevention primarily concerns the relative effectiveness of investment in the different institutional settings. The current legislative outcome of that debate allocates more than half of all DOJ local assistance prevention funding to the police, and another quarter to the rest of the criminal justice system (including prisons). It would be helpful to draw upon a substantial body of evidence to assess the effectiveness of that allocation, but that evidence is unavailable. The current state of crime prevention science does not allow comparisons of crimes prevented per taxpayer dollar across different strategies in different institutional settings. A lack of comparable measures on lifetime effects of each program is compounded by problems in generalizing from local to national samples. For example, while the prevention effect of home nurse visitation is known for a sample of high-risk rural families, the national benefit of that program for all families on average is not known. Some simulations and estimates have been offered comparing early childhood to early adult prevention strategies (Greenwood, et al, 1996), but the empirical basis for such analysis is too limited to make such direct comparisons as police to infant visitation nurses, or school-based prevention programs to prison.

This report does suggest that the most money is going to settings where we have the most evidence about preventing serious crime. The only direct evidence showing programs that work to prevent predatory stranger violence is about police and prisons. While there are good theoretical reasons to believe that communities, families, schools, labor markets and places could be even more effective in preventing predatory stranger violence, that theory has not yet been turned into successfully tested practice. That gap may itself reflect a failure of imagination, or a bias toward testing as well as funding police and criminal justice programs. But in the current state of the evidence, it is fair to say we do not yet know how to spend a large operational investment in other institutional settings that may have profound effects on serious crime.

What we do know how to do is to create such knowledge, by investing substantial funding in rigorous tests of program innovations. The prime opportunities for that investment are clearly indicated by available research, such as universal infant visitation combined with parental-involvement preschool programs. While there is strong evidence that small pilot programs of this kind are effective in preventing child abuse and later delinquency, we do not have a tested model for operating such programs on a large scale. Creating such tests would be the next scientific step for providing the evidence appropriate to the large scale funding required. Unless new legislation is passed to authorize DOJ to pursue a "big science" program of testing universal early prevention, however, it seems likely that federal funding for crime prevention will continue to focused mostly on criminal justice and police--and perhaps miss the most effective approach to preventing serious violence.

Community-Based Prevention: OJP Discretionary Programs

The community is an institutional setting in which the Congress has given a clear mandate to DOJ for developing crime prevention strategies. Viewed from the perspective of a priority on youth violence, however, current legislation could provide more effective tools for using available knowledge, and especially for creating new knowledge, about what works in community-based prevention.

Using Knowledge. The relatively modest amounts of DOJ-funded discretionary programs focused on high-crime neighborhoods effectively use much available scientific knowledge about crime. Operation Weed and Seed, BJA's Comprehensive Communities Program, and OJJDP's Title V Incentive Grants all attempt to concentrate federally funded effort in the small number of communities where the national problems of serious youth violence are most heavily concentrated. Local grantees appear to vary in their use of serious violence as the primary criterion for selecting program neighborhoods, and the programs themselves vary in the extent to which such a focus is called for. Operation Weed and Seed is apparently the most focused on such areas, while all of them share a concern with addressing risk factors cutting across settings: families, schools, labor markets, places, police and criminal justice. While these programs are only a beginning of the long-term effort needed to learn how to combat the interdependent risk factors of hypersegregated urban poverty areas, they are clearly pointed in the direction indicated by the available science. The concentration of resources in high-risk areas makes better use of epidemiological knowledge about violence than the more costly Byrne Formula Grant population-based allocations across States and localities.

Congressional earmarks for community-based DOJ discretionary programs make less effective use of knowledge when allocating funds to programs for recreation, mentoring and crime prevention advertising. There is no strong evidence that these programs are effective in preventing serious youth violence, although Big Brothers/Big Sisters mentoring is "promising" for preventing substance abuse and recreation (Boys and Girls Clubs) is promising for reducing housing project vandalism. While it is possible that this approach is effective, there have simply been no longer term tests providing an adequate scientific basis for evaluation.

Community-based discretionary programs could be more effective in using knowledge about community risk factors if they did not define "comprehensive" programs primarily in terms of interagency collaboration. A more scientifically informed definition would be tied to effective impact on each of the institutional settings affecting community risk factors. This is particularly true of labor markets and families (see below). Finally, discretionary programs with anti-gang components could make more effective use of the available results from gang prevention and intervention evaluations, especially in cautioning grantees against programs that could increase gang cohesion.

Creating Knowledge. Community-based discretionary prevention programs are the prime example of the need for a Congressionally mandated evaluation plan. The current programs are not effective in creating knowledge about what works. Insufficient resources are available for the scientific techniques required to measure program content and impact. Insufficient federal control of the mix of local program elements prevents scientifically controlled comparisons within cities. Insufficient evaluator control of site selection prevents valid comparisons of results across cities, making the data in each site much less useful as a basis for drawing general conclusions about what works. The fact that no controlled experiment in crime prevention has ever used cities, or communities across cities, as a unit of analysis indicates the severe constraints the present legislation imposes upon the work needed to create useful knowledge about program impact.

Community-Based Prevention: OJP Formula Grants

Using Knowledge. DOJ-funded formula grants for community-based prevention may be less effective than the discretionary grants. The available, if moderately weak, scientific evidence shows that community mobilization strategies are ineffective, especially in high-crime neighborhoods. Statutory purpose areas for Byrne formula and Local Law Enforcement Block Grants include community crime prevention initiatives, funded at an estimated $50 million in FY 1996. Programs supported by these funds are most likely to be ineffective, and the funding could be better spent on creating knowledge about more effective approaches to community prevention.

The statutory plan for Byrne Formula Grants also fails to make effective use of epidemiological evidence on the geographic concentration of crime within States. Statutory allocation plans for Local Law Enforcement Block Grants make better use of that evidence in allocations across cities. Neither program incorporates the evidence on geographic concentrations within cities, by neighborhood and even specific places, which inform the Discretionary DOJ programs. Congressional direction of funding to high crime neighborhoods or Census tracts seems likely to be more effective than the current allocations, which may allow State and local decisions to put substantial funding in moderate to low-crime areas.

Creating Knowledge. The current formula grant legislation for community-based prevention lacks a viable statutory plan for evaluating that funding using scientifically recognized standards and methodologies. The current requirement that all Byrne Formula Grants include an evaluation is at best meaningless and at worst wasteful. While some States have attempted to invest more Byrne resources in scientific impact evaluations, the legislation offers no protection against the inevitable political pressures to spend as much money as possible on operational purposes. A quarter-century of State-level evaluation requirements and hundreds of millions of dollars spent for such purposes (Rubinstein, 1977a; Feeley and Sarat, 1980) has failed to produce scientifically rigorous, published impact evaluations that can increase the effectiveness of crime prevention. Uneven capacity across states in their institutional infrastructure needed for rigorous evaluation science compounds the insufficient funding and control needed to learn what works with Formula Grants for local crime prevention.

Family-Based Delinquency Prevention

The family is an institutional setting for crime prevention in which the Congressional mandate to DOJ is not clear. While the Office of Juvenile Justice and Delinquency Prevention has long worked on programs involving families in delinquency prevention, the appropriated funding emphasis has been more on criminal sanctions than family life. Requests by State and local juvenile justice officials for more prevention funding resulted in the 1992 enactment of the Title V delinquency prevention grants, but that program's mandate to work with families is enmeshed in a broader mandate for comprehensive mobilization of local youth-serving agencies. Similarly, the 1994 Crime Act's Violence Against Women grants imply a great deal about family-based prevention, but without a clear mandate for DOJ to work with families. Traditionally the domain of the Department of Health and Human Services, which has funded much of the available delinquency prevention research, the family is the new frontier of crime prevention science. The long-term effectiveness of the national effort to prevent crime may depend on both clarifying and expanding the mandate for DOJ support of family-based prevention efforts, perhaps in closer collaboration with HHS.

Using Knowledge. DOJ discretionary funding of family-based delinquency prevention appears effective in its use of available scientific knowledge. The five-year OJJDP Safe Futures Program in six cities (in collaboration with the Violence Against Women Grants Office and the Executive Office of Weed and Seed) provides much of the research results reviewed in this report to local grantees designing specific programs. The possibility that local grantees, in turn, do not make the best use of the scientific evidence is a risk that results from the current statutory framework of local control. The Operation Weed and Seed plan to replicate the Rochester University early infancy home nurse visitation plan is also consistent with the strongest scientific evidence. The Title V OJJDP support of parent training strategies for troublesome pre-adolescents is also informed by scientific evidence showing success at reducing risk factors. These pre-adolescent efforts are all the more important in light of the strong scientific evidence of the failure of massive investment in adolescent prevention for high-risk youth in high crime areas (Harrell, 1996).

These discretionary programs, however, constitute a very small part of the total DOJ funding to prevent crime. There are no corresponding large scale formula grants making use of the strong scientific evidence now available. Nor is there a legislative basis for DOJ to pursue a "big science" program of developing a universal program of delinquency prevention support of all families, from early infancy onward. The scientifically recognized evidence on developmental crime prevention is now sufficiently strong for the Congress to consider a major effort to discover cost-effective means of family-based prevention.

Creating Knowledge. Congressional attention to family-based prevention of youth violence should also recognize the current weaknesses in creating knowledge. The current structure of funding does not generally provide sufficient funding or control for scientifically recognized methodologies in evaluating family-based prevention. Family programs are often included in a mix of other treatments, making it difficult to separate the effects of each ingredient in the mix. While a commitment to comprehensive programming makes theoretical sense in the long run, it is scientifically problematic in the short run. A more explicit commitment to knowledge building in family-based prevention would allow the accumulation of strong evidence about each specific approach, both separately and in combination with other elements. The funding and statutory mandate for such a big science effort has not been available to DOJ, and the creation of new knowledge with DOJ funds has thus been limited in this area.

The most practical question emerging from the available evidence is how to deliver a universal home visitation-preschool program similar to those already found effective in reducing delinquency. Issues of implementation, such as training and recruitment of effective staff, are as important to answering that question as issues of long-term impact. Collaboration between DOJ and other agencies appears essential, especially if one model to be evaluated would be an extension of the Head Start program. But rapid advances in this direction seem unlikely to occur absent a Congressional mandate.

It is important to note that both OJJDP and NIJ have made significant contributions to basic science of family factors in delinquency causation over the last decade. The OJJDP Program of Research on the Causes and Correlates of Delinquency birth cohort studies in Denver, Rochester and Pittsburgh, and the NIJ-MacArthur Foundation partnership Program on Human Development in Chicago Neighborhoods are both likely to inform the design of delinquency prevention strategies for years to come. The success of these research programs are an indicator of the substantial capacity for rigorous science within DOJ. Applying the same standards of scientific evidence to program evaluation, however, will require more powerful statutory tools for controlled field testing.

Preventing Family Violence

Using Knowledge. The challenge of this task is that there is little knowledge to use. The Congressional mandate for DOJ to prevent family violence, especially against women, is undermined by the paucity of scientific impact evaluations (Reiss and Roth, 1993; Crowell and Burgess, 1996). Most of what is being funded under current programs is therefore necessarily of unknown effectiveness. In recognition of this problem, the Assistant Attorney General for OJP has transferred modest portions of Crime Act funding to the National Institute of Justice for research and program evaluation. At the current rate of investment in program evaluation, however, it will be many years before the development of scientifically recognized impact evaluations to guide some $200 million in annual funding.

Creating Knowledge. The Congress has requested, and the Department of Justice has very recently supplied, a National Academy of Sciences report on a research agenda for preventing violence against women, including family violence (Crowell and Burgess, 1996). The Congress has not, however, yet had an opportunity to respond to that report, which seems likely to require authorizing DOJ to expend $50-60 million to carry out the agenda. Congressional action on that report's recommendations in FY 1998 would therefore be the most effective federal strategy for preventing family violence.

School-Based Prevention

The Congressional mandate to DOJ for school-based crime prevention is even less clear than the mandate for family-based prevention. Despite substantial scientific evidence of the effectiveness of some school-based programs, it remains an opportunity the Congress has lost for preventing crime. The Congressional mandate in this setting for DOJ is for less than $25 million per year, and supports some of the least effective programs available. This includes the earmarked $1.75 million Byrne Discretionary Program funding of Drug Abuse Resistance Education (DARE), as well as the estimated $20 million annual Byrne Formula Grants for the education purpose area (including DARE). These expenditures are small in comparison to over $500 million in annual school-based prevention funds appropriated through DOE and DHHS, and tiny compared to $2 billion invested in police strategies. Given the potential to integrate school-based prevention into a comprehensive strategy on youth violence, the Congress could profitably consider expanding the DOJ role in advancing this area.

Using Knowledge. Congress has not made effective use of available evidence from scientific evaluations. In choosing to support DARE, Congress has passed over other school-based prevention programs with scientific evidence of greater crime prevention effectiveness than DARE. The most widely used version of DARE has been found ineffective at preventing substance abuse. Other DOJ-supported school-based programs have been evaluated, but the evaluation methods were not scientifically adequate for drawing conclusions about program impact. The following programs are therefore of unknown effectiveness: Law-Related Education, Gang Resistance Education and Training, and Cities in Schools. In addition, Byrne funds may be supporting prevention programs that are reasonably certain to be ineffective: peer-group counseling, recreation, and programs based on fear arousal or moral appeal.

The more general conclusion about the legislation funding school-based prevention is that it takes a piecemeal approach. Scientific evidence shows this approach is less likely to be effective than comprehensive interventions in a school's capacity to teach behavioral norms and social competency skills. Stand-alone programs for preventing specific problems, from drugs to shootings, fare less well in the evaluation literature than programs changing the overall climate and order of the school. Many schools, particularly in disorganized urban areas, lack the organizational infrastructure even to provide adequate instruction in basic skills. These schools are staffed by demoralized adults whose failure to exercise control over rebellious students results in chaos, violence, and fear. Schools' failure as agents of social control perpetuates community disorganization, poverty, and crime. Research shows that building schools' organizational capacity to conduct their basic function in society is an effective crime prevention strategy. Programs for communicating and clarifying norms, such as antibullying campaigns, are also effective elements of delinquency prevention programs in schools, as are programs for teaching selfcontrol. Yet DOJ currently lacks Congressional appropriation for investing in these programs of proven effectiveness.

Creating Knowledge. DOJ funding for school-based prevention is least effective in creating new knowledge. Appropriation levels for evaluations have been too low for scientifically recognized methods to be employed. This common problem arises from a good faith effort to support evaluation, within constraints that defeat the purpose of trying to measure program impact. A prime example is the "catch-up" evaluation (post-test only, non-equivalent treatment and control group) design of the $16.2 million Gang Resistance Education and Training program. Funded at only $265,000 for eleven cities with less than a year to produce a Congressionally-mandated report, the evaluators (Esbensen and Osgood, 1996) were compelled to use such a weak evaluation design (Scientific Methods Score = 2) that it has almost no value in measuring the prevention effects of the program (Cook and Campbell, 1979).1

Similarly, the "national evaluation" of DARE commissioned by NIJ in the early 1990s was designed to be a secondary analysis of evaluations that were far too weak scientifically to merit such investment. A higher level of funding for a controlled, long-term field test still remains a high priority for evaluating DARE, especially in comparison to other programs shown to have larger preventive effects. The cost-effectiveness of such approaches is clear. DARE is a well-institutionalized program, nationally and internationally (Lindstrom, 1996). It seems likely to continue with a wide diversity of funding sources, regardless of whether Congress continues to fund it. But only Congress has the capacity to provide a sufficient investment of scientific resources to allow an objective test of the long-term effectiveness of the program at preventing substance abuse.

Evaluations of efforts to implement the elements of school life known to prevent crime also merit a high priority. Programs for increasing school order, and teaching social competency, face a wide range of obstacles in different school systems, especially in high crime neighborhoods. Learning how to overcome those obstacles could clear the way for more schools to prevent more crime. Many schools, particularly in disorganized urban areas, lack the organizational infrastructure even to provide adequate instruction in basic skills. These schools are staffed by demoralized adults whose failure to exercise control over rebellious students results in chaos, violence, and fear. Schools' failure as agents of social control perpetuates community disorganization, poverty, and crime. Research shows that building schools' organizational capacity to conduct their basic function in society is an effective crime prevention strategy. Programs for communicating and clarifying norms, such as antibullying campaigns, are also effective elements of delinquency prevention programs in schools, as are programs for teaching selfcontrol,

Labor Markets

Labor markets are the institutional setting some analysts stress the most in causing serious youth violence (Wilson, 1996). This setting also has the least DOJ history of a Congressional mandate. The available evidence suggests that including this setting in DOJ's mandate will increase the effectiveness of programs focused on high-crime areas. It also suggests that most other approaches to using labor markets to prevent crime have been ineffective.

Using Knowledge. The available evidence supports DOJ's limited involvement in labor market strategies. Employment programs aimed at older male ex-offenders no longer under court supervision appear effective at preventing crime. Such programs cannot prevent serious youth violence, but they are at least known to be a profitable investment for older males. More speculative, but theoretically supported, is the labor market component of Operation Weed and Seed, in which DOJ funding serves as seed money to attract other federal and private funds. This may enhance labor force participation rates in high crime areas, and provide more social capital for the community to encourage individuals to enhance their human capital job skills. This same rationale supports expanded DOJ participation in interagency Enterprise Zones and Empowerment Communities programs, especially to the extent that such programs serve high crime areas.

The evidence is also fairly strong that stand-alone investments in human capital--job skills--will not reduce crime. Rigorous scientific impact evaluations funded by the Department of Labor have generally found no effects of job training on crime rates for high-risk youth. The major exception is the moderately strong evaluation evidence on the Job Corps, a residential employment training program found to produce lower crime rates in program clients than similar persons not provided with Job Corps experiences. There is also mixed evidence on transitional aid programs for inmates leaving prison; while it is possible that DOJ Correctional Options funds spent in this manner are effective, the actual effects remain unknown.

Creating Knowledge. The most important mandate Congress could give DOJ on labor markets and crime risk factors is to add the issue of crime to the agenda of unemployment programs. A wide range of federal initiatives are attempting to solve problems of unemployment. Many of these are being evaluated. Few of them are providing adequate measurement of crime prevention effects. These programs include the 1996 major reform of the welfare system, Community Development Block Grants, reverse commuting, school-to-work programs, employment bonds and targeted wage subsidies. DOJ funded evaluations of these programs from a crime prevention perspective would give the Congress critically important knowledge about the effects of these programs on crime.

Place-Based Prevention

Basic research supported by NIJ for a quarter century has shown increasing evidence that crime is highly concentrated not just in neighborhoods, but at specific premises (see Chapter Two). Extensive investment by the British government in testing place-based crime prevention strategies, as well as increasing numbers of evaluations in the US, show consistent, but scientifically weak, evidence that such programs can be effective. Cameras, cash control, guards, fences, lighting, and other preventive devices are still of unknown effectiveness, but worthy of further evaluation. The promise of the consistently positive but scientifically weak available findings must be treated with appropriate caution, especially given the differences in gun crime between Britain and the US. The growing public and private investment in such strategies, however, suggests that a clearer Congressional mandate to DOJ in this area may be warranted. Recent findings showing the effectiveness of place management strategies in combating drug dealing and alcohol-related violence

provides further evidence of the need for more DOJ attention to crime prevention at places.

Using Knowledge. The federal funds provided to New York City police under the initial COPS programs made use of this DOJ-produced knowledge to focus their resources on high-crime locations. How much if any of New York's 50 percent reduction in homicide was caused by the federal funds cannot be determined with reasonable scientific certainty. But to the extent that place-focused crime strategies are supported by Byrne Formula, Weed and Seed, and Local Law Enforcement Block Grants, there is modest evidence that the funding has some potential to be effective. NIJ support of the Drug Market Analysis Program (DMAP), a computerized crime mapping strategy, has been put to widespread use as police agencies around the country employ similar software to identify and "problem-solve" high crime places. What is still unknown is the extent to which any particular strategy of problem-solving is effective in any particular kind of place with any particular kind of crime problem.

Building Knowledge. With sufficient resources and a clear mandate, a central research arm of NIJ could build a body of scientifically rigorous impact evaluations of place-based prevention strategies. Systematic observations of high crime locations, large samples of similar places with similar problems with half randomly assigned the tested prevention systems, and other scientific techniques are necessary for creating reasonable certainty about the effectiveness of prevention at places. DOJ has been able to produce some knowledge of this kind about police efforts against places. More of it is needed about efforts by owners, managers, residents, liquor licensing boards and other branches of government to develop place-based prevention strategies.

Policing for Prevention

Current appropriations invest far more federal funds in the police than in any other crime prevention institution. The estimated $2 billion annually for police is more than half of all federal assistance for local crime prevention. While there is growing evidence that police can be effective in preventing crime, there are still many questions about how to use the federal funding most effectively. Depending upon how each police agency elects to use its federal funding, the taxpayer investment in that strategy may be more or less effective at preventing crime.

Using Knowledge. There is reasonable scientific certainty that the following police practices prevent crime: extra police focused on high crime "hot spots," police units focused on serious repeat offenders, proactive enforcement of drunk driving laws, and arrests of employed suspects for misdemeanor domestic violence. There is also reasonable certainty that the following practices are ineffective: increased random patrols of entire beats, more rapid response time to 911 calls, neighborhood watch, arrests of some juveniles for minor offenses, arrests of unemployed suspects for domestic assault, arrest crackdowns at drug markets, and community policing programs with no clear focus on crime risk factors. There is no certainty, however, that the current federal legislation produces more police effort with the effective practices than with the practices known to be ineffective.

The accumulated body of police practice evaluations, largely funded by NIJ, provides one of the best examples of using science to identify programs of proven effectiveness. Under the principles of local control, however, there is no requirement that federal funds be used for practices proven effective, except under the Byrne Formula Grants. Even there, lack of statutory clarity about the definition of proven effectiveness makes it little different from programs without that legislative requirement.2 A key question for Congress is therefore whether to use the available scientific knowledge to limit the eligible purpose areas for federal funding of local police. This would not be possible, however, at the level of generality of the current 33 purpose areas under the Byrne and LLEBG programs. Police overtime, for example, cannot be evaluated as a crime prevention practice, but various uses of police overtime can be. Thus within each purpose area it could be possible to distinguish proven and unproven areas of effective practice.

The most basic issue of DOJ funding effectiveness at addressing crime risk factors is the formula basis for allocating police assistance, noted above. Concentrating police funding in the census tracts, and not just the cities, with the highest rates of serious crime and youth violence, would appear to be more effective than present formulas based on population size. Promising evidence on reductions in gun crime from such concentrations lends further support to the value of testing the hypothesis that putting police where crime is most serious would be the most cost-effective means of combating the national epidemic of violence.

Creating Knowledge. DOJ has been generally effective in creating new knowledge about the impact of police practices on crime. This reflects a substantial commitment of resources to several large and costly projects over the past two decades. The results are encouraging evidence that DOJ has a substantial capacity for big science in this area, including controlled evaluations using city-wide police practices (such as traffic enforcement or gun seizures) as the unit of analysis. This capacity could be put to even better use if NIJ had control over some portion of federal funding for local police overtime or extra hiring to allow it to structure controlled tests of patrol dosage and tactics in high-crime areas. Scientific evidence produced by this kind of evaluation design would directly address the key Congressional issue of how best to allocate the federal funds for police. While the COPS office has transferred 1 percent of program funds to NIJ for evaluation purposes in FY 1996, the scientific strength of those evaluations will be limited in the absence of statutory authority to allocate program funds for evaluation purposes.

Additional appropriations are needed for NIJ to replicate the growing number of strategies rated as "promising:" police traffic enforcement patrols to detect illegally carried handguns, "Chicago-style" community policing, community policing efforts for increasing police legitimacy in minority poverty areas, "zero-tolerance" enforcement of minor violations, and adding extra police to cities or areas, regardless of what they do. Untested but theoretically promising strategies such as community-based restorative justice for minor juvenile offenses are also high priorities for creating new knowledge about policing. Several police strategies for preventing serious domestic violence are ready to be evaluated if sufficient funds are available.

Criminal Justice and Crime Prevention

The Congressional mandate for DOJ to fund prisons and other criminal sanctions is clear. What remains unclear is how to spend that money most effectively. Based on the review of the available scientific evidence, the effectiveness of current legislation in funding criminal justice to prevent crime is mixed, but more remains unknown than known.

Using Knowledge. There is reasonable scientific certainty that federal funding is effective to the extent that it supports three broad strategies: prison-based therapeutic community treatment of drug-involved offenders, incapacitation of high-rate serious repeat offenders, and structured rehabilitation programs focused on individual risk factors. These strategies can be supported under DOJ funding for Drug Courts, Residential Substance Abuse Treatment for State Prison Inmates, OJJDP's Intensive Community-Based Aftercare, Byrne Formula Grants under the drug treatment purpose area, and the Violent Offender Incarceration Grant Program (prison construction). Whether the funding is actually used in the ways the scientific evidence finds effective, however, is up to the local control of the grantees.

One of the major concerns about prison construction, for example, is whether it will actually produce incapacitation of the most serious offenders. If increased capacity is eventually used to increase the incarceration of non-violent offenders, there may at least be a pattern of diminishing returns, and at worst a very expensive investment in locking up low-risk offenders when the money could be spent more effectively on other forms of prevention.

There is also reasonable scientific certainty that the following programs are ineffective in preventing crime: correctional boot camps using military basic training models, other "harshness" programs like "Scared Straight" and shock probation, and community-based alternative sanctions lacking treatment programs and services. The latter include Intensive Supervised Probation or Parole (ISP), home confinement, urine testing, community residential programs, and juvenile wilderness programs. (Urine testing combined with drug treatment, however, enjoys scientific evidence that is promising.) These ineffective programs can be supported by DOJ funding under the Byrne Formula Grants for the alternative sanctions and drug testing purpose areas Program, and by the prison construction grants.

Creating Knowledge. Scientific conclusions can be drawn about only a small portion of all correctional strategies and federal funding. The effects of sanctioning on repeat offending are generally found to vary by type of offense, type and dosage of sanction, type of offender, and the characteristics of the communities to which incarcerated offenders return upon release from prison. The identification and codification of the full range of these effects is a massive task that has barely begun, but can be accomplished by a systematic long-term investment in controlled testing of sanctions.

The biggest obstacle to such a program of systematic impact evaluations may not be federal funding, but State and local cooperation with controlled testing. The widespread resistance to controlled testing of early boot camp programs is indicative of the difficulty of creating such partnerships. The major advantage DOJ could bring to creating such knowledge is conditional program funding, which would only be made available to grantees if they cooperated with a controlled field test design. Using DOJ local assistance funding in this way would be particularly helpful in producing controlled tests of programs found promising in Chapter Nine: drug courts, day fines, juvenile aftercare, and drug treatment combined with urine testing. Current evaluations of these programs without scientific controls will not be useful for creating knowledge about their effectiveness.


In the three decades of DOJ funding for local crime prevention assistance, four prior reports have made recommendations for improving program evaluation. Each report was followed by Congressional or administrative action to implement the recommendations, at least in part. Yet as the evidence reviewed in this report suggests, the Congressional mandate to evaluate has yet to be satisfactorily accomplished. This section briefly reviews the prior reports, as well as the conditions needed for effective program evaluation. The report then closes with recommendations for a statutory plan that would create those conditions, and improve the effectiveness of federal funding through stronger evidence on program effectiveness. The prime example of this approach is a comprehensive program for cooling off "hot spot" neighborhoods.

Previous Reports on DOJ Crime Prevention Evaluations

From the earliest days of the Law Enforcement Assistance Administration (LEAA), the issue of evaluation has been paramount. Lacking a large body of scientific knowledge on the effectiveness of crime prevention programs, LEAA was forced to invent new ideas under conditions of high public visibility and enormous Presidential pressure to reduce crime. The results of those programs were crucial to the public debate. By 1972, Congress amended the Safe Streets Act to require LEAA to conduct evaluations of its action grants (Feeley and Sarat, 1980: 113). It also increased the amount of money the States could spend evaluating their LEAA-funded grants. This model of giving primary responsibility for evaluating each and every grant to the State Planning Agencies (SPAs) administering the funds remains largely unchanged today.

The State-level evaluation tasks were supplemented a year later with national evaluation tasks to be performed within DOJ. In 1973 a Columbia University symposium on LEAA concluded the worst flaw in the program was the lack of useful evaluations (Rubinstein, 1977: 148). The Congress responded by amending the Crime Control Act that year to require that the National Institute of Law Enforcement and Criminal Justice (the predecessor to NIJ) "undertake, where possible, to evaluate the various programs and projects carried out under this title." A year later, two GAO reports criticized the Institute for its continued inability to conduct outcome evaluations of the more than 30,000 LEAA-funded grants awarded in the agency's first five years. GAO called the available evaluations inconsistent and relatively useless (Rubinstein, 1977: 148).

At that point the LEAA administrator released a new report on program evaluation policy.3 The report assigned to the National Institute the Congressional mandate to evaluate, focused in its new Office of Evaluation (OE). The Report gave that office three goals: obtain and disseminate information on the cost and effectiveness of crime prevention, ensure that the information gets used in program planning, and develop a capacity for evaluation in state and local units of the criminal justice system. As a National Academy of Sciences report later concluded about this plan, it was an enormous challenge (Rubinstein, 1977: 149):

Perhaps the greatest challenge was to develop a capacity to carry out evaluations in operational agencies at a time when the capacity did not even exist in most research universities. The Institute was unable to give $2 million to the LEAA State and Regional Planning Agencies to develop evaluation units; only 12 of the 500 eligible units even applied for the money. The program succeeded in at least spreading the definition of evaluation as a scientific assessment of effects caused by the program (Feeley and Sarat, 1980: 114), but few of the evaluations done by State agencies to comply with federal rules ever came near that definition.

By 1977, the National Academy of Sciences evaluation of the National Institute of Law Enforcement and Criminal Justice (White and Krislov, 1977) recommended that the Institute try to get away from program evaluation and focus on basic research. Describing the quality of Institute research at that point as "parochial and, overall, mediocre," the report attacked the Institute's emphasis on "immediate solution applied research" (White and Krislov, 1977: 94-95). This report moved the Institute into peer-reviewed competitions for awarding research grants, and established research programs on basic questions like deterrence. Many grants funded under these "basic" research programs, such as the domestic violence arrest experiments, were later treated as "evaluations" (Reiss and Roth, 1993: chapter 7)--thereby suggesting the limitations of the distinction. The 1977 report recommended that evaluation functions be integrated into the broader research programs of the Institute.

To a large extent, NIJ has followed the blueprint of the 1977 report for two decades. Congressional decisions to cap available funding levels and maintain structural separation of NIJ from the DOJ program funding bureaus have helped to constrain NIJ in that direction. That path was not changed despite a bipartisan 1981 Report of the Attorney General's Task Force on Violent Crime that recommended a very different role for evaluation as the centerpiece of federal crime control policy. On August 17, 1981, President Ronald Reagan's Attorney General, William French Smith, received the report of his bipartisan Task Force on Violent Crime, co-chaired by former Attorney General Griffin Bell and Illinois Governor James R. Thompson. The report said this about the federal role in evaluating crime prevention (p. 73):

Put another way, the 1981 Task Force recommended that the federal role should be to advise local governments about proven programs of crime prevention, based on a scientific process of basic research, program development, and--most important--rigorous independent field tests of demonstration programs in several different jurisdictions. Based on our review of the scientific evidence underlying crime prevention programs in 1996, we can only conclude that this recommendation has been ignored. We are unable to find a single program for which the federal government has played the role recommended in 1981. There are numerous programs that have been developed based upon research and implemented in several jurisdictions. What has been lacking is the statutory structure and resources needed to carry out the scientifically rigorous evaluations the Attorney General's Task Force recommended.

In 1988, the passage of the Anti-Drug Abuse Act saw yet another plan for program evaluation enacted into law. The Act required that States conduct evaluations of each grant, but did not require that a certain percentage of Byrne Formula funds be used for that purpose. It also called upon NIJ and BJA to sponsor "a reasonable number of comprehensive evaluations of programs" funded under both formula and discretionary grants.4 The Congress did not however, appropriate any funding for NIJ or BJA to evaluate what is now $475 million in annual formula grants, forcing them to draw on their general appropriations for that purpose. Of the first 5,000 Byrne Grants, NIJ and BJA were able to sponsor evaluations of 150. Once again, the federal agencies attempted to compensate for limited evaluation funding by developing an evaluation capacity in the State Planning Agencies. Just as it did in 1974, NIJ invited States to seek evaluation funding. But in 1990 when the proposals were submitted, as a recent NIJ report on the Byrne Program reported, most were methodologically weak, and as a result few were funded. Not surprisingly, this suggested that many State agencies did not have the research staff necessary to conduct evaluations (Dunworth, et al, 1997: 8).

Once again, as the Institute did in 1974, NIJ and BJA have undertaken "technical assistance programs to expand State evaluation capabilities" (Dunworth, et al, 1997: 8). Also, an annual NIJ-BJA conference on program evaluation funded by BJA brings State Planning officials to Washington to hear the most recent evaluation findings. But these steps can do little to deal with the four underlying structural factors limiting DOJ's capacity to generate satisfactory evaluations of Byrne Grant programs (Dunworth et al, 1997: 8-9): 1) state legislators often resist spending Byrne funds on evaluations, believing the money should be spent for program purposes; 2) Congress has not given NIJ or BJA any funding for Byrne evaluations; 3) insufficient evaluator control over program conditions often compromises the scientific integrity of evaluation results; 4) information on evaluation results has not been accessible to those who need it.

The Crime Act of 1994 began to address at least one of those obstacles, lack of appropriated funding. The Act created the first set-aside directing evaluation of program funding, authorizing the Attorney General to use up to 3 percent of COPS program funds in any given year for studies or furtherance" of the purposes of the COPS program.5 The law also contained authority for evaluations to be funded from the Violence Against Women Act, the Drug Courts program, and the Corrections Title. But these steps do not address the structural issue of control over program funds that is central to the scientific standards of an evaluation. Thus despite repeated reports and legislation, the nation still lacks a satisfactory solution to the problem of achieving the Congressional mandate to evaluate--using scientific standards. The next section describes those standards and the statutory elements needed to create them.

Scientific Standards of Program Evaluation

The Omnibus Crime Control and Safe Streets Act of 1968 defines an "evaluation" as follows:

By this definition, an evaluation cannot be only a description of the implementation process, or "monitoring" or "auditing" the expenditure of the funds. The terms "impact" and "accomplishing" require claims about cause and effect. The scientific standards for inferring causation have been clearly established in the literature on research design and methods (Cook and Campbell, 1979; Federal Judicial Center, 1981). They include, at minimum, the following:

1) the ability to measure the dosage, timing and content of the project or program,

2) the ability to gather baseline data prior to the start of the project, if necessary, and

3) the ability to gather comparable data from both the program group and appropriate comparison groups where the program is not operating.

These elements of science are not unrealistic for crime prevention program evaluations. They do not rise to the "gold standard" of level 5 scientific methods scores used by the Food and Drug Administration in testing new drugs. Rather, these elements constitute the requirements for a level 3 evaluation, the agreement of two of which generally satisfied this report's criteria for programs that "work"--programs of proven effectiveness.7 Crime prevention program evaluation would be better served by raising that scientific standard to level 4, with this additional element:

4) the ability to eliminate or control for most known rival hypotheses that could account for the same results other than the program or project being evaluated

It would even be better to adopt the gold standard, in which the evaluator is given this capacity:

5) the ability to select program and comparison groups in advance of the program by use of equal probability formulas.

But the basic task of the Congress in fulfilling the mandate to evaluate is to insure that the first three elements can be achieved with reasonable certainty whenever DOJ funds a program evaluation. That is not the case at present. The structural separation of program administration and evaluation, combined with local control over the expenditure of grant funds, makes the first three elements extraordinarily difficult to achieve. Thus despite three decades of defining evaluation in these scientific terms, the Safe Streets Act has never included a realistic statutory plan for achieving adequate impact evaluations. The next section offers such a plan, based upon "rigorous and scientifically recognized standards and methodologies."

A Statutory Evaluation Plan

Three principles for evaluating crime prevention programs emerge from the evidence reviewed for this report. One principle is that not every grant needs to be evaluated. A second is that scarce evaluation funds should generally be conserved for strong scientific evidence of program impact. The third principle is that every impact evaluation should be done at a scientific methods score of no less than level 3, and where possible at level 4 or 5.

Not every grant requires an evaluation. Since the early days of LEAA, the use of the term evaluation has been confused by the requirement that every grant be evaluated. Absent the resources and the skill needed for achieving the statutory definition of an evaluation as an impact assessment, the requirement that everything be evaluated has resulted in almost nothing being evaluated. There are enough similarities in program content across sites so that the Attorney General's Task Force on Violent Crime recommendation is a reasonable one. Testing a program in several different sites under different conditions provides an adequate basis for recommending that the program can be used generally, if the results are consistently favorable. Spending large amounts of money for strong evaluations in a few sites is far more cost-effective than spending little amounts of money for weak evaluations in thousands of sites.

Evaluation Funds Should be Conserved for Impact Assessments. Inadequate funding levels have forced DOJ to choose between many descriptive evaluations or a few impact evaluations. In general, the choice has been to appropriate $200,000-$300,000 for a national evaluation of each major program, regardless of how much or little can be learned for that amount. Very little can be learned about the impact of JUMP (Juvenile Mentoring Program) for the less than $200,000 that was available for its evaluation. Even less can be learned about the HIDTA (High Intensity Drug Trafficking Area) program for the $200,000 budgeted for a national evaluation of a $100 million program operating in six sites. That funding level limits an evaluation to only the most superficial description of what the program is doing, and cannot say anything about its impact. While funding for basic research must be preserved, and while methods for tracking field innovations should be included, studies of specific federal funding programs do not provide Congress with the information it requires unless there is enough funding for a scientifically valid impact assessment. Such studies routinely cost $15 million or more in other agencies, but there is no precedent for such "big science" at DOJ.8 Since there was no precedent for a $30 Billion Crime Bill either, there is clearly room for adopting a new approach to evaluation.

Impact Evaluations Should Be Conducted at a Level Three Scientific Methods Score or Higher. If the Congress needs to know the effectiveness of a program, it needs to know that answer to a reasonable degree of scientific certainty. Just as the U.S. Supreme Court has asked Federal Judges to be the gatekeepers of valid science to be placed in the hands of a jury,9 Congress can ask that independent peer review panels serve the same function for Congressional evidence. The panels can be asked to certify that impact evaluations recommended for funding by DOJ are at least designed with a Scientific Methods Score of 3 or more. The criteria employed by this report for assigning that score are sufficiently broad as to correspond to the "flexible inquiry" the court requested of the federal bench into the validity of scientific evidence. But they are also sufficiently clear as to change the current pattern of funding descriptive studies and calling them "evaluations." Imposing this standard will force the issue back to its proper decision point: the Congress. The issue of evaluation funding levels will become much clearer in light of a bright line between impact evaluations and all other ways of studying federal programs.

All three principles should be firmly established by their enactment in a legislative plan for program evaluation. But in order to make the third principle possible, the statutory plan must restructure the entire evaluation process as a central purpose of the federal role in crime prevention. That restructuring can meet the scientific requirements for achieving that goal with a ten percent set aside of program funding for evaluated programs, a ten percent set-aside for program evaluations, and a structural location of both set-asides in a central research office within OJP. This plan closely tracks the 1981 recommendation of the Attorney General's Task Force on Violent Crime, as well as modern business practices in large corporations.

The Business Precedent: Beta Sites. The Attorney General's Task Force recommendation reflects a pattern of new product development widely used by large American corporations, called "beta sites." This pattern employs test sites as a means to determine the potential success of a new product, and possible conditions in local markets that could predictably affect sales. For example, in recent years the McDonalds' restaurant chain developed a new low-fat hamburger, then tested its customer appeal in a limited number of cities. These tests apparently found some demographic correlates for the success of the product. The product was subsequently offered in many, but not all, local markets nationwide, with selections made on the basis of local market demographic factors revealed by the beta site testing. Similar variations in the effectiveness of crime prevention programs could be revealed by the use of such a strategy, but only if the program "beta sites" are selected for the explicit purpose of conducting a test, rather than just supporting local operations.

This model can be achieved by Congressional enactment of the following recommendations:

1. Set aside ten percent of all DOJ funding of local assistance for crime prevention (as defined in this report) for operational program funds to be controlled by a central research office within OJP. This recommendation solves the inadequate evaluator control over program conditions for inferring cause and effect. A wide variety of strong scientific research designs become possible when program funding is available as an incentive for local agency evaluation partnerships. Police overtime, prison treatment programs, school-based prevention strategies could all be implemented in ways that may be less than optimally convenient for the local operational units, but which greatly increase the strength of the scientific evidence.

2. Authorize the research office to distribute the ten percent "evaluated program" funds on the sole criteria of producing rigorous scientific impact evaluations, the results of which can be generalized to other locations nationwide. Allocating these funds for research purposes simply adds to the total funding for which any local jurisdiction is eligible. Thus the "evaluated program" funding becomes an additional incentive to cooperate with the research plan on a totally voluntary basis.

3. Set aside an additional ten percent of all DOJ local assistance appropriations for crime prevention as defined in this report to fund the scientific evaluation costs. This recommendation makes clear the true expense of using rigorous scientific methods to evaluate program impact. The imperfections of most indicators of crime and justice require multiple measurement, already reflected in the statutory language authorizing COPS program evaluations. Victimization interviews, offender self-reported offending, systematic observation of high crime locations, observations of citizen-police interaction, and other methods can all cost as much or more than the program being evaluated. The costs can also be amortized over the many years of use to which strong scientific evidence can be put in guiding effective crime prevention.

A prime example of the value of this approach is implied by the central hypothesis that emerges from this report: that serious youth crime in America can be reduced most substantially by a simultaneous investment in all seven institutional settings for crime prevention, focused on the small number of neighborhoods in the nation where serious youth violence is concentrated. The results of the Children at Risk evaluation (Harrell, 1996) suggests that programs focused only on individual adolescents at risk may be unable to succeed because the neighborhood around them does not change. It may be that the best results would come from an expanded version of Weed and Seed, in which a systematic effort to build community social capital includes

o helping families to establish clear and consistent discipline and emotional bonding, using home visits and preschool involvement from early infancy

o helping schools to establish a capacity for self-regulation of student conduct with clear norms and expectations, as well as adequate physical security

o helping labor markets to raise labor force participation rates in the neighborhood from 20% to 80%,

o using physical and other place-based prevention to reduce opportunities for crime,

o using massive increases in neighborhood police patrols (and respectful interactions with youth) to get guns off the streets and maintain high standards of civil conduct in public places, and

o using courts and corrections to provide highest priorities to cases arising from these neighborhoods for effective treatment and control of convicted offenders.

This "hot spots" hypothesis has substantial support from theoretical models and the indirect evidence supporting those models. It can only be tested in practice, however, by the kind of statutory evaluation plan recommended above. Only a large number of neighborhoods with and without such a program can provide an adequate sample for using scientifically recognized standards and methodologies. And if only such a program can make a long-term difference in serious youth violence, the Congress has a clear opportunity to consider. For if the hypothesis is correct, a comprehensive program to cool off the nation's hot spots of youth violence could not only lower the national crime rate. It may also create a tipping point against an epidemic of youth violence, and make the rest of the nation safer as well.


1Here again, legislative requirements cause poor science. The Treasury Department bill funding GREAT contained the following clause: "The Committee further instructs ATF to provide semi-annual reports evaluating the programs and identifying the affect [sic] GREAT has had on deterring gang violence." This requirement included no appropriation for evaluations. But even with massive appropriations for evaluation, the concept of semi-annual evaluations of program effects has no precedent under scientifically recognized standards and methodologies.

2The 1988 Anti-Drug Abuse Act's statutory definition of "proven effectiveness" is as follows: "a program, project, approach, or practice has been shown by analysis of performance and results to make a significant contribution to the accomplishment of the objectives for which it was undertaken.." 42 U.S.C. 3782 Sec. 801 (b) (19).

3U.S. Department of Justice (1974), LEAA Evaluation Policy Task Force, LEAA EVALUATION POLICY TASK FORCE REPORT. Washington, D.C.: USDOJ.

442 U.S.C. S 3766 (a)(2); 3782 (b).

542 U.S.C. Section 3793 (a)(11)(B).

642 U.S.C. Section 3791 (10)

7It was also necessary that the preponderance of the other evidence be in accord with the results of the two level 3 or higher evaluations.

8For example, several of the Department of Labor job training experiments have cost in excess of $15 million.

9Daubert v. Merrell Dow, 113 S. Ct. 2786 125 L. Ed. 2d 469 (1993).


Cook, Thomas D. and Donald T. Campbell

1979 Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago: Rand-McNally.

Crowell, Nancy A. and Ann W. Burgess, eds.

1996 Understanding Violence Against Women. Washington, DC: National Academy of Sciences.

Dunworth, Terence, Peter Haynes, and Aaron J. Saiger

1997 National Assessment of the Byrne Formula Grant Program. National Institute of Justice Research in Brief. Washington, DC: US Department of Justice.

Esbensen, Finn-Aage and Wayne Osgood

1996 GREAT Program Effectiveness: Results from the 1995 Cross-Sectional Survey of Eighth Grade Students. Unpublished Technical Report Available from the Author.

Federal Judicial Center

1981 Experimentation in the Law. Washington, DC: Federal Judicial Center.

Feeley, Malcolm and Austin Sarat

1980 The Policy Dilemma: Federal Crime Policy and the Law Enforcement Assistance Administration, 1968-78. Minneapolis, MN: University of Minnesota Press.

Greenwood, Peter, K.E. Model, C.P. Rydell & J. Chiesa.

1996 Diverting Children from a Life of Crime: Measuring Costs and Benefits. Santa Monica, CA: RAND. et al

Harrell, Adele

1996 Evaluation of the Children At Risk Program: Preliminary Findings. Paper Presented to the National Evaluation Conference, National Institute of Justice, Washington, DC.

Lindstrom, Peter

1996 Paper Presented to the Conference on Problem-Oriented Policing as Crime Prevention. Stockholm: Swedish National Police College.

Reiss, Albert J. Jr., and Jeffrey A. Roth, eds.

1993 Understanding and Preventing Violence. Washington, DC: National Academy of Sciences.

Rubinstein, Juanita L.

1977a Case Study: Office of Evaluation. In Susan O. White and Samuel Krislov, eds. Understanding Crime: An Evaluation of the National Institute of Law Enforcement and Criminal Justice. Washington, DC: National Academy of Sciences.

1977b Case Study: Impact Cities. In Susan O. White and Samuel Krislov, eds. Understanding Crime: An Evaluation of the National Institute of Law Enforcement and Criminal Justice. Washington, DC: National Academy of Sciences.

White, Susan O. and Samuel Krislov

1977 Understanding Crime: An Evaluation of the National Institute of Law Enforcement and Criminal Justice. Washington, DC: National Academy of Sciences.

Wilson, William Julius

1996 When Work Disappears: The World of the New Urban Poor. NY Alfred A. Knopf.