II. Evaluating the Impacts and Effectiveness of Drug Courts

Participants at the March 1997 focus group meeting, reflecting on the state of evaluation research on drug courts, generally believed that the development and implementation of drug courts had far outpaced research on their effectiveness. The first drug court programs, developed in the late 1980s and early 1990s, were truly "grassroots" efforts, born out of frustration with "revolving door" justice and a sense that there had to be a better way to deal with the crush of cases resulting primarily from a crack cocaine epidemic and stepped-up law enforcement efforts that focused heavily on arrests for possession of illegal drugs. These programs, responding to a perceived crisis in the courts, operated on shoestring budgets with little or no capacity to document their effectiveness with independent evaluations. Yet the results were readily apparent to observers, and other jurisdictions -- responding to the same sorts of pressures -- began developing their own drug courts. From only two drug courts operating in 1990, the number grew to ten in 1992, 41 by 1994, and more than 160 at the start of 1997. As Professor John Goldkamp observed at one of the focus group meetings, the diffusion of ideas about drug courts has spread far faster than the empirical knowledge of what is happening in the field.

So great was the need that many jurisdictions embraced drug courts before their effectiveness could be demonstrated empirically. Yet some drug court practitioners did seek to quantify what they were seeing in the courtroom and in reports of drug test results, and began collecting data on a variety of outcome measures. And, as the concept began to take hold, a few (very few) federally funded evaluation studies were undertaken. By early 1997, approximately 20 evaluation studies of drug courts -- most of them undertaken locally, often with minimal resources -- had been completed. Reviewing this body of research, the General Accounting Office (GAO) found that the drug court programs studied in these 20 evaluations varied widely with respect to eligibility requirements, percentage of eligible persons who completed the programs, length and nature of the treatment provided, frequency of drug testing and court appearances, and other variables. Further, the 20 studies themselves varied widely in their objectives, their scope, and the methodologies they employed. Only 6 of the 20 studies involved both a comparison group and an assessment of how participant and non- participant arrest rates compared after program completion. Moreover, many of the drug court programs covered in the 20 studies were still relatively new at the time the evaluations were conducted. In light of these limitations, the GAO report, published in July 1997, concluded that while existing evaluations provided some information about the courts studied, they did not provide a sufficient basis for drawing firm conclusions about drug court impact.

The GAO study was particularly concerned with what existing evaluations revealed about six specific issues: (1) criminal profile of drug court participants compared with that of similar offenders processed through the traditional adjudication system; (2) completion rates of drug court participants; (3) differences in characteristics between drug court completers and noncompleters; (4) sanctions imposed on persons who failed to complete drug court programs compared with those similarly situated but processed through the traditional criminal justice system; (5) drug use and criminal recidivism rates of program and nonprogram participants; and (6) costs and benefits of drug courts to the criminal justice system. Participants at the two focus group meetings were concerned with most of these issues, too. The types of information that these participants identified as essential for routine data collection and reporting by all drug courts (see pages 4-5 and 18-19) should provide a foundation for future evaluations that will address the issues of interest to drug court practitioners and those targeted by GAO.

Participants at the focus group meetings were also concerned, however, with a range of evaluation questions -- and with issues about the conduct and funding of evaluations -- that were not central to the GAO study. They addressed the following four main sets of questions:

1) What is the current state of knowledge about the impact and effectiveness of drug courts? What do we think we know, and on what basis?

2) Who are the primary audiences for evaluations of drug courts? What are likely to be the principal concerns of these stakeholders with respect to evaluation?

3) What should be the primary objectives of local-level evaluation research on drug courts? What should be the main objectives of nationally funded evaluation research?

4) What are the principal obstacles to sound evaluation research on drug courts, and how can these obstacles be overcome?

A. The Current State of Knowledge About Drug Court Impact and Effectiveness

The Drug Court Clearinghouse and Technical Assistance Project at American University, which is funded by DCPO, periodically publishes information about the activities and performance of drug courts throughout the Nation. The Clearinghouse's 1997 Summary Assessment of the Drug Court Experience, based on information compiled from surveys, observations of AU staff during the course of providing technical assistance to drug court programs, and the experiences reported by drug court officials, provides an overview of practitioners' accumulated knowledge and beliefs about drug courts' impact and effectiveness. It points to a number of areas in which the drug court approach seems to be more effective than traditional case processing, including:

The perceptions that experienced practitioners have about the effectiveness of the drug court approach are important components of an overall evaluation of their impact. However, it is important to supplement these perceptions and the drug courts' reports of their activities with other data -- particularly data that enable comparison of the performance and outcomes of drug court participants with those of nonparticipants who have similar relevant characteristics.

B. Audiences for Drug Court Evaluations

Participants at the focus group meeting identified a number of different stakeholders who are likely to be interested in the operations and impact of a drug court, and who are thus potential consumers of evaluation research. Some, but far from all, of these stakeholders may also be funders of the drug court program or of an evaluation of it. At the local level, they include:

While virtually all of these stakeholders will be interested in an evaluation that focuses on certain kinds of hoped-for outcomes (for example, reduced recidivism), some will be extremely interested in potential evaluation questions that are of little interest to others. Court administrators, for example, are likely to be very interested in the impact of drug courts on court caseloads and the use of courtroom time for drug-related cases, while public health officials will probably be much more interested in the impact of drug courts on reducing parti-cipants' substance abuse and improving their overall health. Given the wide range of potential focal points for drug court evaluations, as well as limited resources and serious methodological difficulties in addressing some types of questions, choices will have to be made with respect to evaluation priorities at different levels of government and for different types of drug court programs.

C. Evaluation Objectives and Priorities

1. Local-level Evaluations. At the local level, the fundamental challenge for an evaluator is to determine the effects of the specific drug court in terms of the goals that have been set and the outcome measures agreed upon by the drug court leaders and the local-level policy group responsible for drug court operations. Often this group includes representatives of current and prospective sources of funding for the drug court. If not, it should. One of the primary purposes of an evaluation is to provide a sound basis for making decisions concerning future financial support for a program.

Simply determining the effects is only part of the challenge, however. Optimally (and assuming sufficient resources and access to relevant data about both program participants and comparison group members), evaluators may also be able to explain the reasons for particular effects, taking account of the full range of operational components of the drug court. That is, what is it about a particular drug court that makes it "work" or fail to work? Given the large number of operational components in a drug court, answering this question will be a formidable task. For example, which of the following factors, or what combination of factors, explains a favorable outcome such as significantly reduced recidivism or reduced substance abuse?

Other potential explanatory factors could be listed, but the point should be clear: There are a great many reasons why a particular drug court program may work or not work, and explaining why is difficult and methodologically challenging. From a policy and funding standpoint, however, answering the "why" question is important. Different approaches to drug court structure and operations can result in significantly different outcomes and are also likely to require different types and levels of resources.

Given the limited resources available for local-level evaluations, it is probably asking too much to expect these evaluators to explain the outcomes they find in terms of all of the potentially relevant variables. However, it is not asking too much to expect that the evaluators describe the programs in considerable detail, so that a base of knowledge can be developed about the components of drug court programs that succeed (or fail) on key outcome measures.

As noted above, different stakeholders are likely to have different priorities with respect to evaluation topics. However, there is a core group of potential "client outcomes" of drug court programs that participants in the focus group meetings identified as especially important to assess through evaluation -- preferably by comparing the performance of drug court participants with a group of nonparticipants who are similar on relevant dimensions, such as current charges, prior criminal histories, nature and severity of substance abuse problems, age, race/ethnicity, and sex. They are:

1) Reductions in recidivism

2) Reductions in substance abuse

3) Improvements in participants' life circumstances, including:

In addition to outcomes for drug court participants compared with similarly situated nonparticipants, learning about potential effects of the drug court on the operation of the criminal justice system and on the communities served by the drug court is a high priority. Here the comparisons are not of groups of participants, but rather between the drug court and other systems or between the drug court and the pre-drug court operation of the system. The focus group noted the following potential outcome variables with respect to the "system effects" of drug courts in comparison with more traditional criminal justice processes:

The focus groups discussed the development of a framework for analyzing drug court costs and benefits. They recognized that a great many data elements are potentially relevant to a cost-benefit analysis. There was general consensus that, important as this type of analysis is, it is extremely difficult to carry out successfully because of the difficulties in quantifying both costs and cost avoidance.

2. National-level Evaluations. National-level evaluation efforts can and should have a different scope and focus from local-level evaluations. Focus group members welcomed the news that the National Institute of Justice (NIJ), in collaboration with the DCPO, will arrange for evaluations of established drug courts. National-level evaluations will be sufficiently funded to address a wide range of questions and permit the use of appropriate methodologies. The NIJ studies will use carefully selected comparison groups and will seek to obtain postprogram data on substance abuse and life circumstances as well as on new arrests and convictions.

As drug courts around the Nation develop the kinds of databases envisioned by the focus group (see pages 3-4), it should be possible for federal agencies to fund more single-court studies that undertake a comprehensive evaluation of the drug court program under exami-nation. And, as knowledge is developed about the components of specific drug courts, it should be possible to undertake meaningful cross-jurisdiction studies that permit the assessment of the relative effectiveness and impacts of different approaches to implementing drug courts. The objective of such studies is not to compare drug courts with each other but, rather, to learn what approaches are most effective with specific target populations.

It is important for federally funded evaluations to look beyond data on recidivism and substance abuse and to help develop an understanding of how drug courts are affecting the attitudes and practices of justice system and treatment practitioners. There is a strong sense, particularly among drug court judges, that many of the principles and techniques used in drug courts -- for example, immediate response to noncompliance, collaborative planning and problem-solving, and creativity in devising a range of responses (rewards as well as sanctions) to participants' performance in the program -- are forerunners of broad changes in court and justice system operations. It is important for national-level evaluators to gain a thorough understanding of the evolution of drug courts and their impact on the full range of justice system processes. Perhaps most important, national-level evaluations should address the "why" questions, developing knowledge not only about what type of approaches are most effective in achieving reduced recidivism, reduced substance abuse, and other goals, but also about why they are effective.

D. Overcoming the Obstacles to Sound Evaluation Research

There are six major obstacles to sound evaluation research on drug courts. These obstacles, and suggestions for ways to overcome them, are discussed in the remainder of this section of the report.

1. Difficulty in obtaining relevant data on the behavior of comparison group members and of drug court participants who have completed the program or been terminated for other reasons. The identification of appropriate comparison groups -- consisting of persons who are similar in terms of salient characteristics to the persons admitted to the drug court program, and for whom it will be possible to collect essential data -- is a key element of sound evaluation. Follow-up data on the behavior of participants who have completed the program is also very important to gain a sense of the long-term impact of the drug court. Identifying comparable groups could be difficult but not an insurmountable obstacle. However, obtaining some kinds of data on the behavior and life circumstances of comparison group members, as well as on drug court participants who have graduated or left the program for other reasons -- especially over an extended period of time -- could be extremely difficult. The 1997 GAO report on drug courts recognizes these difficulties (Report, page 86) and observes that drug courts ultimately "may have to be quite innovative in their strategies for collecting such data and may need to resort to sampling and surveying participants and nonparticipants."

Given the current state of drug court management information systems, obtaining data even on active drug court program participants can be very difficult. Learning about criminal behavior of program completers and comparison group members is feasible, because criminal history data are accessible. However, obtaining data on the substance abuse behavior and life circumstances of persons over whom the drug court program has no supervising authority is at best extremely difficult and quite expensive. The March 1997 JMI Working Paper Drug Court Evaluations and Related Research, which was among the materials considered by the evaluation focus group participants, contains a number of suggestions about possible sources of such data, including searches of criminal history records and probation department files, searches of statewide substance abuse treatment databases, and locating individuals to interview. While these strategies can be useful for a well-funded study such as those now being conducted by NIJ, they are currently almost certainly beyond the capacity of most locally funded evaluations. Development of the capacity to conduct comparison group evaluation studies that use follow-up data on drug use relapse will require close collaboration of justice system and treatment officials at all levels of government, as well as major investments in improved management information systems for drug courts.

One type of comparison that is feasible -- and can be valuable for evaluating the impact of drug courts -- is a before-and-after comparison of drug court participants themselves. Drug court programs typically obtain a considerable amount of baseline information on participants' life circumstances -- including criminal history data and information on substance abuse, employment, housing, family situation, and physical health -- when participants are first admitted to the program. It is clearly feasible to examine the extent to which retention in the drug court program is correlated with changes in participants' criminal behavior, substance abuse patterns, income, employment, and other life circumstances variables.

2. The Changing Nature of Drug Courts.
Drug courts are "moving targets" for an evaluator. Because the field is evolving so rapidly and practitioners are constantly learning from their own and others' experiences, almost every drug court is undergoing changes in variables such as the target population, the points in the process at which a person can enter the drug court, the components of the testing regimen and treatment program, and the personnel involved in the program. For evaluators even to know what it is they are evaluating, they must have a very good understanding of the history of the program and how all of its components have developed over time. For evaluations of programs already in operation, the only ways to acquire this type of information are to review documents, interview the practitioners and policymakers who have been involved in the program, and perhaps also interview some of the former and current drug court participants.

3. Lack of a Cadre of Experienced Researchers. One obstacle to obtaining good evaluations is a paucity of researchers who are familiar with both the criminal justice and substance abuse treatment systems and have the necessary methodological skills. This, however, should not be a major barrier. Most drug courts are located reasonably close to at least one college or university. It should be possible to interest professors and graduate students in conducting an evaluation of the drug court, and there are now enough materials on drug courts to enable them to educate themselves rapidly on the drug court concept and the key issues that need to be addressed through evaluation. Further, they can learn quickly about the drug court concept and about the goals and operational procedures of the local drug court by spending time observing drug court and interviewing practitioners and program participants. Other potential evaluators can be found in government agencies and in nonprofit organizations and for-profit consulting firms. DCPO can help educate novice drug court researchers about key evaluation issues and problems by bringing them together with researchers and practitioners who have been grappling with issues in the field for several years. Additionally, the AU Drug Court Clearinghouse has extensive materials on drug courts that will be helpful to potential evaluators.

4. Resources for Evaluation. This is a major issue, especially for local-level evaluations. It is not realistic to expect a highly sophisticated, comprehensive evaluation of a drug court program, using comparison groups and analyzing follow-up data on recidivism and drug abuse relapse, to be undertaken on a shoestring budget. However, very few drug courts are in a position to obtain the substantial funding necessary for a comprehensive and methodically sophisticated multiyear evaluation. It is possible to learn a great deal about drug court program impact and effectiveness from relatively modest evaluation efforts that are well focused. Experienced researchers who participated in the focus group commented that one key to sound and affordable evaluation is ongoing interaction between the researcher and the drug court policy group, optimally beginning while the drug court is still in the initial planning stages. With early involvement of the prospective evaluator, it should be possible to collaboratively shape goals, identify relevant outcome measures, and devise ways to ensure that the data necessary for the evaluation can be obtained at a cost that the program can bear.

5. Inadequate Management Information Systems. The state of management information systems in most jurisdictions is generally inadequate. With a few notable exceptions, most jurisdictions simply do not have the capacity to maintain, in an automated format, the kind of detailed client information needed to support drug court management or evaluation. A closely related problem is the difficulty of linking the drug court management information that does exist with other data sets that could shed light on the long-term impact of drug courts -- including data systems dealing with employment services, welfare, medical services, and substance abuse treatment services. There are problems both with linking the data electronically and sharing the data, because of regulations that govern the disclosure of personal information. Suggestions for developing improved MIS capabilities are discussed in Section III of this report.

6. Lack of Clarity About Evaluation Priorities. The relationship between local-level and national-level evaluations is not clear. In particular, what should be the priorities for local-level evaluations, and how should evaluation at the local level relate to national-level evaluations funded by NIJ and other Federal agencies? Particularly in view of the need for local drug court programs to use limited funds to address the treatment and supervision needs of drug court participants, it is difficult to justify large expenditures of local funds for expensive though methodologically sophisticated evaluations. Further, some of the questions that DCPO or GAO might want to have answered for purposes of shaping national policy with respect to drug court funding might not be of high priority to local-level policymakers and funding authorities. These issues were not explored in depth at the focus group meetings, but there was a general sense that they warrant further attention and that it would be helpful for local-level drug court leaders to have access to the best thinking about evaluation priorities and to develop a clear sense of the respective responsibilities of local and national-level evaluators. One way to do this would be to establish an "evaluation steering group" that includes practitioners and researchers involved in both national and local-level evaluations. Such a group, working with DCPO and NIJ officials, could help bridge gaps, identify problems, and strengthen national-local linkages with respect to evaluation. Additionally, as State administrative offices of courts and State alcohol and drug abuse agencies become increasingly involved in shaping policy for drug courts, it will be important to ensure that these agencies are involved in developing evaluation plans.

Back to Table of Contents