Title: Solicitation for Evaluations of Office of Juvenile Justice and Delinquency Prevention Discretionary Funds Projects Evaluability Assessment: Teens, Crime, and Community Program and Community Works Training Series: Solicitation Author: National Institute of Justice Published: June 2003 Subject: Funding resources 24 pages 50,000 bytes ------------------------------- Figures, charts, forms, and tables are not included in this ASCII plain-text file. To view this document in its entirety, download the Adobe Acrobat graphic file available from this Web site. ------------------------------- Preface: Teens, Crime, and Community Program and Community Works Training Staff Contact: Joannie Delgado Collins Academy Program Coordinator 602-340-7279 Lori Jackson National Crime Prevention Council (NCPC) Program Director for Youth Services 202-261-4161 NIJ has identified some key outcome variables and other parameters of interest for this project and also has provided some guidance on possible evaluation designs. Applicants may depart from this guidance by providing appropriate rationale. NIJ believes that evaluating the program's outcomes through a national sampling of TCC/CW sites is cost-prohibitive. Because of some of the data advantages mentioned in the evaluability assessment, NIJ asks applicants to consider a multi-site evaluation of TCC programs being implemented in Arizona. NIJ suggests a maximum project length of 4 years. NIJ is interested in two broad questions regarding program outcomes: o Does the TCC/CW program influence students in terms of academic performance, involvement in community affairs, and involvement in risk- seeking behaviors? The evaluation should take into account delays of program effects over time. o Suppose that a strong, well-implemented TCC/CW program can produce measurable, positive short- and long-term outcomes. At what reduced levels or intensities of implementation are program benefits no longer measurable? NIJ expects the cost of this evaluation to be no less than $1 million. Total funds available for all six evaluations covered by this solicitation are approximately $5 million. ------------------------------- Evaluability Assessment: Teens, Crime, and Community Program and Community Works Training SYNOPSIS Grantee: National Crime Prevention Council (2002-JS-FX-0016) Grant Period: October 1, 2002-September 30, 2003 Grant Award: $810,000 Funding History: Previously, the grant has been funded at the amount of $900,000 a year for years 1999-2001 and $1,025,000 a year for 1997 and 1998. The project has also been funded by OJJDP in years previous to 1997. Project Summary: The National Crime Prevention Council (NCPC) and Street Law, Inc. have been administering the Teens, Crime, and the Community (TCC) program for many years with OJJDP funding. The program relies on the Community Works (CW) curriculum to provide training and education to teens to assist them in developing skills to protect themselves from becoming crime victims and to help them avoid drugs and delinquent behavior in schools and the community. Scope of Evaluation: The evaluation would use a test and control group. It is also possible to obtain before and after data on the individuals in the test group and some aggregate data on the comparison group. Obtaining "before" data on the comparison group may be difficult because individuals will not be identified. Summary of Evaluability Assessment Activity: Consultant reviewed materials related to the project, including the grant application, program brochures, the CW training notebook, and more. On February 20, 2003, consultant and Ed Zedlewski, NIJ Senior Scientist, met with NCPC project staff, which included Jack Calhoun, Steve Edwards, Marilyn Bassett, and Lori Jackson. Meeting participants viewed a PowerPoint presentation on the project, and consultant and Dr. Zedlewski conducted in-depth interviews of key project staff. In the following weeks, they visited three expansion center sites--Rhode Island/New England, Arizona, and South Carolina. Summary reports from the three site visits are included in this assessment. Findings: While the CW program can be evaluated using the Arizona demonstration site's format, tracking youth cohorts for several years will be expensive. In addition, it may be impossible to prove, reliably and definitively, that taking the CW curriculum prevented teens from becoming victims or acquiring criminal records several years later. Too many other variables (e.g., finding role models outside the CW program or having strong moral behavior before entering the CW program) also may have influenced the teens' behavior. Therefore, causality cannot be demonstrated easily. Such short-term goals as improvement in self-esteem, involvement in school and community projects, or learning about the law can be tested more reliably. ANALYSIS What is already known about projects like these? What could an evaluation of this project add to what is known? Quite a bit of research related to risk factors shows that a youth's environment (home, neighborhood, school) can influence future behavior in terms of delinquent acts or a productive life.[1] The more a teen is exposed to high risk factors, the greater the likelihood of delinquency. Some research shows that exposing youths to positive factors, such as the presence of helpful role models or education and training in problem solving, impulse control, and esteem building, also helps build and foster resiliency to deal with risk factors.[2] The Law-Related Education (LRE) program, developed by Street Law, Inc. has been in existence for more than 20 years. The initial LRE programs targeted youths under the control of juvenile courts. The educational thrust focuses on teaching the values of the rule of law and democracy and principles and skills to function lawfully. The first LRE program was introduced into the Alexandria Juvenile Court System under Chief Judge Joseph Peters in the early 1980s. In this project, NCPC has teamed with Street Law, Inc. to develop and deliver the CW curriculum to teenagers around the country. The CW curriculum is delivered in a variety of settings--middle and high schools, community centers with after-school programs, juvenile court and corrections facilities, and more. The training is also delivered by a variety of trainers--teachers, police officers, probation officers, community members, Americorps staff, and others. It is a very comprehensive and well thought out program with years of foundational experience (i.e., NCPC's prior training efforts and the extensive history of LRE). Unlike programs that rely solely on classroom teaching (i.e., Drug Abuse Resistance Education (D.A.R.E. [registered trademark])), the CW program also includes youth-directed "action projects" designed to provide teens with a practical experience of improving the school and community environments. A comprehensive evaluation of this program would help determine the influence of the combination of activities--CW training curriculum, exposure to role model/resource persons, and the action projects--on reducing teen victimization and helping youths become resilient enough to avoid delinquent behavior. What audiences would benefit from this evaluation? Audiences interested in the evaluation of this program would include juvenile justice field practitioners and researchers, some people in the health and wellness field, school administrators and educators, and Federal agencies that fund juvenile prevention-related programs. What could they do with the findings? Positive findings would help school administrators make more informed choices about how valuable curriculum hours should be allocated. Is grantee interested in being evaluated? This grantee is interested in being evaluated. NCPC feels that a positive evaluation would help it market the program, especially to school administrators. NCPC indicated that its staff would cooperate and help support an evaluation. Are local sites interested in being evaluated? NCPC has contracted with Caliber Associates to do some preliminary process evaluation work and develop an outcome evaluation plan. In terms of process evaluation, the contractor conducted a survey of CW sites and visited several expansion centers to conduct more in-depth interviews on CW usage. The contractor has delivered a draft report and a draft "next steps" on outcome evaluation. Three local sites were visited, and each expressed interest and support for an evaluation. What is the background/history of the program? Under the Teens, Crime, and Community Program, NCPC has been working with communities; schools; and Federal, State, and local governments to implement mentoring and educational programs for youths. TCC, created in 1985 with OJJDP funding, is a partnership between NCPC and Street Law, Inc. The latest curriculum is called "Community Works: Smart Teens Make Safer Communities." The entire TCC program includes: o Training youths using the CW curriculum, which includes 31 sessions of interactive lessons dealing with such topics as guns, violence, gangs, hate crimes, substance abuse, conflict management, and preventing victimization. o Using community resource people (e.g., teachers, police officers, doctors, lawyers) as role models to help deliver the curriculum. o Guiding youths to implement "action projects" (e.g., school or neighborhood cleanups, conducting essay or poster sessions on crime prevention) that allow teens to apply what they have learned to school and community settings. NCPC reports indicate that CW has more than 1,400 users and has reached 700,000 teens across the United States. The program relies on 11 expansion centers, regionally located throughout the country, that serve as mini-program offices to help administer the curriculum and oversee local user sites. Does headquarters monitor fidelity of implementation? Program management monitors the program but also allows and encourages flexibility in the local programs. NCPC and Street Law, Inc. staff oversee the 11 expansion centers. In turn, the expansion center staff oversee local CW user sites (a middle or high school, for example, or a community after-school center) that actually deliver the training. Because NCPC and Street Law, Inc. are not paying large sums of money to the expansion center staff (the average subgrant is about $15,000 a year) and are not paying local teachers and trainers at all, they have to be flexible to get program buy-in. In addition, local schools often fit the CW curriculum into another core curriculum, such as health or civics. Thus, teachers may pick and choose curriculum lessons that meet certain needs at the time and skip over others. What are the headquarters' roles in the program? The main roles of NCPC and Street Law, Inc. staff are to (1) support expansion center staff, (2) train new resource persons in the use of the CW curriculum, (3) help with technical assistance, and (4) provide the CW curriculum to trainers. The program also has a newsletter and Web site. What population does the program target? The target population is young people in the age range of 12 to 18 years-- from teens in schools to juveniles under the control of the juvenile justice system. What are the project's goals and objectives? Project goals are to (1) reduce teen victimization, (2) involve youths in positive service in schools and communities, and (3) reduce delinquent behavior. Additionally, the program aims to help youths develop positive attitudes toward teachers, school, law enforcement and authority; raise their self-esteem; improve their sense of responsibility; reduce their disruptive behavior in schools; and improve their communication skills. What project activities comprise the interventions? The main project activities include (1) delivering the CW curriculum in an interactive style in schools, community centers, juvenile court and corrections facilities, and other venues; (2) using resource people to serve as trainers and role models; and (3) enabling young people to engage in "action projects" that allow them to apply the principles they have learned in the training to improve their schools or neighborhoods. What is the logic that connects project activities to project goals? Youths face risk factors in their homes, neighborhoods, and schools; they also have a fear of becoming crime victims. Most youths have some desire to be part of a community (e.g., a school or neighborhood) but that notion has to be nurtured. TCC programs assume that youths, brought together in natural settings and provided with positive role models/teachers, can learn to avoid victimization and crime, gain self esteem and a sense of responsibility, develop leadership skills, and become productive members of society through lessons based on the CW curriculum and applied in "action plans." Is the logic supportable by empirical evidence? Some previous research has underscored the validity of these assumptions.[3] Are there apparent contradictions or conflicts between certain activities and the outcomes expected? No. The activities are well designed to produce the desired outcomes. Of course, the implementation of the activities makes all the difference. All teachers/resource people are not the same--they do not all have the same enthusiasm, teaching skills, and time--and they do not all command the same respect from the teens. In addition, in some instances, the full program is not always delivered. Is the project being implemented as advertised? NCPC and Street Law, Inc. have implemented the project as planned at the management level. At the field level, the program is implemented flexibly by various sites. All teachers or resource people receive training in CW before they can deliver it. However, depending on the circumstances of the site, teachers may deliver lessons on an ad hoc basis rather than in a more complete manner. The CW curriculum is often used to supplement such other core classes as civics or social studies. In South Carolina, CW was used to supplement the Street Law text. Funding does not allow most of the expansion centers to closely monitor the training activities of CW users after they have been trained. At what stage of implementation are sites? The sites represent a mix and variety of implementation styles and stages. The Caliber process evaluation found that "among respondents who were implanting CW, there were low rates of implementation as intended." The evaluation also noted that of the 15 CW sites observed, none was implementing CW as a "truly stand-alone course." In fact, only one site offered more than 30 contact hours. The most promising sites seem to be the "demonstration sites," such as Arizona, that are in the process of implementing more consistently defined programs. What are the staffing levels? At the NCPC and Street Law, Inc. management level, the project funds two program directors from each organization nearly full time. These program directors divide the oversight of the expansion center sites. The project also partially funds other management and support staff. Altogether, the project funds nearly five full-time employees (FTEs) in both organizations for 1 year. At the 11 expansion centers, the funding permits about 5 to 10 percent of an FTEs' time to help administer the program, with the exception of demonstration sites, which receive a little more funding for staff. Is the project stable over time? The program has been very stable over time. The TCC program began in 1985. The current version of the CW curriculum has been used for several years. The experiences of the expansion center sites with the CW curriculum vary from more than 10 years to recently. However, new local sites are developing every day as the program is marketed by NCPC, Street Law, Inc., and the regional expansion centers. What outcomes could be assessed? By what measures? It is possible to evaluate the outcome of the CW training in a fully implemented and carefully monitored setting. The outcome measures might include lower teen victimization, greater teen involvement in positive community/school projects, and lower teen delinquency. What alternative evaluation designs would work (before/after; comparison group)? How could an appropriate comparison group be created? Are sample sizes statistically significant? Is random assignment possible? The best model for evaluation is the program being developed by the Arizona demonstration site. It matches a team of teacher and police (school resource officer (SRO)) or probation officer to deliver the curriculum to middle school students. The teaching team would deliver the CW curriculum to selected students. For example, in a given middle school, there might be three civics teachers at the same grade level. Each teacher might have 125-150 students overall who receive lessons in classrooms of 25-30 students each. One of the civics teachers (teacher A) would be selected to pair with an SRO and be part of the demonstration. This team would teach the CW curriculum to 125-150 students. The contact time would be about 45 minutes a day, 5 days a week, for a school quarter (16 weeks). This would be the test group. The other 250-300 students who also take civics at the same grade level would serve as the comparison group. In addition to the similarly situated comparison group, evaluators might be able to gather "before-and-after" data on the test group of students and more aggregate data on the comparison group. It is unlikely, however, that "before" data could be gathered on the comparison group, since the students would not be identified by name. One would have to rely on aggregate "before" data for the class as a whole. If the target test group began the CW curriculum in the eighth or ninth grade, school, criminal justice, and self-report data could possibly be tracked several years later while the teens are still in high school. It may also be possible to track this data in aggregate for the comparison group. The use of random assignment does not seem likely. It would be too disruptive for the schools to administer. Since the Arizona program plans to select 30 teams, the sample sizes of the test and comparison groups at the school should be adequate. For example, the test group could number more than 2,000 students. What strengths and weaknesses do the designs have? The strength of the design is the sample size and built-in comparison group. The main weakness is the variability of the teaching efforts--some teams will have better teachers than others; some teams will deliver more lessons than others. To make the design work, the site will have to be monitored to ensure some level of consistency across schools. For example, all teams will have to deliver a minimum number of CW lessons; all teams will have to implement action projects, etc. To have the maximum potential impact, the CW curriculum should be delivered to the test group for the entire year--16 weeks is not long enough. In addition, the teams and schools will also have to maintain better records than they do now (e.g., which students missed lessons, which did not participate in action projects, etc.). Also, to maintain consistency, the training teams cannot change during the year of training. Another weakness of the design is the inability to control extraneous variables. For example, the evaluation design does not control the extent to which the control group students receive training similar to CW from their parents, boy and girl scouts or clubs, sports teams, or other means. Nor does it control for the fact that the test group students might also receive citizenship or other training or be involved in other extracurricular activities that have an influence on them that overrides the CW influence (e.g., finding role models outside of the CW program or having strong moral behavior before entering the CW program). Some of this extraneous influence on the test group could be examined by self-report instruments, but it would be costly. Thus, the evaluation design may be reliable to test some of the short-term goals (e.g., improvement in self-esteem, involvement in school and community projects, learning about the law, etc.). However, the methodological shortcomings may be too significant to reliably evaluate the long-term outcomes (i.e., avoiding victimization and delinquency). How long in duration would the evaluation be? Ideally, the evaluation should begin tracking students who receive the CW curriculum in eighth or ninth grade. The students should receive the curriculum for 1 full year. These students should then be tracked through at least 11th grade. Thus, the evaluation would last a minimum of 3 years. What aspects of the project make an evaluation more difficult? Monitoring to ensure consistency of implementation will be time consuming and costly. The sites do not receive enough funding to conduct extensive monitoring. These evaluation funds may be needed to supplement this task. In addition, the existing record keeping may need to be supplemented with special record keeping, such as standard descriptions to compare action projects and student evaluations after certain lessons. The evaluator may need to take annual self-report surveys of a sample of the test and comparison students. What specific outcome variables would be included? What specific activity measures and implementation measures would be included? It will be important for the evaluator to work carefully with the program to ensure agreement on the definitions and measurability of outcome measures. Some examples include the following: [See PDF file for table.] The evaluator will have to use or develop standard instruments that measure self-esteem, leadership ability, and other attitudes. Can services delivered be identified? The teachers need to provide much more documentation about what they do in terms of delivering the CW curriculum. For example, they may need to document the lessons delivered to which students on which days. Not all lessons are delivered during a semester. Not all students attend class every day. It may also be necessary to develop some evaluation instrument on teaching styles, which may have an impact on learning. Can target populations be tracked over time? Several different tracking systems are required to provide data for the evaluation. The trainers can maintain some of the needed records--level of training delivered, student attendance, and details of action projects. Some information, such as changes in attitudes and knowledge about the law and authority, improvement in problem solving or decisionmaking, new attitudes about responsibility and leadership, the incidence of past and future victimization and delinquent acts, will have to come from the student groups by way of self-report instruments. The challenge is to get parental consent for the teens' self reporting. Another challenge will be getting access to the school records and justice system records to track test and comparison groups with such data as grades, in-school behavior records, and juvenile arrest records. Would the evaluator have to generate new or additional data? It depends on the extent to which the demonstration site received enough funding to adequately monitor the implementation of the CW curriculum. In order to evaluate the consistency of the program, reporting and monitoring will have to be done to ensure that the teaching teams remain constant, training of teachers is similar, the same number of CW lessons are delivered, the teaching styles are similar (or not different enough to be an extraneous variable), action projects are implemented, and more. If the demonstration site cannot do all the monitoring and reporting, then the evaluator will have to supplement the resources to do it. Is there routine reporting of specific data from the local sites? Expansion centers report summary information on the number of presentations they made of the CW program. They also report the number of new local training sites and new trainers. They report on the number of new trainers who were trained in the CW curriculum. However, there is no consistency in the reports by local sites on how often and to whom they delivered the CW curriculum. CONCLUSIONS While the CW program can be evaluated using the Arizona demonstration site format, tracking youth cohorts for several years will be expensive. In addition, it still may not be possible to reliably and definitively prove a clear causal relationship between teens taking the CW curriculum and several years later not being victims or having a criminal record. It simply may not be possible to account for all possible extraneous variables that may have influenced the teens' behavior (e.g., finding role models outside of the CW program or having strong moral behavior before entering the CW program). The appendixes to this report discuss findings from the review of the three expansion centers in Arizona (appendix A), New England (Rhode Island) (appendix B), and South Carolina (appendix C). In summary, the sites vary considerably by the degree to which they have implemented the CW curriculum in a comprehensive and consistent manner. Multisite comparisons of the expansion centers would be largely descriptive, and a synthesis of results would require considerable professional judgment. In terms of commonalities among the three sites, they all follow the same basic program: They use the same CW curriculum, recruit and train new resource people, and use NCPS and Street Law, Inc. as technical assistance resources. The sites differ in the following ways: o Some of the centers have sites that implement a "full" curriculum, but most do not. o The degree to which local sites implement the "action project" with classes varies considerably. o The centers do not have the resources to extensively monitor how the local sites implement the program. NOTES 1. See training materials that are based on the work of J. David Hawkins and Richard F. Catalano: Developmental Research and Programs, Inc., Communities That Care: Risk-Focused Prevention Using the Social Development Strategy; An Approach to Reducing Adolescent Problem Behaviors, 1993, NCJ 143996. 2. Wright, N.D., From Risk to Resiliency: The Role of Law-Related Education, Calabasas, California: Center for Civic Education, 1996. 3. Gottfredson, D., and G. Gottfredson, "Quality of School-Based Prevention Programs: Results from a National Survey," Journal of Research in Crime and Delinquency, 39 (1) (2002). ------------------------------- Appendix A: Arizona TCC Expansion Center SYNOPSIS Contact: Joannie Delgado Wilson, academy program coordinator, Joannie.Delgado@azflse.org, 602-340-7279, Arizona Foundation for Legal Services & Education Funding: NCPC has selected the Arizona Foundation for Legal Services & Education (AZFLS&E) as a demonstration site for the TCC program. As such, NCPC is providing more funding than it provides to the regular expansion sites. In this case, AZFLS&E will receive $51,000 from NCPC- -and match these funds with its own $31,200 "in kind" services and $8,000 in "in kind" services from Street Law, Inc. Scope of Evaluation: The evaluation would use a test and control group. It is possible to obtain before-and-after data on individuals in the test group and some aggregate data on the comparison group. The "before" data on the control group may be more difficult to obtain because individuals will not be identified. Summary of Evaluability Assessment Activity: Consultant interviewed the local project director (PD) by telephone, reviewed the current grant application and other materials about the foundation, and talked with NCPC staff about the site. On March 20, 2003, consultant visited the site with Ed Zedlewski, senior scientist at NIJ, and Cornelia Sorenson of NIJ's evaluation unit. During this visit, the site visit team conducted detailed interviews of the PD, Jeffrey Schrade, Technology Education Coordinator, and Susan Nusall, State LRE Program Coordinator. The team also interviewed a police officer/trainer from the city of Chandler. Finding: Of the three TCC expansion centers visited, the Arizona site shows the most promise for implementing and helping manage a local program evaluation. ANALYSIS Grantee Level of Cooperation: This sub-grantee is interested in being evaluated and feels that a positive evaluation would help market the program, especially to school administrators. There is no local evaluation. Background: Since 2000, the AZFLS&E has managed the Law Related Education Academy for the State of Arizona and delivers training to school safety officers, in conjunction with the State department of education, under Arizona's mandated program to create safer schools. Since January 2002, the foundation has trained more than 90 police and probation officers (school safety officers funded by the State) in the TCC program. However, due to limited funding, subsequent to providing the training to the officers, the foundation was not able to offer any followup technical assistance or monitor the delivery of training. As a consequence, the foundation did not track the extent to which officers delivered CW training after they received the foundation's training. NCPC recently asked AZFLS&E to become a demonstration site for the TCC program. This will provide more funding to the foundation and allow it to do more followup and monitoring. What population does the program target? The target population is teens in the age range of 12-18 years. The youths range from teens in schools to juveniles under the control of the juvenile justice system. What are the project goals and objectives? Project goals are to (1) reduce teen victimization, (2) involve youths in positive service in schools and communities, and (3) reduce delinquent behavior. Additional objectives include developing positive attitudes toward their teachers, school, law enforcement and authority; raising their self-esteem; improving their sense of responsibility; reducing their disruptive behavior in schools; and improving their communication skills. What project activities comprise the interventions? The main project activities are (1) delivering the CW curriculum in an interactive style in schools, (2) using resource people to serve as trainers and role models, and (3) involving youths in "action projects" that allow them to apply the principles they have learned in the training to improve their schools and neighborhoods. What is the logic that connects project activities to project goals? Youths face risk factors in their homes, neighborhoods, and schools. They also have a fear of becoming crime victims. Most youths have some desire to be part of a community (e.g., a school or neighborhood), but that notion has to be nurtured. The TCC program assumes that youths, brought together in natural settings and provided with positive role models/teachers, can learn to avoid victimization and crime, gain self- esteem and a sense of responsibility, develop leadership skills, and become productive members of society through lessons based on the CW curriculum and applied in "action projects." Is the logic supportable by empirical evidence? Some previous research has underscored the validity of these assumptions (see Gottfredson and Gottfredson). Are there apparent contradictions or conflicts between certain activities and the outcomes expected? No. The activities are well designed to produce the desired outcomes. Of course, the implementation of the activities makes all the difference. All teachers/resource people are not the same--they do not all have the same enthusiasm and teaching skills--and they do not command the same respect from the teens. In addition, in some instances, the full program is not always delivered. Implementation Issues: The implementation of the TCC program in Arizona has been flexible. The foundation has not had the staff resources to follow up on the individual sites/trainers to see to what degree they have delivered the CW curriculum. Now that the expansion center has become a demonstration site, with some additional funding from the project, staff may be able to establish more consistent programming. Currently, the foundation is testing a new program format. They are attempting to match police and probation officers with teachers to create 30 teams throughout the State. The teaching team would deliver the CW curriculum to selected students. For example, in a given middle school, there might be three civics teachers at the same grade level. Each teacher might have 125-150 students overall who receive lessons in classrooms of 25-30 students each. One of the civics teachers (teacher A) would be selected to pair with an SRO and be part of the demonstration. This team would teach the CW curriculum to 125-150 students. The contact time would be about 45 minutes a day, 5 days a week, for a school quarter (16 weeks). Evaluation Design: The above format may present an opportunity for evaluation. The students who are taught the CW curriculum by these new teams would be the test group; the other students in the same grade who did not receive the CW curriculum might be the control group. In addition to the similarly situated comparison group, evaluators might be able to gather before-and-after data on the test group of students and more aggregated before-and-after data on the comparison group. If the target test group began the CW curriculum in the eighth or ninth grade, one might be able to track school, criminal justice, and self-report data several years later while the teens are still in high school. It may also be possible to track this data in aggregate for the comparison group. The use of random assignment does not seem likely. It would be too disruptive for the schools to administer. Since the Arizona program plans to select 30 teams, the sample sizes of the test and comparison groups at the school should be adequate. For example, the test group could number more than 2,000 students. What strengths and weaknesses do the designs have? The strength of the design is the sample size and built-in comparison group. The main weakness is the variability of the teaching efforts. Some teams will have better teachers than others; some teams will deliver more lessons than others. To make the design work, the sites will have to be monitored to ensure some level of consistency across schools. For example, all teams will have to deliver a minimum number of CW lessons; all teams will have to implement action projects, etc. In addition, the teams and schools will have to maintain better records than they do now, maintaining data on topics such as which students missed lessons or did not participate in action projects. Another weakness of the design is the inability to control extraneous variables. For example, the evaluation design does not control the extent to which the control group students receive training similar to CW from their parents, boy and girl scouts or clubs, sports teams, or other means. Nor does it control for the fact that the test group students might also receive citizenship or other training or be involved in other extracurricular activities that have an influence on them that overrides the CW influence. Some of this extraneous influence on the test group could be examined by self-report instruments, but it would be costly. Measurement Model: It will be important for the evaluator to work closely with the program to ensure agreement on the definitions and measurability of the outcome measures. Some examples include the following: [See the PDF file for table.] The evaluator will have to use or develop standard instruments that measure self-esteem, leadership abilities, and other attitudes. Can services delivered be identified? The teachers need to provide much more documentation about what they do in terms of delivering the CW curriculum. For example, they may need to document which lessons were delivered to which students on which days. Not all lessons are delivered during a semester. Not all students attend class every day. Can target populations be tracked over time? Several different tracking systems are required to provide data for the evaluation. The trainers can maintain some of the needed records--data on the level of training delivered, student attendance, details of action projects, etc. Some information, such as data on changes in attitudes and knowledge about the law and authority, improvement in problem solving or decisionmaking, new attitudes about responsibility and leadership, and the incidence of past and future victimization and delinquent acts, will have to come from the student groups by way of self-report instruments. The challenge is to get parental consent for the teens' self reporting. Another challenge will be getting access to school and justice system records to track test and comparison groups with such data as grades, in- school behavior records, and juvenile arrest records. Would the evaluator have to generate new or additional data? It depends on the extent to which the demonstration site received enough funding to adequately monitor the implementation of the CW curriculum. In order to evaluate the consistency of the program, reporting and monitoring will have to be done to ensure that the teaching teams remain constant, training of teachers is similar, the same number of CW lessons are delivered, the teaching styles are similar (or not different enough to be an extraneous variable), action projects are implemented, and more. If the demonstration site cannot do all the monitoring and reporting, then the evaluator will have to supplement the resources to do it. Is there routine reporting of specific data from the local sites? Expansion centers report summary information on the number of presentations they made of the CW program. They also report the number of new local training sites and new trainers. They report on the number of new trainers who were trained in the CW curriculum. However, there is no consistency in the reports by local sites on how often and to whom they delivered the CW curriculum. CONCLUSIONS An evaluation of the TCC program as implemented at the Arizona demonstration site is possible. However, the reliability and strength of the causal links--proving that the CW curriculum was responsible for any long- term changes the youth displayed (e.g., no victimization, no delinquency)-- are tenuous. ------------------------------- Appendix B: New England TCC Expansion Center SYNOPSIS Contact: John Mattson, jomattson@aol.com, 401-854-5506, ext. 116 Funding: The local program received $17,000 in funding from NCPC in 2002. About $13,000 of the funding was used to reimburse some labor costs; the rest was used for materials, transportation, and a cluster conference (to bring all training sites together for a meeting). Summary of Evaluability Assessment Activity: Consultant interviewed project director by telephone, reviewed materials (grant application, annual report for 2002), and talked with NCPC staff about the site. On March 5, 2003, consultant visited the site with Ed Zedlewski, senior scientist at NIJ. During this visit, the site visit team conducted a detailed interview of the project director. The team also interviewed a police officer/trainer at a Warwick, Rhode Island, middle school and observed him teach a module to 11 students. In addition, the team met with two other program trainer/facilitators. Finding: There is too much variability and not enough consistency in the way that the local sites use the CW curriculum to conduct an empirical evaluation of the TCC program at this expansion center. ANALYSIS Grantee Level of Cooperation: This grantee is interested in being evaluated and feels that a positive evaluation would help market the program, especially to school administrators. There is no local evaluation. Background: The Warwick Youth Program Advisory and Prevention Committee (WYPAPC) has been an expansion center for NCPC's TCC program for 13 years. It has been delivering the CW curriculum for several years. Eight to 10 trainers deliver CW in a variety of settings including middle schools, truancy courts, community centers, and other after-school program settings. Program Design: The target population for the program is 12- to 18-year- old youths. The program goals include the following: o Develop leadership ability--promote leadership involvement at school. o Advance participant strengths--promote a career orientation. o Provide positive role models, ideals, and opportunities in the community- -promote community involvement. o Reduce delinquent behavior. What project activities comprise the interventions? The main project activities are (1) delivering the CW curriculum in an interactive style in schools, community centers, juvenile court and corrections facilities, and other venues; (2) using resource people to serve as trainers and role models; and (3) enabling youths to engage in "action projects" that allow them to apply the principles they have learned in the training to improve their schools or neighborhoods. What is the logic that connects project activities to project goals? Youths face risk factors in their homes, neighborhoods, and schools. They also have a fear of becoming crime victims. Most youths have some desire to be part of a community (e.g., a school or neighborhood) but that notion has to be nurtured. The TCC program assumes that youths, brought together in natural settings and provided with positive role models/teachers, can learn to avoid victimization and crime, gain self- esteem and a sense of responsibility, develop leadership skills, and become productive members of society through lessons based on the CW curriculum and applied in "action projects." Is the logic supportable by empirical evidence? Some previous research has underscored the validity of these assumptions (see Gottfredson and Gottfredson). Are there apparent contradictions or conflicts between certain activities and the outcomes expected? No. The activities are well designed to produce the desired outcomes. Of course, the implementation of the activities makes all the difference. All teachers/resource people are not the same--they do not all have the same enthusiasm and teaching skills--and they do not all command the same respect from the teens. In addition, in some instances, the full program is not always delivered. Implementation Issues: The project director has extensive experience in youth programs and puts a great deal of effort into them, obviously working far more than the minimal compensation that NCPC provides. In addition to the project director, who works part-time on the program, there are 8-12 active trainers, who deliver the CW training on a part-time basis. Most of the trainers are not paid. They deliver the training on a volunteer basis or are paid by their own employer (e.g., police department, Americorps). The curriculum has a variety of modules. Many of the trainers pick and choose modules depending on the setting and the cooperation of the school administrators, juvenile court judges, after-school program administrators, and others. For example, in the middle school visited, the police officer was delivering CW once a week for 1 hour over 9 weeks. The school was fitting the curriculum into one-quarter of the main health curriculum. Thus, the program is implemented in a variety of ways. It may be difficult to find what would be considered full and complete implementation in many of the training settings. Evaluation Design: There is too much variability and not enough consistency in the way that local sites use the CW curriculum to conduct an empirical evaluation of the program at this expansion center. Measurement Models: N/A Data: Because of limited staff resources, very little data is collected by the expansion center about who receives the CW curriculum and how often. CONCLUSIONS There is too much variability and not enough consistency in the way the local sites use the CW curriculum to conduct an empirical evaluation of the program at this expansion center. ------------------------------- Appendix C: South Carolina TCC Expansion Center SYNOPSIS Contact: Lisa Burgess, lisa.burgess@scbar.org, 803-252-5139 Funding: Approximately $15,000 in 2001. Scope of Evaluation: See "Finding." Summary of Evaluability Assessment Activity: Consultant reviewed materials (grant application, annual report for 2002) and talked with NCPC staff about the site. On March 24, 2003, consultant visited the site with Ed Zedlewski, senior scientist at NIJ, and Cornelia Sorensen, of NIJ's evaluation unit. During this visit, the site visit team conducted a detailed interview of the project director and also interviewed a teacher/trainer by phone. Finding: There is too much variability and not enough consistency in how the local sites use the CW curriculum to conduct an empirical evaluation of the program at this expansion center. ANALYSIS Grantee Level of Cooperation: This grantee is interested in being evaluated and feels that a positive evaluation would help market the program, especially to school administrators. There is no local evaluation. Background: The South Carolina Bar Law Related Education (LRE) Division serves as an expansion center for the TCC program and CW training. The South Carolina Bar LRE division uses CW, along with other curriculums (such as the Street Law book), to deliver law-related education to students throughout the State. The division works with teachers, law enforcement officers, other juvenile justice staff, lawyers, and others to deliver the CW curriculum. The division conducts awareness sessions to recruit new trainers and users. The division is collaborating with the Department of Public Safety to deliver an awareness program at the annual Statewide conference for school resource officers. In an effort to get more acceptance from the State school system, the division hired a consultant and had the CW curriculum correlated to South Carolina social studies standards. Unfortunately, CW was not approved by the State to be an approved school text. Program Design: The target population is 14- to 18-year-old teens. What are the project goals and objectives? Project goals are to (1) reduce teen victimization, (2) involve youths in positive service in schools and communities, and (3) reduce delinquent behavior. Additional objectives include developing positive attitudes toward teachers, school, law enforcement and authority; raising self- esteem; improving the sense of responsibility; reducing disruptive behavior in schools; and improving communication skills. What project activities comprise the interventions? The main project activities include (1) delivering the CW curriculum in an interactive style in schools and other outreach locations; (2) using resource people to serve as trainers and role models; and (3) enabling youths to engage in "action projects" that allow them to apply the principles they have learned in the training to improve their schools or neighborhoods. What is the logic that connects project activities to project goals? Youths face risk factors in their homes, neighborhoods, and schools. They also have a fear of becoming crime victims. Most youths have some desire to be part of a community (e.g., a school or neighborhood) but that notion has to be nurtured. The TCC program assumes that youths, brought together in natural settings and provided with positive role models/teachers, can learn to avoid victimization and crime, gain self- esteem and a sense of responsibility, develop leadership skills, and become productive members of society through lessons based on the CW curriculum and applied in "action projects."-- Is the logic supportable by empirical evidence? Some previous research has underscored the validity of these assumptions (see Gottfredson and Gottfredson). Are there apparent contradictions or conflicts between certain activities and the outcomes expected? No. The activities are well designed to produce the desired outcomes. Of course, the implementation of the activities makes all the difference. All teachers/resource people are not the same--they do not all have the same enthusiasm and teaching skills--and they do not command the same respect from the teens. In addition, in some instances, the full program is not always delivered. Implementation Issues: The CW curriculum is implemented in a very flexible manner. The expansion center staff do not have the resources to comprehensively track the users to see how fully or completely they are delivering the program. For example, the Street Law text is used more fully in the classroom as part of social studies. A teacher who has been trained to use CW may use a single module in place of or to supplement a Street Law module. Evaluation Design: There is too much variability and not enough consistency in the way that local sites use the CW curriculum to conduct an empirical evaluation of the program at this expansion center. Measurement Model: N/A Data: Because of limited staff resources, the expansion center collects very little data about who receives the CW curriculum and how often. CONCLUSIONS There is too much variability and not enough consistency in the way that local sites use the CW curriculum to conduct an empirical evaluation of the program at this expansion center.