Title: Solicitation for Evaluations of Office of Juvenile Justice and Delinquency Prevention Discretionary Funds Projects Evaluability Assessment: Father Flanagan's Boys' Home (Girls and Boys Town) Series: Solicitation Author: National Institute of Justice Published: June 2003 Subject: Funding resources 22 pages 50,000 bytes ------------------------------- Figures, charts, forms, and tables are not included in this ASCII plain-text file. To view this document in its entirety, download the Adobe Acrobat graphic file available from this Web site. ------------------------------- Preface: Father Flanagan's Boys' Home (Girls and Boys Town) Staff Contact: Mona Way Director of Girls and Boys Town USA Support Services National Headquarters 402-498-3215 Alex Figueroa Director of Newark (New Jersey) Girls Town 973-824-5095 NIJ has identified some key outcome variables and other parameters of interest for this project and also has provided some guidance on possible evaluation designs. Applicants may depart from this guidance by providing appropriate rationale. NIJ requires applicants to consider a multi-site evaluation of a minimum of three fully implemented programs for females, one of which must be the Newark, New Jersey, site. NIJ suggests a maximum project length of 3-4 years. NIJ is interested in two broad questions regarding program outcomes: o Does the Girls and Boys Town (GBT) program established for female participants influence participants in terms of the short-term outcomes described in the evaluability assessment? In addition to short-term outcomes, NIJ proposes a 12- and 24-month followup on program participants and a non-participants' comparison group that examines long- term outcomes, especially those related to public safety. o The GBT target population is referred from several sources including juvenile court, family crisis court, or secure facilities. Are there differential outcomes for participants based on source of intake? NIJ expects the cost of this evaluation to be no less than $400,000. Total funds available for all six evaluations covered by this solicitation are approximately $5 million. ------------------------------- Evaluability Assessment: Father Flanagan's Boys' Home (Girls and Boys Town) SYNOPSIS Grantee: Father Flanagan's Boys' Home (2002-JSF-X0009) Grant Period: July 1, 2002-June 30, 2003 Grant Award: $1,350,000 Funding History: Unknown. Project Summary: Father Flanagan's Boys' Home (Girls and Boys Town) is operating staff-secure detention centers for female juvenile offenders in two sites under the grant: Newark, New Jersey, and New Orleans, Louisiana. These centers replicate the model already established in 17 cities across the Nation. The program provides a safe detention placement for female juvenile offenders who do not require placement in a secure detention facility and cannot return home. Generally, the length of stay in these facilities is expected to range from 14 to 90 days. The young women are treated individually with a focus on achieving long- and short-term goals, including teaching life skills as well as academic subjects, and promoting behavior change in targeted problem areas. The centers teach these skills through a community-based philosophy, as they create a family-style atmosphere, including family-style meals, sharing of chores, and an emphasis on relationships with staff. The ultimate goals are to equip female juvenile offenders with the skills and tools to prevent recidivism and to reintegrate them into the community. The program is dedicated to gender- specific policies and programs in response to a growing young female offender population. Scope of Evaluation: Grant funds were awarded to Girls' and Boys' Town (GBT) to replicate its teaching model in two sites: Newark and New Orleans. The model provides short-term residential care for female delinquent offenders who are either awaiting long-term placement by the juvenile court or hoping to avoid more secure placement by successfully completing the program. The family-style atmosphere incorporates a cognitive-behavioral approach to equip female juvenile offenders with the necessary skills and tools to prevent recidivism. The young women are treated with a focus on both long- and short-term goals, including teaching life skills as well as academic subjects, and promoting behavior change in targeted problem areas. Summary of Evaluability Assessment Activity: Program evaluability was assessed through document review, telephone interviews, and a site visit to the Newark Girls' Town program. Preliminary activities included a review of the 2002 grant application, the progress report covering August through December 2002, and a telephone conversation with the grant monitor. A telephone interview was also conducted with the grantee at GBT in Omaha, Nebraska, who also provided data collection forms and a GBT evaluation conducted with the University of Nebraska. The site visit to Newark included interviews with the program director, the program manager, youth-care workers, supervisors, and program residents. Findings: The program would have a high probability of success. It has staff support, a number of data collection processes already in place, and a national headquarters to monitor program implementation and analyze program outcomes. Effective alternatives to prevent recidivism among juvenile delinquents have not been rigorously addressed in the literature or in practice, and delinquency is rising faster among girls and young women than among boys and young men. Therefore, researchers, funding agents, and policymakers would benefit from an outcome evaluation of this program. ANALYSIS Although overall juvenile crime rates have declined during the past 10 years, female involvement in the juvenile justice system has increased during this same time.[1] A review of FBI statistics demonstrates that, between 1994 and 1998, the number of arrests of female juvenile offenders either increased more or decreased less than the number of arrests of male juvenile offenders in most offense categories (OJJDP 2000). Between 1988 and 1997, the number of delinquency cases involving males increased 39 percent, while cases involving females increased 83 percent.[2] Female juvenile arrests increased by almost 19 percent between 1992 and 2001.[3] In previous decades, girls who have broken the law have entered a juvenile justice system designed to serve a predominantly male population and engineered to guide troubled boys toward becoming law-abiding men.[4] Clearly, as more girls continue to commit crimes with increasing frequency and severity, the prevention and treatment communities must develop more effective ways to serve female juvenile offenders. What is already known about projects like these? Studies have evaluated the effectiveness of the GBT teaching model as it has been applied in numerous settings. A study by the Center for Effective Collaboration and Practice examined how the GBT model works to increase positive outcomes in school settings.[5] Its research included the following outcome data: o Students' on-task behavior increased by 10 percent after the first year and 17 percent after the second year. o There were significantly fewer office referrals for both verbal and physical violence in the target elementary schools after the model was applied. o Office referrals in the middle schools studied decreased 58 percent after the implementation of the model. Additionally, citing direct evaluation data regarding the Philadelphia model of the Staff Secure Detention Facility for Female Juvenile Offenders, the Children's Defense Fund reports that results from the program seem to be positive. The Philadelphia program offers short-term detention for girls age 11 through 18 who are awaiting more permanent placement by the juvenile court. After release, girls in the program were less likely to be arrested or to run away, compared to youth in other Philadelphia institutional programs.[6] A longitudinal evaluation of the same Philadelphia program assessed 155 program participants during their stay in the facility and for 4 years following their release. Twelve percent of the sample recidivated within a year of their placement in the GBT program, and 38 percent reoffended within 4 years.[7] The study found that these recidivism rates were substantially lower than those of similar national programs. Other, similar gender-specific programs have demonstrated positive outcomes as well. The PACE Center for Girls operates a day-treatment program that offers comprehensive prevention, early intervention, and high school education to adolescent girls who are at risk of delinquency and in the same age range that the Staff Secure Detention model serves. The majority of PACE participants (98 percent) successfully completed the program and did not reoffend within 1 year of graduation. Furthermore, 88 percent of graduates demonstrated at least one of several positive outcomes, including staying in school, earning a high school diploma or General Education Development (GED) equivalency, securing gainful employment, and moving on to higher education or vocational training. What could an evaluation of this project add to what is known? Few studies have assessed the effects of various placement alternatives on female juvenile offenders. Even less is known about short-term secure placement for either preadjudicated or postadjudicated female delinquents. Therefore, an evaluation of the GBT program would help bridge this gap in the field of juvenile delinquency--an increasingly important gap given the rising rates of female delinquency in comparison to male rates throughout the 1990s. Which audiences would benefit from this evaluation? What could they do with the findings? Both policymakers and researchers would benefit from an evaluation of this program. Policymakers could use the results of the evaluation to inform future funding decisions for GBT and similar programs that offer short- term placement for female delinquents. Furthermore, communities that currently have too few placement options for these delinquent offenders could learn the potential costs and benefits associated with implementing the GBT model. Finally, researchers in the field of female delinquency would benefit from more rigorous evaluations regarding the impact of the program on recidivism, risk factors for more serious delinquency, and long- term placements. Is the grantee interested in being evaluated? The grant monitor, the national program staff, and the Newark program staff have all indicated that the grantee is very interested in being evaluated. GBT national headquarters sponsored a longitudinal evaluation of its program in Philadelphia.[8] Furthermore, one program goal identified in the grant application and progress report is to implement a formal process for evaluating program treatment and implementation. The national headquarters also places great emphasis on collecting and analyzing individual program data. The director of program data management analyzes data and prepares monthly reports on critical success factors for each program. The staff in Newark also expressed interest in participating in an outcome evaluation. They discussed several questions related to program outcomes, including recidivism and long-term placement, that could be answered through such an evaluation. What is the background/history of this program? GBT has provided long-term residential care to male juvenile offenders at its Nebraska campus since it was founded in 1917 and to female offenders since 1979. The organization is run by a nonprofit, nonsectarian charitable foundation that has worked to develop new treatment options to address the complex problems of children coming into care. In 1988, the organization decided to expand its GBT teaching model to major metropolitan areas across the Nation. The short-term secure detention program currently operates in 17 locations. Each site is accredited by the Joint Commission on Accreditation of Health Care Organizations. The national headquarters also monitors the faithful implementation of the GBT teaching model at each program site. The model involves short-term residential care for female delinquent offenders who are either awaiting long-term placement by the juvenile court or hoping to avoid more secure placement by successfully completing the program. The family-style atmosphere incorporates a cognitive-behavioral approach to equip female juvenile offenders with the skills and tools to prevent recidivism. Girls and young women are treated with a focus on both long- and short-term goals, including teaching life skills as well as academic subjects, and promoting behavior change in targeted problem areas. The current OJJDP grant supports expansion of this model to two new metropolitan sites: Newark and New Orleans. At what stage of implementation are the sites? The short-term secure placement for juvenile offenders model has been implemented since 1988. The programs supported by the current OJJDP grant, however, began implementation in 2002. Both the Newark and New Orleans sites spent the early part of 2002 renovating buildings to house female delinquent offenders. Both programs began receiving residents in late summer/early fall 2002. At the time of the evaluability assessment, the programs were at or near full capacity. What are the project's outcome goals in the view of the project director? The main goals discussed in the grant application and progress report relate to the successful replication and implementation of the GBT teaching model in Newark and New Orleans. Other program goals identified in these documents and through conversations with the staff at GBT national headquarters include: o Improved behavior in specific target areas. o Improved ability to demonstrate appropriate social skills. o Improved decisionmaking skills and nonoffending behavior (recidivism). o Improved relationships with parents and family. o Successful community reintegration and/or permanent, less restrictive placement. Does the proposal/project director describe key project elements? Do they describe how the project's primary activities contribute to goals? Both the proposal and the national headquarters staff outline many program elements that are key to achieving the program goals discussed above. The program uses a relationship-based cognitive-behavioral approach to improve the social and life skills of female delinquent offenders. Each resident receives an individual treatment plan based on identified behavioral problem areas, academics, and long-term placement needs. The model uses a family-style atmosphere to teach the residents basic life skills and to emphasize positive adult and peer relationships. Conversations with the national headquarters staff revealed a clear link between these activities and anticipated program outcomes. For example, many of the residents have never had positive relationships with adults or been exposed to positive adult role models. Furthermore, they have never been taught many basic social and life skills that can help them negotiate everyday tasks and avoid problem behaviors. The family-style atmosphere reinforces positive relationships and teaches the participants important life skills, such as cooking, cleaning, hygiene, and decisionmaking. Academic and individual treatment plans are designed to equip the residents with the skills to avoid recidivism and reintegrate into the community successfully. Can you sketch the logic by which activities should affect goals? Yes. See the logic model in appendix A. Are there other local projects providing similar services that could be used for comparisons? No other local programs provide a holistic treatment approach similar to that of the GBT program. In fact, the absence of such short-term placement options for this target population is one of the primary reasons for selecting the metropolitan areas for GBT. Will samples that figure in outcome measurement be large enough to generate statistically significant findings for modest effect sizes? The sample sizes will be large enough to generate statistically significant findings for modest effect sizes if all residents who participate in the program over an extended period of time (e.g., calendar year 2003) are included in the sample. The combined OJJDP grant-supported bedspace of the 2 facilities is 16. With an average stay of approximately 1 month, the treatment sample size collected over 1 year would be nearly 200. An additional option would be to include female delinquent offenders from all short-term placement facilities in all the GBT metropolitan areas (17 existing, and the 2 additional OJJDP-supported sites) in the sample. This selection method would generate a sufficient sample size without a lengthy collection period. Is the grantee planning an evaluation? The grantee is not currently planning an outcome evaluation of the GBT in Newark and New Orleans, although it has recently completed a longitudinal study of 155 participants in its Philadelphia program (discussed above). The grantee does conduct comprehensive assessments of the program activities with the participants during their stay at GBT. These assessments are based on the progress of the residents and the occurrence and handling of critical incidents at the facilities. Residents' progress is analyzed from data on youth behavior and treatment planning and on critical incidents, such as suicidal threats or behavior, violent acts, or runaways. The director of program data management at the national headquarters analyzes the data collected from each individual program and provides monthly reports. What data systems exist that would facilitate evaluation? What are the key data elements contained in these systems? Data systems now in place at the GBT national headquarters would facilitate an evaluation, although they focus primarily on outcomes achieved during the girls' stay at GBT. The facilities track progress on treatment behaviors such as lethality, aggression, sexual issues, problem avoidance, school problem behavior, and substance abuse. Immediate program outcomes are also tracked, including moves to less restrictive placement, favorable departures, length of stay, and proportion of youth goals met at departure. Data are collected monthly at the site level and monitored at the national level. Additional data to track long-term outcomes exist but would have to be accessed through other avenues, such as arrest records, placements in long-term detention facilities, and high school graduation/GED rates. Are data available to estimate unit costs of services or activities? Individual program sites are funded and monitored through the national headquarters, which tracks the cost of these services and activities. Are there data about possible comparison samples? No comprehensive data are now available regarding the activities or outcomes of potential comparison samples. However, several potential data sources within the New Jersey Department of Juvenile Justice (DJJ) may help to identify and track comparison sample members over time. Comparison sample data are discussed further in the "Site Visit Evaluability Assessment" section. In general, how useful are the data systems to an impact evaluation? The data systems currently in place and monitored by GBT are very useful for assessing the activities and short-term outcomes of GBT residents. Appropriate databases exist for comparison sample members and for assessing long-term outcomes such as recidivism, but the evaluator would have to secure access to those data systems. Is the project being implemented as advertised? Interviews were conducted with program directors, supervisors, and youth- care workers during the site visit to the Newark facility. The site visit team also talked informally with several program residents. In nearly every area discussed, the program was being implemented as advertised in the grant application and in interviews with national headquarters staff. These implementation areas include: o Training of program staff. National headquarters staff and program documents report that all GBT workers are brought to Omaha for a 2- week training on the GBT model. Because so many Newark staff were hired at the same time, trainers from the headquarters came to Newark to conduct the training. Staff reported that they were trained in basic treatment components (e.g., daily activities and their role in program goals), appropriate conflict resolution, appropriate consequences for bad behavior by program participants, tracking progress on behavioral goals, and completing data collection forms. Some of the data collection forms are included in appendix B. GBT also uses other data collection instruments, such as the Child Suicide Risk Assessment, the Suicide Probability Scale (SPS) Interview Questionnaire, the SPS Profile Form, and the SPS Rating Form, which are not included in the appendix. o Treatment activities. National headquarters staff described the daily activities of the GBT model, including the family-style atmosphere, group completion of chores and tasks, academics, and treatment activities. Staff and residents at the Newark facility described the same daily schedules and the logic linking these activities to program outcomes. The assessment team also observed the residents in an academic setting and the consequences for inappropriate behavior during class. o Data collection. As discussed earlier, a number of data collection forms now track the progress and activities of GBT residents at the facility and immediate outcomes following their departure. Both national headquarters and Newark program staff described the implementation of the data collection and the usefulness of the data. The Newark staff also presented a whiteboard that tracks all of the current residents, their referral sources, their presenting problems, and their regular treatment activities. Furthermore, the program provided a chart that tracks each of these outcomes on a monthly basis. The monthly report also tracks such facility- level measures as staff turnover and critical incidents. o Participant profile. For the most part, the residents at the Newark facility matched the target population described in GBT documents and by the national headquarters staff. The girls were between the ages of 11 and 18, and either had been sentenced directly to the facility by a juvenile judge or were awaiting a more permanent placement. The program has 14 beds, 10 of which are OJJDP funded for delinquent offenders. The other four are reserved for the Department of Youth and Family Services (DYFS) for youth it is unable to place in other facilities (e.g., foster homes). Currently about half of the beds are filled by DYFS youth, however. These beds are for DYFS emergency use only and are reviewed each week. Newark GBT expects to fill the allocated 10 beds with delinquent female offenders as referral sources and methods are defined further. This is the only area where the Newark GBT implementation did not meet expectations. What is the intervention to be evaluated? GBT provides secure placement for female juvenile offenders who are ordered to the facility by the juvenile court. The girls spend at least 30 days in the program and are released after they have had 30 "good" or problem- free days. Stays can stretch to 90 days in some cases. Within 24 hours of arrival at the facility, each girl receives a treatment plan that is based on her presenting problem behaviors and other needs (e.g., long-term placement, needed social and/or life skills, academics). Residents also complete several forms related to behaviors and risk factors for delinquency and suicide. They are introduced to the basic skills and expectations of GBT. An individual treatment plan is prepared within 72 hours of the girl's arrival, and a youth-care worker is assigned to her. The youth-care worker is in close contact with the girl at all times and reviews her progress toward treatment goals daily. During the day, the girls participate in many activities designed to enhance their life and social skills and to promote positive behavior. In the morning, they are expected to practice good hygiene, clean their rooms, and help prepare breakfast. The girls prepare and clean up after all meals and eat together as a group. The girls spend most of their day in school, instructed by a teacher placed in the facility by DJJ. Youth-care workers are present in the classroom to identify and respond to inappropriate behaviors. Free time in the afternoons and on weekends is filled with group activities (e.g., group outings, recreational activities) to strengthen peer relations and to promote appropriate behaviors with each other. Family meetings also occur each day to promote self-government and self-empowerment. Throughout each day, all GBT staff stress the importance of following instructions, problem solving, asking permission, and teaching alternative behaviors. Progress in these skills is measured through a point system. Upon arrival at the facility, each girl is given a certain number of points that she must work off to be released. Aggressive and other problem behaviors result in more points, while positive behaviors are rewarded with points taken away from the girl's account. At the end of each day, youth-care workers review each girl's progress and point standing with the girl and set future goals. A resident is informed of her imminent release shortly before her scheduled departure. The facility works with the juvenile court and other community agencies to promote participation in aftercare services or treatment programs. Although aftercare is not an official component of GBT, staff believe that it is a critical step in successful reintegration. There are no followup activities with girls once they are released from the program. What outcomes could be assessed? By what measures? Short-term outcomes could be assessed easily through existing data systems. These measures include those identified by the project director: o Improved behavior in specific target areas. o Improved ability to demonstrate appropriate social skills. o Improved decisionmaking skills and nonoffending behavior (recidivism). o Improved relationships with parents and family. Other short-term or in-program outcomes include academic improvement and attendance at indicated treatment programs. All short-term outcomes are available at both the local and national levels through data collection forms completed by both staff and the girls themselves. Long-term outcomes are more critical to an outcome evaluation, but some work would be required to identify appropriate measures and data sources. These outcomes would center on successful community reintegration. Measures could include recidivism, placement in a less restrictive setting, continued participation in school or GED programs, employment (when applicable), participation in indicated treatment programs, probation completion, and teen pregnancy rates. Are there valid comparison groups? No local programs offer similar services to this target population. Comparison samples could be drawn from demographically similar target populations who receive more traditional placement options in each of these communities. These placement options include small shelters, group homes, day-reporting programs that may focus on an isolated component of the GBT teaching model (e.g., substance abuse), foster homes, or more secure and longer term detention facilities. The Newark site offered contact information to investigate the availability of comparison group data in each of the counties that place female juvenile offenders in the GBT facility. These contacts (e.g., probation officers) could not only identify the services received by comparison sample members but also provide access to such key outcome measures as probation violations, rearrests, and placement types. Is random assignment possible? Random assignment is possible but would require the approval of juvenile justice officials as well as program staff. Random assignment would be most feasible if the program was consistently at capacity and a waiting list were created. This is not currently the case, however, as the Newark site is still establishing referral sources and processes in the community. In the event that all beds were taken, however, random assignment might be an ethical way to assign beds to girls waiting to enter the program. No risk assessment is completed before the girls enter the program, so admitting only the most at-risk offenders is not an obstacle to random assignment in this case. Juvenile judges formally or informally assess the risk of offenders who come before them, however, and may be reluctant to turn some of their decision-making power over to random assignment. What threats to a sound evaluation are most likely to occur? A few threats to a sound evaluation were found in the assessment of GBT. First, dosage is a concern. Girls are in the program for as little as 1 month and then move either back home or to another DJJ-sponsored placement. Although all program activities are logically linked to identified and expected outcomes, the short exposure may not be enough to realize those outcomes. GBT staff recognized this dosage problem. They pointed out that a few girls may be going along with the program without really changing their problem behaviors. Once the girls are released into the environment that fostered the problem behaviors in the first place, the month or so they spent at GBT may not be enough to help them resist returning to old habits. Other evaluation threats are the sample size and target population characteristics. Both program sites are small (6-10 beds), so an adequate sample size would need to be gathered from all target population admissions over a period of time (most likely a year). Another option is to include 1 or more of the 17 other GBT programs currently in operation. As mentioned earlier, both the Newark and New Orleans sites are still establishing referral sources and procedures with key partners in the community. The Newark site in particular has not filled its allotted beds with girls from DJJ and has used some of its available beds for DYFS girls, who should not be included in an outcome evaluation of this program. The Newark staff are continually meeting with juvenile judges and other key officials to make their presence and the benefits of their program known. It is therefore likely that obstacles in the referral process will be overcome by the time an outcome evaluation gets underway. Any evaluation that incorporates retrospective data will have to document the referral processes and exclude DYFS girls from the sample. Final threats to a sound evaluation are the availability and usefulness of data to inform long-term outcomes. Local and national GBT staff document short-term outcomes thoroughly, so they would be readily accessible to an outcome evaluation. Data systems to inform long-term measures would have to be identified and accessed, however. Tracking systems would also have to be put in place to monitor the comparison sample members and the GBT girls after they are released from the program. Are there hidden strengths in the project? Although the program is short, it is intense. Program activities are in effect 24 hours a day. Furthermore, each girl is assigned a youth-care worker who works closely with her to achieve individual treatment goals. Sample contamination and dropout, a concern in many treatment program evaluations, are not particular obstacles here. Although there are no locks on the doors at the facility, staff report that no girls have run away. It appears that the girls leave the program for one of two reasons: (1) they successfully complete the program, or (2) their continued problem behavior necessitates a more secure placement. Staff also reported that no girls are readmitted to the program after their departure, so it is unlikely that a sample member will show up in the program twice. What are the sizes and characteristics of the target populations? How is the target population identified (i.e., what are eligibility criteria)? Who/what gets excluded as a target? The Newark facility admitted 64 girls to the program between July 2002 and March 2003. Some of those girls were placed by DYFS and, therefore, should not be included in an evaluation of the program. At the time of the April 2003 site visit, about half of the beds were filled with DYFS youth. Program staff reported that they expected DJJ referrals to increase and become more consistent over the next several months. The New Orleans program has 6 beds, with an average stay of 1 month, and should serve about 70 girls and young women each year. The target population is referred from the juvenile court in three New Jersey counties. The program has received requests for information from other counties and may therefore expand its referral base. Girls in the program are 11 to 18 years old; their average age is 15. Girls placed at GBT either are at risk for delinquency or have already been arrested. At- risk females are generally identified through the Family Crisis Court. These girls come from turbulent families and exhibit problem behaviors that their parents, school, or other community actors are unable to modify. Delinquent females have been arrested and sentenced by a juvenile judge. The judge may sentence them to GBT instead of a more secure detention facility. GBT is therefore a test whereby girls can avoid being sent to the juvenile detention center. Other girls and young women in the program may have completed a sentence at a more secure facility and been sent to GBT to facilitate their successful community reintegration. Have the characteristics of the target population changed over time? The characteristics of the target population are defined and monitored by GBT national headquarters and have not changed over time, nor are they expected to. The only potential change in the target population is the location from which they are referred. Currently the Newark GBT receives DJJ referrals from three counties, but may expand its referral sources to include additional counties. As referral sources and processes are solidified, the target population should more accurately reflect the expected GBT characteristics. How large would target and comparison samples be after 1 year of observation? Assuming that all 10 beds at the Newark facility and the 6 beds at the New Orleans facility are filled with girls referred by the juvenile justice authorities, sample collection over a year will result in a treatment sample size of just less than 200. The target GBT capacity is 90 percent, however, so initial sample sizes may be closer to 150. Because tracking procedures are not currently in place, it is difficult to estimate how many of those girls could be tracked accurately after 1 year of observation. What would the target population receive in a comparison sample? The target population would receive traditional placement options for female delinquent offenders. These options could include less restrictive settings, such as group homes, foster homes, and day-reporting centers, or a longer and more secure placement in the State's juvenile detention facility. It seems that none of these other placement options offer the unique GBT components--a low resident-to-staff ratio, consistent consequences for negative behaviors and reinforcement for positive ones, a family-style atmosphere, and extensive data collection on progress toward targeted behaviors. What are the shortcomings/gaps in delivering the intervention? The primary shortcoming in delivering this intervention is its short duration. The impact of a 1- or 2-month program on long-term behaviors may be difficult to determine in an outcome evaluation. However, the program delivery is intense and reinforced 24 hours a day. Reintegration into the community is also encouraged through family involvement where possible, group outings, and referrals to aftercare programs upon departure from GBT. What do recipients of the intervention think the project does? How do they assess the services received? During the Newark site visit, the assessment team was able to observe and interact with the GBT residents. Overall, the residents displayed the positive behaviors expected by the program and were participating in such activities as group meals and academics. No information regarding the residents' perceptions of the program was gathered, however. What kinds of data elements are available from existing data sources? What specific input, process, and outcome measures would they support? Youth-care workers complete a number of forms to track suicidal behaviors (as described above), other critical incidents, and progress toward individual treatment goals. When suicidal behaviors are mentioned, the youth-care worker completes a checklist that covers many indicators of suicide (e.g., trouble sleeping, acting out). Youth-care workers also complete a treatment progress checklist on their assigned residents each day. Several behaviors are tracked throughout the day, including school misbehavior, swearing and/or obscenities, lying/cheating, and defiance. Staff then assess each girl's progress toward treatment goals and set short- term goals for the following day and week. Residents also complete several data collection forms. Suicidal behavior is a particular concern at GBT. On admission, girls complete the SPS rating form to assess their feelings and behaviors. Whenever a suicidal behavior is noted, the girl is taken aside to discuss her behavior with a youth-care worker. The youth-care worker takes appropriate action and asks her to complete a Child Suicide Risk Assessment. On intake, girls complete the Computerized Diagnostic Interview Schedule for Children (C-DISC) to permit the assessment of a variety of mental and behavioral problems. Finally, the residents complete a Shelter Youth Consumer Survey that assesses their knowledge of program expectations and goals as well as how safe they feel at the facility. How complete are data records? Can you get samples? Each local program compiles a monthly report that tracks admissions, departures, length of stay, and staff turnover. Individual resident outcomes are also assessed, including treatment behaviors (lethality, aggression, sexual issues, problem avoidance, school problem behavior, and substance abuse), and client outcomes (e.g., favorable departures and moves to less- restrictive placements). Risk management indicators are also collected, such as injuries, medical errors, and use of safety holds. These indicators, as well as any mentions of suicidal behavior, are forwarded to the national office and reviewed daily. The monthly report received from the Newark GBT program appears to be complete. GBT has asked that the data contained in this document not be released to the public without its permission, however. What routine reports are produced? Daily reports track the progress of residents toward individual treatment goals and document critical incidents. The national headquarters also reports to the local sites monthly. Monthly outcomes are also routinely reported. Can target populations be followed over time? The only current obstacle to following target populations over time is the lack of a tracking mechanism. Program staff are receptive to following up with past residents and will work with an outcome evaluator to implement these tracking procedures. Program staff have also identified individuals in the three referring New Jersey counties who should be able to help an evaluator follow the comparison sample over time. Can services delivered be identified? Yes. As discussed earlier, staff tell consistent stories about program activities, and observations at the Newark facility reinforced this conclusion. Furthermore, GBT national headquarters monitors faithful implementation of services delivered. Can systems help diagnose implementation problems? GBT national headquarters already has effective processes in place to train all program staff and ensure faithful program implementation. Any implementation problems will be identified and remedied by the local staff or, if necessary, the national headquarters. GBT also conducts onsite visits and prepares assessment reports as part of its quality assurance effort. Do staff tell consistent stories about the project? Yes. No inconsistencies in program activities or target population characteristics were evident in any of the evaluability assessment activities. Are staff's backgrounds appropriate for the project's activities? GBT program staff are thoroughly trained in its teaching model. Youth- care workers must have at least a high school level education, although GBT prefers a bachelor's degree. Shift supervisors are former youth-care workers who have an advanced degree or have demonstrated exceptional ability to work with the girls. A clinical therapist, typically an M.S.W., provides group and individual therapy. Furthermore, the staff members who have the most consistent ongoing contact with the residents are demographically similar to the residents themselves and serve as facilitators in effectively modeling appropriate behavior. Shift supervisors and program directors have prior experience working with female delinquent offenders, either in other GBT programs or in other settings. What do partners provide/receive? How integral to project success are the partners? The key partners to the GBT model are within the organization itself. The national headquarters provides guidance and training, monitors implementation, provides feedback and monthly reports, and analyzes program outcomes. It is a key partner in ensuring fidelity in program implementation and consistency in data collection. Other key partners are referral sources in the community, such as juvenile court judges and probation officers. These partners are critical to program success because they identify and refer target population youth to the program. What changes is the director willing to make to support the evaluation? The program director is very supportive of an outcome evaluation. Staff are willing to work with an evaluator to put a tracking system in place so that prior residents can be followed over time. GBT has also provided contacts to help identify and track a comparison sample. Staff offered unrestricted access to program files and data for evaluation purposes. They are also willing to discuss other data collection processes or evaluation activities with an outside evaluator. CONCLUSIONS Would you recommend that the project be evaluated? Why or why not? An evaluation of this project is recommended for several reasons. First, in comparison to male juvenile offenders, female juvenile delinquency has received less attention in research and practice. Furthermore, female delinquency is increasing much faster than male delinquency, and alternatives to address this problem deserve attention in the literature. Many audiences, including funding sources and local policymakers, would also be interested in the results of a rigorous outcome evaluation of GBT. The Philadelphia program was listed in an OJJDP report of "promising practices"[9] and was the focus of a single-sample longitudinal study,[10] but the model has not yet been rigorously evaluated. The program has several characteristics that would facilitate an outcome evaluation. Program implementation fidelity is monitored by the GBT national headquarters, which has acted in this capacity since the model was first expanded in 1988. Data collection is an integral part of this process, so several outcome indicators are already in place and are consistent across multiple program sites. GBT national headquarters participated in the evaluation of its Philadelphia program and operates a national research institute for child and family studies. GBT is also interested in creating a matching database so that it can track girls participating in its programs against other delinquent female offenders. Staff also repeatedly demonstrated a willingness to participate in and support an outcome evaluation. The program's target population, activities, and outcomes are all clearly defined, as is the logic linking them together. Two program characteristics may affect its impact and cost-effectiveness, however, and should be incorporated into an evaluation of GBT. First, as mentioned previously, the treatment duration is generally 1 to 2 months but very intense. An outcome evaluation could therefore yield important evidence as to the effect of this modality versus longer but less intense treatment options. Second, cost- effectiveness should be a significant component of any outcome evaluation. The grantee has received $1,350,000 to implement programs in 2 cities that can house 16 females at any one time (serving approximately 200 girls a year if each operates at capacity). Furthermore, the low staff-to-resident ratio may be important to the program's effectiveness, but it is costly. These costs should be weighed carefully against any documented program benefits. What kinds of evaluation designs would you propose? A rigorous outcome evaluation would hinge on the two programs operating over their capacity (which is currently not the case). With a waiting list for beds, juvenile court judges could randomly assign eligible female offenders to the program. Treatment and control samples could then be tracked on treatment services received (i.e., GBT components versus traditional placement options), length of placement, and change in behavior while in placement. Each sample would then be tracked until the age of 18 to determine the program's long-term effect on their behavior. Outcomes would include rearrest, successful probation completion when indicated, permanent placement 1 year after release, completion of high school or GED equivalency, employment, and teen pregnancy. The random design described above would depend not only on the two programs increasing their referral base but also on agreement to random assignment by the juvenile bench. An evaluation design that used a matched comparison group would be more realistic. The treatment sample would be gathered from all DJJ females admitted to each GBT facility starting 6 months after startup (to allow the referral process to be finalized). Sample collection would continue until it reached 150 at each facility. This initial sample size is necessary because some sample members are expected to be "lost" during the followup period. At the same time, a comparison sample would be gathered in another, demographically similar, county in the same State that is not able to refer female delinquent offenders to the GBT program. Treatment and comparison samples would then be tracked on placement characteristics and short- and long-term outcomes, as in the experimental design described above. Sample size could be further augmented by also including one or more of the established GBT sites across the Nation. What should OJJDP's grant manager know about this project? The grant monitor received positive feedback from program staff. GBT indicated that his visit to the Newark facility helped staff to work through existing referral obstacles and to identify additional referral sources. The grant monitor is already aware that the Newark program is not yet filling all its DJJ-allocated beds but is working with national and local GBT staff to address this obstacle. Otherwise, program activities, target population, and expected outcomes are all accurately reflected by the grant application and progress report. NOTES 1. Budnick, K. and E. Shields-Fletcher, OJJDP Fact Sheet: What About Girls? Washington, DC: U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention, 1998. 2. Office of Juvenile Justice and Delinquency Prevention, OJJDP Annual Report 2000, Washington, DC: U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention, 2001, NCJ 188419. 3. See Children's Defense Fund (http://www.childrensdefense.org/ss_jjfs_girlsjj.php). 4. Office of Juvenile Justice and Delinquency Prevention, Guiding Principles for Promising Female Programming: An Inventory of Best Practices, Washington, DC: U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention, 1998, NCJ 173415. 5. See Center for Effective Collaboration and Practice (http://cecp.air.org/resources/success/boystown.htm). 6. See Children's Defense Fund. 7. Huefner, J.C., R. Xia, J.F. Teare, and J.L. Davis, Staff Secure Detention Program for Female Juvenile Offender: A Long-Term Follow-up Study, Omaha, Nebraska: Girls' and Boys' Town, undated. 8. Ibid. 9. See Office of Juvenile Justice and Delinquency Prevention, Guiding Principles for Promising Female Programming: An Inventory of Best Practice. 10. See Huefner, J.C. et al.