National Evaluation of the YouthARTS Development Project
The national evaluation of the YouthARTS Development Project was designed as a cross-site evaluation with both process and outcome components. The process component gathered information on program implementation and operations, and the outcome component assessed the extent to which the three YouthARTS programs had immediate and long-term positive effects on the knowledge, attitudes, and behaviors of program participants. To both conserve limited evaluation resources and engage local program staff in the process, the evaluation design relied heavily on local staff and local data collectors hired by the programs to assist with much of the data collection effort.
The process component of the evaluation used a qualitative approach, including a review of program documents and interviews with staff, participants, and key stakeholders, to document program operations. The outcome component of the evaluation used a quasi-experimental design, supported by qualitative data, to determine the extent to which:
- The programs achieved the desired immediate effects on participants (e.g., increased art knowledge and improved program-related skills such as communication and cooperation).
- The programs had the desired intermediate effects on the attitudes and behaviors that affect delinquency and academic performance (e.g., healthier attitudes about drug use, increased positive peer and adult associations, improved self-esteem).
- The programs had the desired long-term impacts on juvenile delinquency and academic performance (i.e., decreased court referrals and increased academic achievement).
The goal of using a quasi-experimental design was to compare the attitudes and behaviors of program participants with those of matched comparison-group youth at multiple time points to attribute improvements in skills, attitudes, and behaviors, with some degree of confidence, to program participation. Implementing
a quasi-experimental design with these programs proved to be a significant challenge, as described below in the evaluation site summaries and the section on evaluation lessons learned.
Although some processes and instruments were tailored for each site, the overall evaluation used a common set of data collection procedures and instruments designed by the national evaluator. For the process evaluation, all interviews were conducted by the national evaluator. For the outcome evaluation, the national evaluator provided the following seven standardized instruments for use by the local data collectors:
The evaluation was conducted from fall 1996 through spring 1999. The overall evaluation design called for data to be collected before program participation (pretest), at the end of the program cycle (posttest), and for an identified period after the program cycle (14 months in Atlanta, 19 months in Portland, and 22 months in San Antonio).2 The national evaluator provided training and detailed data collection manuals to the local staff on how to use each of these data collection instruments. The national evaluator also assisted the local sites in their selection of procedures for choosing appropriate comparison groups and obtaining data from and about them.
- Art knowledge survey tailored for each site.
- Participant skills assessments to document the artists' perceptions of participants' art and program-related skills (e.g., expressing anger appropriately, communicating effectively with adults and peers, cooperating with others, participating in sessions, working on tasks from start to finish).
- Your Opinions Count (YOC) survey to measure changes in participants' self-reported attitudes and behaviors (e.g., attitudes about school, drug use, and the future; self-esteem; peer pressure).
- Participant focus group interview guide to collect supporting qualitative data about program operations and outcomes.
- Probation officer/caseworker feedback survey to obtain probation officers' and caseworkers' perceptions of program outcomes.
- Academic data form to gather academic data for participants and comparison youth.
- Court information form to record basic demographic data and court information for participants and comparison youth.
In fact, identifying comparison groups and relying on local data collectors posed the most formidable challenges for the outcome evaluation. Selecting appropriate comparison groups was difficult in each of the sites, primarily because of the need to identify youth similar to the program participants (because random assignment was not possible) and then to retain their involvement throughout the course of the study. Given the populations involved in the YouthARTS Development Project (i.e., system-involved or at-risk youth), this was also a time-consuming undertaking. Competing demands on staff time and attention, turnover, and lack of data collection experience and/or commitment to the evaluation process all affected the timeliness and quality of some of the data collected. These constraints, coupled with the relatively few participants served in the Atlanta and Portland programs, resulted in findings that, although promising for arts-based prevention programs, should be considered preliminary. The small sample sizes and lack of true comparison groups limit evaluators' ability to generalize the findings or attribute positive outcomes unequivocally to the arts program interventions. Nonetheless, the findings presented below for each program are encouraging and support the underlying theory that arts-based programs help contribute to reduced juvenile delinquency. In 1998, the Americans for the Arts published The YouthARTS Tool Kit, containing information on program planning, staff training, evaluation, and costs and resources for arts-based prevention and intervention programs.3 The Tool Kit includes a section on how to evaluate arts-based programs based on what was learned from the national evaluation.
Each of the programs is described in the following sections, with a summary presented in the table. These overviews describe each program's first year of operation, including startup activities, program goals, youth served, and program activities. They also describe the evaluation process and findings for each site, followed by important lessons learned in Atlanta, Portland, and San Antonio. Other agencies may find these lessons helpful as they evaluate their own arts-based programs for at-risk youth.
Summary of YouthARTS Programs
|Youth Arts Public Art
San Antonio, TX
|Local collaborative partners
||Fulton County Arts Council and Fulton County Juvenile Court.
||Portland Regional Arts and Culture Council and Multnomah County Division of Juvenile Justice Service.
||City of San Antonio Department of Arts and Cultural Affairs, Department of Community Initiatives, and San Antonio Independent School District.
||Truant youth ages 1416 referred by current probation counselors.
||Adjudicated youth (except those adjudicated for sex offenses) ages 1416 referred by current probation officers.
||Nonadjudicated, at-risk youth ages 1012 referred by teachers, principals, and self-referrals.
||15 youth per program.
||15 youth per unit per session (gang reduction unit, North unit, and Southeast unit).
||60 youth at each of 7 schools.
||4 sessions (812 weeks each) with the same group of participants.
||1 session per unit (12 weeks each).
||1 session (16 weeks) per school per year.
||8 hours per week during school, 25 hours per week during summer.
||6 hours per week.
||9 hours per week.
||Program director, project manager, lead artist/program coordinator, professional artists, and probation counselors.
||Program director, project manager, professional artists, and probation officers.
||Program directory, project manager, teacher liaison, professional artists, and caseworkers.
||Two-day artist training focusing on child development, conflict resolution, problem solving, and classroom management.
||Informal training of artists and probation officers focusing on program design, goals and objectives, background of participants, and rules and regulations.
||Five-day cross-training of artists and caseworkers focusing on working with at-risk youth, child management, curriculum development, and school rules and regulation.
||Arts-based afterschool and summer education and job-training program designed to serve one group of youth for 2 years.
Part of each day is spent learning art skills and producing saleable art. Projects have included furniture design and application, ceramics, mosaics, photography, drama, and computer graphics. Part of each day is spent learning entrepreneurship and planning exhibits. Students help organize exhibits at the end of each session. All proceeds support the program.
|Arts program designed to involve youth in the production and administration of a public arts project, from design to production and public exhibition.
Youth work with the artists twice per week during afterschool hours. Probation counselors are present at each session to help artists control problem behaviors.
Each session focuses on a different art medium. The media include printmaking, photography, poetry, drama, and videography.
A final exhibit or presentation designed to provide youth with recognition for their accomplishments is scheduled for the end of each session.
|Afterschool arts education program for youth at seven schools.
Each school is assigned three artists who design and implement the art activities (i.e., dance, visual arts, drama, creative writing, and storytelling). The program also provides educational field trips.
Transportation home from the program and snacks during program hours are provided.
Case management is provided by the Youth Services Division of the Community Initiatives Department. Every youth referred to the program receives a home visit from a caseworker for intake and assessment of the youth and family.
||Students receive $5 per hour during school year and $100 per week during summer vacation.
||Participants from the North and Southeast units receive time off probation or community service hours for successful completion.
Gang reduction unit participants are given $100 incentive and are required to participate.
Vocational/ entrepreneurial skills
Vocational/ entrepreneurial skills
|The YouthARTS Development Project
|| Juvenile Justice Bulletin May 2001