Evaluation Results

From the outset, the FAST program has been evaluated for quantitative outcomes, and its ongoing processes have been monitored with each new implementation. In 1990, McDonald and Billingham developed a FAST Evaluation Package to measure the outcome of the program for children and families at each new replication site of the Wisconsin statewide initiative. Evaluating the local impact of each site and monitoring the processes of the local program adaptation and implementation are ongoing FAST commitments. The data not only show the program's impact on children and families, but allow the team to assess the unique local fit and facilitate site improvements.

McDonald and Billingham's FAST Evaluation Package (1998) includes only standardized questionnaires with established validity and reliability and published norms for children and families. Teachers and parents complete these measures to evaluate the child's mental health functioning at home and at school before and after FAST.

Pretreatment, posttreatment, and followup assessments are performed for the following indicators:

  • Child mental health functioning at school (assessment by a teacher using the Quay-Peterson 1987 Revised Behavior Problem Checklist (RBPC)).

  • Child mental health functioning at home (assessment by a parent using the above instruments).

  • Family functioning (using Moos' Family Environment Scale (FES) (Moos, Insel, and Humphrey, 1974) and/or Olson's Family Adaptability and Cohesion Evaluation Scales (FACES III (Olson, Portner, and Lavee, 1987)).

  • Family social isolation (using Abidin's subscale of the Parenting Stress Inventory (Abidin, 1986)).

  • Parent involvement in schools (using Epstein's Parent Involvement Scale (Epstein, 1995)).

  • Consumer feedback and satisfaction (McDonald and Billingham, 1998).

Across hundreds of school FAST programs, assessments show high statistical significance in pretreatment-to-posttreatment improvements on the conduct disorder scale, the anxiety-withdrawal scale, and the attention span problem scale of the RBPC. The improvements on these scales have been correlated in several studies with reduced violence and substance abuse in adolescents. The FAST Evaluation Package has been used in more than 300 schools and communities, and the improvements are predictable and consistent.

Replication Evaluation Data in Two Statewide FAST Initiatives

A Cambodian family, in ceremonial attire, graduating from FAST in Wisconsin.
A Cambodian family, in ceremonial attire,
graduating from FAST in Wisconsin.
Outcome evaluation data were collected from statewide FAST replications by McDonald and Billingham at 30 Wisconsin schools using antidrug funds (Billingham, 1993; McDonald, 1993) (see tables 3 and 4) and by the State of California at 12 California schools using Office of Child Abuse Prevention funds (see tables 5 and 6). Data from both statewide implementations showed high statistical significance (p<0.01) in improvements on the five measures described above using paired, two-tailed t-tests (these tests indicate whether the improvements are due to chance or to the program). Parents reported 25-percent improvement at home, and teachers reported 20-percent improvement at school after only 8 weeks. Reductions occurred in several categories of problems:

  • Behavior problems, such as bullying, hitting, stealing, and lying.

  • Withdrawal and anxiety, such as insecurity and social isolation.

  • Attention span problems, such as lack of focus and distraction.

FAST Evaluation Across Time in Madison, WI, With Comparison Groups

CSAP funded evaluations of the long-term impact of FAST in Madison, WI, with outside evaluator Thomas Sayger, Ph.D., of the University of Memphis, TN. The CSAP evaluations used several measures, including Achenbach's CBC and Moos' Family Environment Scales. These measures showed statistically significant pretest-to-posttreatment improvements, and gains were maintained at the 6-month followup evaluation (Sayger, 1996; McDonald and Sayger, 1998).

In addition, a complete followup study of all FAST families in Madison surveyed the improvement in child functioning; parents reported that gains were maintained 2 to 4 years later. Using comparison groups of other Title I children in the Madison Schools, the followup study determined that participating in FAST helped children improve their third-grade reading scores. Based on a 2-year followup of 250 FAST families in Madison, the improved functioning of the child, the improved family cohesiveness, and the increased social involvement of FAST parents in their children's schools and in the community seem to be long-term impacts of the FAST program.

Participation, completion, and eventual leadership in the ongoing FASTWORKS programs are characteristic of low-income family participants. In the CSAP long-term impact study, McDonald and colleagues (1997) talked to 10 FAST parents in open-ended interviews and transcribed the interviews for qualitative analysis to better understand the process of change. Parents were asked to discuss and rate their experiences in FAST using the McDonald/Billingham followup questionnaire. Qualitative reports by parents and children were enthusiastic. Teachers, administrators, and school social workers/counselors were also positive in their evaluation of FAST's impact on increased parent involvement and bonding between families and schools.

Experimental Studies on FAST

In an experimental study by Billingham (1993), outcomes were statistically significant: FAST youth improved more than controls (p<0.05). Five experimental studies of FAST with special populations that use randomized trials are being funded by Federal research institutes. Three of these studies have McDonald as the coprincipal investigator in collaboration with Thomas Kratochwill, Ph.D., and Joel Levin, Ph.D., of the University of Wisconsin-Madison School of Education; Paul Moberg, Ph.D., Director, University of Wisconsin-Madison School of Medicine, Center for Health Policy and Program Evaluation; and Holly Youngbear-Tibbits, Ph.D., College of the Menominee Nation. The first study is funded by the U.S. Department of Education, Office of Education Research and Improvement (OERI), through the Institute of At-Risk Students, to study FAST with three Indian nations. The second study is funded by the U.S. Department of Education, Office of Special Education and Rehabilitation Services (OSERS), to study FAST as a strategy to reduce referrals to special education for emotional disabilities. The third study, funded by the National Institute on Drug Abuse (NIDA) and supplemented by the Office of National Drug Control Policy (ONDCP), studies cultural adaptations of FAST at inner-city schools with predominantly African American and Hispanic populations. In addition, Phil Leaf, Ph.D., of Johns Hopkins University School of Public Health, Center for Prevention Research, is conducting research on the Baltimore Head Start FAST program in Baltimore, MD. The Baltimore study is funded by the U.S. Department of Health and Human Services (HHS), Substance Abuse and Mental Health Services Administration (SAMHSA). Jean Layzer of Abt Associates, Inc., and Lynn Kagan of Yale University are conducting research in 10 schools in New Orleans, LA, funded by HHS, Administration for Children, Youth, and Families, to determine the impact of FAST as a theoretically grounded family support program. OERI is also funding a study which includes the FAST program and its impact on the development of social capital in three inner-city schools in Chicago, IL. Dr. Tony Bryck, University of Chicago, and Metropolitan Family Services are involved.

Thirteen-State Site Evaluation of Children's Mental Health in FAST Schools

FAST has been implemented in many new settings with team training by certified trainers. The FAST Outcome Evaluation Package is used with each new implementation. The mental health scores of children ages 6 to 12 on scales related to conduct disorder, anxiety/withdrawal, and attention span problems are of particular interest to juvenile justice professionals (see figures 1 and 2). High scores on "conduct disorder" correlate with delinquency and incarceration; high scores on "anxiety/withdrawal" correlate with alcohol and drug addiction; and a combined high score on "conduct disorder" and "anxiety/withdrawal" correlates with violence. High scores for "attention span problems" correlate with dropping out of school (high scores also indicate the problems are severe).

Figure 1

Pre- and post-FAST data were collected on children's mental health (using RBPC's) for the first 53 trained FAST sites of the Alliance National Dissemination Initiative.1 The data assessed the impact of FAST on more than 420 FAST children at 53 sites in 13 States (1 site did not collect teacher data). Outcomes are summarized in figures 1 and 2 (McDonald, Pugh, and Alexander, 1996). Of the children evaluated, 50 percent were European American, 23 percent were African American, 25 percent were Hispanic, 1 percent were Asian American, and 1 percent were American Indian. Thirty-four percent were female and 66 percent were male. The age range of 70 percent was from 6 to 8 years. As these figures show, the average child being referred to FAST in these schools was not just at risk, he or she was already in serious trouble.2 These data also indicate that most of the children referred to FAST across 13 States began with severe problems, as measured by both teachers and parents using a standardized scale. In only 8-10 weeks of multifamily programming, the average severity of conduct disorders, anxiety/withdrawal, and attention span problems dropped significantly, from the clinically severe to the at-risk level. In other words, parents and teachers observed an improvement of 20 to 25 percent in the behavior of FAST children at home and at school in just 8-10 weeks, shifting the average score closer to normal functioning for that age.

Figure 2

  1. Between 1993 and 1995.
  2. These data were collected by the Alliance for Children and Families with McDonald's consultation (1993-98). The Alliance used the McDonald-Billingham FAST Evaluation Package with Alliance-member family countseling agencies that were initiating FAST programs and whose training was funded by DeWitt-Wallace Reader's Digest Foundation.

Families and Schools Together: Building Relationships Juvenile Justice Bulletin   ·  November 1999