The results of this pilot study of an observation-based implementation assessment scale, SIOF, of staff competence delivering a multifaceted intervention show initial promise. The scale demonstrated reasonable psychometric properties. From a practitioner perspective, the SIOF appears to offer a promising approach for assessing staff implementation of a complex treatment model in residential care. The instrument allowed administrators the ability to examine the item-level implementation of individual staff members in order to identify the specific training needs within each subscale, allowing a more fine-tuned approach to staff development efforts. The tool and protocol combined help staff meet three goals: ensure accountability and promote public safety through enhancing youths’ moral development; provide mechanisms for youth to develop a healthy identity; and connect youth to community, family, and peers. The SIOF was developed over the course of 5 years by a variety of program experts from several departments at Girls and Boys Town, including staff training, staff evaluation, research, and residential services. Future research is needed to replicate these results, determine the relationship between more intricate implementation patterns of the intervention, examine changes in staff implementation competence over time, and investigate the relationship between staff implementation and youth outcomes. Data were collected from 92 staff couples from 11 different Girls and Boys Town locations throughout the United States. Tables, figure, references