Author ORCID Identifier
0000-0001-9638-2223
Defense Date
2026
Document Type
Thesis
Degree Name
Master of Science
Department
Psychology
First Advisor
Bryce D. McLeod
Second Advisor
Kevin Sutherland
Third Advisor
Michael Southam-Gerow
Abstract
Treatment integrity is the degree to which a program and its practices are delivered as intended. Treatment integrity measures are critical for evaluating the effectiveness of evidence-based programs (EBPs) in effectiveness research, but few are designed for research conducted in community settings, such as schools. The Treatment Integrity Instrument for Elementary School Classrooms – Observer Report (TIES-O) is an observational measure used to assess the treatment integrity of teachers implementing EBPs in their classrooms. This study evaluated the psychometric properties of the TIES-O, which captures Adherence (frequency and thoroughness of delivery) Competence (skillfulness of delivery), and Differentiation (the extent to which practices not part of an EBP are delivered) of teacher-delivered practices that support elementary school students’ social, emotional, and behavioral functioning. Data from a randomized controlled trial that evaluated the effectiveness of BEST in CLASS – Elementary, a 14-week Tier 2 classroom-based EBP that trains and coaches teachers to support students with unwanted problem behaviors, were used to conduct a psychometric evaluation of the TIES-O. One hundred fourteen K-3rd grade elementary school teachers (51.7% aged 18-35; 96.5% female; 70.2% White) and 237 students (M age = 7.00 years, SD = 1.16; 70.9% male; 43.9% African American/Black) were included in the present study. Teachers were randomized to either the BEST in CLASS – Elementary condition or business as usual (BAU). Trained coders observed teachers during instructional periods and rated their Adherence and Competence for each TIES-O item. For interrater reliability, based on double-coded observations (45% of the sample), the mean single-measure intraclass correlation (ICC [2,1]) ranged from “poor” to “good” for Adherence and Competence items, with the majority falling in the “poor” range. Current findings also suggested initial evidence of convergent, divergent, and discriminative validity at the item- and scale-level. The magnitude and strength of corresponding Adherence and Competence items and scales demonstrated large correlations, while correlations between Differentiation items and scales (e.g., student responsiveness, disruptive behavior, and other non-BEST in CLASS – Elementary practices) were mostly in the “small” to “medium” range, indicating that TIES-O Adherence and Competence items and scales capture what they purport to measure. Further, results provide evidence for discriminative validity, as teachers trained in BEST in CLASS – Elementary practices scored significantly higher on the TIES-O Adherence and Competence scales than those in the BAU condition at the end of training. The TIES-O demonstrates favorable characteristics for use in effectiveness research. However, future research will need to address the poor interrater reliability and use item response theory to inform improvements to the item design.
Rights
© The Author
Is Part Of
VCU University Archives
Is Part Of
VCU Theses and Dissertations
Date of Submission
3-31-2026