• By Laura Whybra
  • Posted on Wednesday 25th March, 2015

In implementation, “how well” is as important as “how much”

strong>Most studies of implementation look at whether facilitators cover the material in the program manual. But they tend to neglect the role of competence in delivery. A recent study of a school anti-bullying program found that teachers who taught with warmth and praise, and who covered the material clearly, were also the ones who got the best results. Bullying prevention programs in schools typically have limited impact. However, when programs monitor implementation, the effect is doubled. One reason that programs fail to deliver results, then, may be the level and quality of implementation – together known as “treatment fidelity.”A study of an anti-bullying program in two US middle schools aimed to break down treatment fidelity into four component parts, addressing different aspects of facilitator behaviors. Researchers then examined how these four types of behaviors related to students’ responsiveness.All four behaviors were linked to student engagement and willingness to follow the anti-bullying rules. It made a great deal of difference whether teachers followed 10 specified behaviors, such as praising students’ engagement and showing warmth and interest toward the students.However, when the other behaviors were taken into account, it mattered relatively little whether teachers followed the pre-agreed class meeting outlines. In other words, it seems that it mattered “how well” the program was delivered, independent of “how much” was delivered.Olweus Bullying Prevention ProgramThe Olweus Bullying Prevention Program is a violence prevention program that aims to enhance the school environment. It has three components: individual, classroom and school. This study focused on the central component, the classroom level. In weekly sessions, teachers cover a range of topics such as coping with anger, problem-solving, and stress management. The program uses role-play, small group activities, and discussions. Of 22 weekly meetings in each school year, the first six were based on the program manual and the rest were developed in collaboration with the school’s Bullying Prevention Coordination Committee, taking into consideration student and teacher feedback. In this study, researchers observed 44 teachers in two middle schools in the southeastern US. Both schools involved students in grades 6, 7, and 8 (approximately ages 12-14). Four aspects of fidelityMost studies of fidelity focus on adherence (the extent to which the program was delivered according to design). In this study, the authors also looked at quality (how competently the intervention was delivered). They divided adherence into two categories. “Procedural adherence” was the degree to which teachers conducted the program according to the pre-agreed outline of the meeting. “Instructional adherence” was a measure of whether teachers used 10 specified behaviors, such as demonstrating warmth toward students.They also divided competence into two categories. “Procedural competence” was how well teachers followed the meeting outline, and how clearly they explained the program content. “Instructional competence” was a measure of how well teachers delivered the 10 specified behaviors. They then examined the relationships between each of the four aspects of facilitator behavior and student responsiveness. Responsiveness included both the level of students’ engagement and their willingness to abide by the intervention.What matters most?The authors found that the amount of specific instructional behaviors teachers followed (instructional adherence) predicted student engagement more than how much of the meeting outline they delivered (procedural adherence). For example, if the teacher extensively demonstrated warmth and interest toward and involvement with the students, student engagement would be higher as compared to when the teacher extensively explained the purposes of class meetings. On the other hand, how well teachers followed the meeting outline (procedural competence) predicted student involvement more than how well teachers delivered instructional behavior (instructional competence). Overall, the researchers point out, delivery with competence gives a boost to program results, over and above the effect of delivery with adherence. It seems intuitively clear that quality should matter on top of quantity, and this study helps to show that this intuition is correct. For further researchThis study demonstrates that the quality of facilitators’ delivery is important in engaging participants and delivering results. Therefore, strategies to improve teachers’ practices, such as directed consultation and practice-based coaching, may improve the results of class-based interventions. Future studies should also consider how student engagement relates to teacher-student relationships, student behavior, and other outcomes such as levels of bullying. This is important because participant responsiveness is an aspect of implementation, and not the final goal. ************Reference:Goncy, E. A., Sutherland, K. S., Farrell, A. D., Sullivan, T. N., & Doyle, S. T. (2014). Measuring teacher implementation in delivery of a bullying prevention program: the impact of instructional and procedural adherence and competence on student responsiveness. Prevention Science, DOI 10.1007/s11121-014-0508-9

Back to Archives