• By Kevin Mount
  • Posted on Thursday 16th October, 2008

Trialing fidelity among the Welsh faithful

Collaboration between researchers and practitioners in Wales is shedding light on how to implement replication randomized controlled trials of evidence-based programs in mainstream policy and practice.The work focuses on the Incredible Years (IY) program, which strengthens relationships between parents and children by emphasizing the importance of play and encouraging good behavior through limit-setting and positive discipline strategies.The effectiveness of the IY parent, child and teacher series has been researched in various combinations and identified in many systematic reviews of evidence-based interventions that prevent and/or reduce violence. Programs have been delivered and researched internationally and shown to be equally useful with different cultural groups in the US and in countries such as England, Wales, Norway and Canada.In Wales, Judy Hutchings and colleagues from Bangor University have conducted three pragmatic RCTs focusing on children at risk of conduct disorder: two involved the Incredible Years (IY) BASIC parent program – one in the context of Sure Start, one with foster carers, and one involved with the IY teacher program. All three took place in regular service settings. Staff from the provider agencies delivered the program. Participants were randomly allocated to intervention or waiting-list control conditions as a basis for the evaluation.There is growing political interest in implementing evidence-based programs in this area because anti-social behavior and conduct disorder (its clinical manifestation) are absorbing an increasing amount of state funds on both sides of the Atlantic.Government and service providers need to be able to determine what qualifies as "effective". However, when proven programs with high standards of evidence are actually delivered in service settings by non-specialist staff, it is often difficult to replicate the established findings. The reasons are becoming increasingly familiar. Program developers may fail to provide the necessary tools, such as manuals, adherence checks, training and access to supervision to enable faithful replication. Even when programs include them, the people responsible for delivery may add or discard components, use insufficiently trained staff or try to make do with inadequate resources.The Bangor team found that partner services needed to have prior experience of the program and trained staff ready to deliver it. It also emerged that anecdotal feedback about success elsewhere was an important encouragement and helped to instill an understanding of the program requirements. The IY teacher study, for example, resulted from an initiative in a local authority that had already undertaken a trial. Service managers were keen to obtain more rigorous evidence of its effectiveness to justify the cost of rolling it out to all of their over 100 primary schools.Early involvement of service managers in study planning and shared management arrangements were other enabling factors. They were involved as partners in developing the bids for funding in all three Welsh studies. As a result, the projects ran with few problems.In all three studies service providers understood that they would be required to run the program twice: once for the intervention group and subsequently for the control group. Likewise, in all three studies the research funding provided additional resources, including books, session materials, leader and parent evaluation sheets, parent and teacher session handouts and parent group raffle prizes. This ensured that participants received all program components.In the Sure Start study, for instance, research funding provided lunches for parents and children, transportation costs, a cash incentive for informing the research team of a change of address (to minimize attrition) and a payment to parents for completing the research evaluations. Service providers regarded supervision as a benefit that would give them a more skilled workforce, and 21 of the 22 leaders achieved leader certification during the project.Another key to the success of the partnerships was the way the research team enhanced the fidelity of implementation. IY programs incorporate a rigorous accreditation process for program leaders. In the case of the Sure Start study, having the research team deal with all the administrative aspects of running the programs was found to be particularly helpful to teachers – for example in photocopying and supplying of handouts, folders and raffle prizes. Lack of administrative support is invariably a contributory factor when replication fails.The second set of lessons pointed to the benefits of participating in RCTs. The Bangor team point out that the trials demonstrated to service managers what was needed to ensure a high standard program delivery, and that the outcome data they generated proved useful for future service planning. The partnerships also created a pool of skilled program leaders and so increased the capacity of the organization to deliver evidence-based services.Hutchings and her colleagues conclude: “The US led the way in researching effective child mental health initiatives to prevent or reduce violence. The UK, with its better funding of child health and education services, now has the opportunity to take a lead in addressing how to deliver effective services and implement them with fidelity into the mainstream.” ReferencesHutchings J, Bywater T, Eames C & Martin P (2008) “Implementing child mental health interventions in service settings: lessons from three pragmatic randomised controlled trials in Wales”, Journal of Children’s Services 3 (2), 17-27.Hutchings J, Bywater T & Daley D (2007) “A pragmatic randomised controlled trial of a parenting intervention in Sure Start services for pre-school children at risk of developing conduct disorder: how and why did it work?”, Journal of Children’s Services 2 (2), 4-14.Hutchings J, Daley D, Jones K, Martin P, Bywater T & Gwyn R, (2007) “Early results from developing and researching the Webster Stratton Incredible Years Teacher Classroom Management Training Programme in North West Wales”, Journal of Children’s Services 2 (3), 15-26.• for more about the Incredible Years and its creator Carolyn Webster Stratton, see: An "odd couple" traveling a long road, Sure Start made more credible by success of Incredible Years and Incredible journey for one Skinner pigeon!; for more about program fidelity, see How to be sure the song remains the same

Back to Archives