• By Dartington SRU
  • Posted on Friday 10th August, 2012

Shining light into the black box

Randomized controlled trials have many benefits. These experiments, when done well, leave little doubt over a program’s or treatment’s effectiveness. But many randomized controlled trials suffer from a common limitation: they treat evaluation like a “black box”, carefully monitoring what goes in and what comes out of an intervention, but overlooking the active ingredients and processes that bring about change. All good evidence-based programs (EBPs) have a logic model or theory of change that links the activities of the program to the outcomes it hopes to bring about (see “logic model”). A bundle of activities may be included in a program because developers speculate they will have certain effects. However, without teasing apart the individual impact of each of these activities on the collective change demonstrated in a randomized controlled trial, their effects can only remain speculative. When an intervention works, the “black box” means that it’s impossible to tell what parts are most important to keep faithfully (see “fidelity”) and which parts can be tinkered with. More importantly, when an intervention doesn’t work, a randomized controlled trial often can’t tell researchers why. Is it because part or all of the logic model didn’t bring about the expected change – or worse, does one or more of the activities have damaging effects?Therefore, without this crucial information, those developing evidence-based programs are flying blind, having to use expensive trial-and-error to refine an intervention or to understand why a ineffective program is not working. Finding the mediators: the light for the black boxThis needn’t be the case, as demonstrated by David DeGarmo and colleagues at the Oregon Social Learning Center. In a randomized controlled trial of the “Linking the Interests of Families and Teachers (LIFT)” prevention program, the researchers outline how mediational analysis can be used to shine a light into the black box, and to show how such programs bring about change. Mediators are the factors that link an input (such as an intervention) to outcomes. Because it takes time for interventions to lead from one mediator to another and finally to long-term outcomes, mediators have to be measured over time – and time is one of the things that randomized controlled trials (RCTs) have on their side. Individuals in an RCT are often followed up over a long time. Studies that follow individuals in the course of their development are very good at assessing the causal processes that lead to an outcome. As DeGarmo and colleagues point out, when researchers collect the right data at the right time, an RCT can be used to assess the causal process of program activities. The research team set out to demonstrate the value of good follow-up data using LIFT as a test case. The LIFT prevention program reduces elementary-age children’s anti-social behavior, and bolsters their pro-social behavior. The intervention aims to create change by positively influencing interactions in important social arenas – namely, between parents and youth at home, and between children and their peers in the classroom and the playground. LIFT therefore has several activities: a classroom problem solving and social skills component, a “Good Behaviour Game” to modify playground behavior, and group parent training. LIFT has already been shown in RCTs to improve behavior. DeGarmo and colleagues were primarily interested in the reduction in tobacco, alcohol, and drug use. Therefore, they set out a theory of change, suggesting that both aggression and family problem-solving skills would be important mediators between the program and the outcome. In other words, they believed that LIFT prevention program would improve both problem-solving and aggression. Better problem-solving and lower aggression levels would, in turn, reduce substance use. These were the “active ingredients” of the program the research team wished to test. Testing their speculationsOnce DeGarmo and his team had outlined their theory of how LIFT would improve substance use, and collected the right data, they went about statistically testing their speculations. First, as in a typical RCT, they tested whether the intervention influenced the initiation or growth of tobacco, alcohol, and drug use.But then they took a less typical step. For the substance use outcomes that were improved by LIFT, they statistically teased apart how this happened – and found that the mediators were different for each of three outcomes. First was LIFT’s impact on the growth of tobacco use: the researchers found that the intervention improved family problem-solving and reduced aggression, which both reduced the growth in tobacco use over time. Second was the reduction in growth of alcohol use, which was mediated by better family problem-solving. And finally, the growth in use of illicit drugs was improved by LIFT’s impact on playground aggression. It was clear that for different substance use outcomes, LIFT brought about changes in different ways. Unfortunately, the relatively small number of children in the trial was reduced by losing subjects at follow-up, limiting the statistical power of the study, and the effect sizes were small. Nethertheless, DeGarmo’s evaluation demonstrates that for interventions with well-defined logic models, randomized controlled trials can shine light inside the black box – not only demonstrating if a program can affect outcomes, but also allowing researchers to understand how this is being achieved. *********Reference:DeGarmo, D.S., Eddy, J.M., Reid, J.B., & Fetrow, R.A. (2009). Evaluating Mediators of the Impact of the Linking the Interests of Families and Teachers (LIFT) Multimodal Preventive Intervention of Substance Use Initiation and Growth Across Adolescence. Journal of Prevention Science, 10, 208-220.

Back to Archives