• By Dartington SRU
  • Posted on Thursday 22nd March, 2012

When all is not as it seems

How to make prevention strategies, perfected in the social laboratory, a reality in the real world? The All Stars is a year-long, after-school program (ASP), which aims to prevent substance abuse, and reduce bullying, violence and other behavioral problems among young people.Research, based on an independent replication evaluation using an experimental methodology, by Denise Gottfredson and a team from the University of Maryland and Rowan University, USA, reinforces what are the now well-rehearsed challenges in making such transfers. All Stars has two components. The first is the “Core," which includes 14 lessons intended to help young people shift to a healthier lifestyle, while making a public commitment to abstain from drugs. Positive parental attention and bonding with social organizations, such as community groups, are encouraged. The second component - “Plus” - includes 13 lessons designed to reinforce changes in behavior and attitudes produced by the “Core.” Young people learn to set goals and make decisions, as well as how to resist peer pressure.The program’s emphases on promoting self-control and establishing expectations for young people's behavior have been shown by meta-analyses to be the most effective strategies for reducing problem behavior. The approach makes sense.Previous evaluations of All Stars have found a positive effect on desired outcomes and mediators (factors understood to contribute to those outcomes), like parent-child communication and the child’s beliefs about the prevalence and acceptability of risk behaviors. However, until recently it has not been subject to a randomized controlled trial.In the USA, policy to emphasize academic achievement has reduced the provision of substance abuse prevention programs during the school day. Fortunately, a parallel upsurge of interest in after-school programs offers an alternative.The researchers randomly assigned 447 middle school students (aged 11-14) in five urban schools who registered for the after school program to All Stars or a control group in which students received services as usual. The sample included approximately equal proportions of boys and girls, was 70 per cent African American, and 59 per cent of students received subsidized meals at school. The average age was 12.2 years. The treatment (n=224) and control (n=223) groups were comparable in terms of these demographic characteristics and their academic performance prior to All Stars being implemented.Information on outcomes and mediators was drawn primarily from a survey which the young people completed themselves, supplemented by ratings by teachers, using validated measures comprising multiple questions. Graduate student observers and program staff provided data on the extent to which the program was implemented as intended.The authors’ sobering conclusion was: “All Stars does not appear to have been effective in preventing problem behavior or in promoting health behavior in the treatment group youth.” Results showed no difference between the treatment and control students on any of the outcomes or mediators. This is made more interesting by the fact that All Stars appears in several respected databases of effective practice.So, what went wrong? An obvious explanation that the program was not delivered properly is not borne out by the study. Although the quality of implementation was lower than in earlier efficacy trials, it was delivered with reasonably integrity to the program design in the five sites.Student exposure to All Stars lessons was lower than expected owing to low levels of attendance in the after-school programs. For example, the average hours received was 78 per cent of what the developer recommends, and only 41 per cent of the possible number of lessons was received.But analyses that took variability in quantity and quality into account showed no positive effects of the program for those young people who received a higher dosage or better quality delivery, or both.“These disappointing results suggest that ASPs may not be an ideal venue in which to provide effective prevention programming,” the authors argue. “The typical voluntary ASP does not provide the kind of consistent attendance required to achieve program effects. Furthermore, resources beyond what are typically available for ASPs in community settings will be required to achieve high quality implementation."“The youth service workers who delivered All Stars in this study were not accustomed to delivering prevention curricula. It is possible that more highly skilled personnel, more extensive training, or even more time devoted to program development may have improved the quality of program delivery in ways not directly assessed in our study and produced positive outcomes".But Gottfredson and her colleagues are unconvinced by this, and offer two other possible explanations for the negative findings. One is the poor fit of the program to the study population: “Although our sample was similar to those used in prior evaluations in terms of age, it differed in terms of race, socio-economic disadvantage, and gender.” Future studies clearly need to explore this hypothesis.Another intriguing possibility is that All Stars and similar group-based interventions may facilitate “deviance training” – this is, when young people reinforce with each other deviant comments or conduct by responding with approval and attention. The authors note that while this is unintended, the program offers “an unusual opportunity for youths to openly discuss drug use and other negative behaviors with adults without the expectation of reprisal.”There is a tentative suggestion that this could be the case judging by the evidence from the implementation observations conducted in the study. Instances of deviant talk were higher during All Stars sessions than in activities that made up the control, and teachers who implemented the program the best were also best at making it seem as if deviance is normal.Further, the students who were in the schools where the number of hours of the program was greater and a larger number of lessons were attended, were more likely than their counterparts in the control group to believe that drug use and anti-social conduct is commonplace.The lesson, the authors advise, is that even good intentions need testing: “Even programs that are designed to reduce problem behaviors and are implemented with high fidelity may produce subtle mechanisms of reinforcement which counteract the intended program effects on mediating attitudes and thus diminish the positive effects of the program.”**********Reference:Gottfredson, D. C., Cross, A., Wilson, D., Rorie, M. & Connell, N. (2010). An experimental evaluation of the All Stars prevention curriculum in a community after school setting. Prevention Science 11 (2), 142-154.

Back to Archives