• By Kevin Mount
  • Posted on Thursday 16th October, 2008

Good intention, right design – “wrong” answer?

Tutoring programs for young children have been promoted widely in the US since the 1990s as promising strategies for improving academic performance, particularly in reading and maths. Bill Clinton’s America Reads Challenge, for example, urges higher education institutions to increase their commitment to volunteerism within the surrounding local communities.The benefits of tutoring provided by accredited professionals are known but the evidence for tutoring programs staffed by adult volunteers or college students is rather slimmer. A systematic review of the literature on volunteer tutoring could find only 21 articles that met the criteria, namely randomized field trials published between January 1985 and August 2005 that examined academic impacts, assessed programs for students in grades K–8 and considered effectiveness.For most the sample sizes were very small: only three included more than 100 children. So although an average effect size of 0.30 was identified for the impact on reading outcomes for elementary pupils – it is possible, based on the general findings in relation to sample size, that for larger programs with a more diffuse focus the outcomes would be less significant.This calls into question the value of large-scale university partnerships such as the West Philadelphia Tutoring Project (WPTP), which was part of the Clinton initiative and uses University of Pennsylvania students as volunteer tutors. The majority of those tutored are elementary and middle school students (between the ages of five and 12) who are are helped for around an hour a week from late-September until mid-April.Gary Ritter and Rebecca Maynard of the University of Pennsylvania undertook a randomized field trial in which pupils with low test scores from 11 West Philadelphia elementary schools who enrolled in the program were randomly assigned to the tutoring group (n=196) or the control group (n=189) for one school year (1998-1999). Several hundred tutors worked in the schools each week. The research focused on academic outcomes as measured by academic grades and standardized test scores and as well as on confidence in academic ability, motivation and school attendance.Interviews and conversations with teachers, students and tutors indicated that everyone believed that the program was having a positive impact on the participants. However, when they analysed the results using multivariate regression models, Ritter and Maynard could find no difference in any of the measured outcomes between the tutored group and the control group. Bluntly, the analysis revealed no significant relationship between WPTP participation and any of the academic outcome measures.In each case, ”improvements” were explained primarily by prior performance and, to a lesser extent, by demographic, school and grade variables. Even substantial differences in family background did not seem to alter performance trajectories: “the students were following their expected academic path, and the WPTP intervention was not strong enough to move them to a higher performance path. Participation in WPTP tutoring does not result in systematic program impacts over this one-year observation period,” Ritter and Maynard concluded.

It may not work but the kids still love it

Late in the 1998-99 school year, a veteran public school teacher in her first year as a WPTP site coordinator commented: “I do not think this program will show an improvement in grades or tests, but the kids and the tutors love it and there are real improvements in the kids that may not show up in tests – but [they] are real – and teachers notice.” Just so. But why these findings? Ritter and Maynard offer several possible reasons.One is that the program was badly design in that it failed to incorporate characteristics that were known to be effective. For example, the literature stresses the importance of tutor training, yet in most of the WPTP school sites the tutors were not expertly supervised and there were only limited opportunities for training.The relatively low dosage of tutoring is likely to have been another factor. As one female tutor put it, “I’m really glad I’m doing it. It would be nice to make more of an impact, but I guess you can’t expect too much when you only tutor an hour a week.” Once holidays and absences were taken into account, the majority of students received fewer than 20 hours of tutoring throughout the year.A further likely drawback is that the WPTP has multiple objectives and must satisfy various objectives and stakeholders. Tutors are urged not only to promote academic success but also to “be a friend” to the students and increase self-esteem. This lack of focus may have diluted the program’s impact.Every robust RCT study illuminates the important distinctions between programs that do and do not work. As Ritter and Maynard point out, “this is critically important because of the opportunity cost associated with ineffective programs: time and energy expended on ventures that do not work diminish the time and energy available for programs that may actually be effective.” Further, well-conducted evaluations can also inform designers and implementers about ways to modify and improve existing programs.On the face it, Ritter and Maynard were fortunate to be able to employ the "right" design for evaluation of impact – the so-called "gold standard" of random assignment – but unfortunate in that they found the "wrong" answer. They argue, however, that an alternative view is that there is no "wrong" answer because it is useful and important to learn that a program is not meeting its intended purpose.ReferencesRitter, G W and Maynard, R A (2008) "Using the right design to get the 'wrong' answer? Results of a random assignment evaluation of a volunteer tutoring programme", Journal of Children’s Services 3 (2), 4-16.Ritter, G W, Denny, G E, Albin, G R, Barnett, J H and Blankenship, V L (2006). The Effectiveness of Volunteer Tutoring Programmes: A Systematic Review, Campbell Collaboration Register of Interventions and Policy Evaluations.

Back to Archives