• By Dartington SRU
  • Posted on Tuesday 22nd September, 2009

Bang for buck studies too often miss the mark

It is fast becoming the accepted wisdom that in order to make more skillful decisions about the value of social interventions, policy makers need better information about the financial implications of programs and policies.An overview of research carried out by the Campbell Collaboration, the acknowledged masters of the systematic review, has found that the majority of such studies do indeed talk about costs and cost benefits, but the economic components have in the main been insubstantial and methodologically weak.In an article published in the journal Evidence & Policy, Ian Shemilt and Miranda Mugford report that nearly three quarters of Campbell Collaboration reviews published in the past decade commented on economic perspectives, but only a quarter incorporated actual evidence from primary studies.Of the 31 reviews published between 2000 and 2008 they examined, one third got to grips with the issue of cost-effectiveness. A quarter calculated current levels of expenditure in the policy area at which the program was targeted, and a similar proportion took into account the economic burden of the social issue under consideration. A quarter of the reviews considered the resources - in terms of manpower, materials and organization - that would be necessary to put the program into action, and a fifth focused on the potential resource consequences – for instance, by asking whether a preventive program for juvenile offenders actually reduced the need for prison space.The Campbell Collaboration was set up to help people make informed decisions about social programs and policies by creating a world library of systematic reviews of the available evidence in education, crime and social welfare. Ian Shemilt and Miranda Mugford at the University of East Anglia, UK  are part of the Campbell Collaboration’s Economic Methods Group. As they explain, the Collaboration’s mission, first and foremost, is to establish a picture of “what works” in terms of the balance between the beneficial and harmful effects of social interventions. They hope to encourage decision-making based on evidence that is less biased and more statistically powerful than anything likely to be obtained from the evidence of a single study. However, policy makers operating in a context of limited resources and now entering a period of harsh restraint need to consider not only the balance of benefits and harm but also investment and return. Economic data will indicate that one program produces better results for the same money, or even for less money than another. And from an economic point of view, programs which may not be the best of the bunch in terms of their outcome performance can nevertheless be viewed favorably, if they achieve moderate success at an acceptable price.Despite addressing some of these economic issues and in certain cases providing evidence on them, the Campbell Collaborations reviews had many shortcomings. “They fail to meet important quality criteria for methods recommended for reviews of economics studies,” say the authors. There were also flaws in the way economic information was presented. It was not described consistently and was often distributed across different sections of reviews. Shemilt and Mugford are working on a systematic framework for collecting and analyzing the available data on intervention characteristics and resources as part of the systematic review process.They were also critical of what until recently has been the North American focus of many of the Campbell reviews. “Given the intended international audience of end users, it would be preferable if they were relevant to a range of jurisdictions and settings, including low- and middle-income countries where possible,” they argue.  In laying out the case for the inclusion of economic data in systematic reviews, the authors warn against taking this type of evidence at face value. They counsel the careful critical appraisal of evidence on cost-effectiveness. The routine randomized controlled trial of cost benefit predictions has yet to arrive in the land of prevention science, but it is a sign of the times that the University of East Anglia Institute of Health and Social Sciences Research where Shemilt and Mugford are based has just launched an MSc in Health Economics. The University Health Economics Group’s portfolio includes a role in a European study of the socio-economic impact of food allergy and the economics of chronic pain.See: Shemilt I and Mugford M (2009), “Including economic perspectives and evidence in Campbell reviews to inform policy and practice: a critical review of current approaches”, Evidence and Policy, 5, 3, pp 229-46.

Back to Archives