• By Michael Little
  • Posted on Thursday 18th June, 2009

Twenty-one things I learned in Washington State

I recently participated on a study tour to Washington State with a group of policy makers, practitioners and investors in children from England, Ireland and the US. Each day I tried to summarize two or three lessons that might be helpful to me and to colleagues as we go about the task of improving outcomes for children.History1. We were reminded several times about how far we have come, and how far there is to travel. When most of the originators of evidence-based programs started their work they did so against the backdrop of evidence that ‘nothing works’ (Douglas Lipton’s 1974 book is the most commonly cited source). We now have lots of evidence about programs that work, but struggle to get those programs into the mainstream.Definitions2. There was a lot of discussion among the study tourists about what we mean by ‘prevention’. For some, the emphasis was breaking into the inter-generational continuities that see a child achieving the same poor outcomes as the parent who has followed in the footsteps of the grandparent. For others, the focus is breaking into the chains of effects that produce poor outcomes in any generation and at any stage of development. The group decided to work on a common definition; this was considered fundamental to efforts to reform children’s services, many of which are plagued by misunderstanding and miscommunication.3. The critical importance of children’s developmental stage came through strongly. All of the proven models we saw in Washington State operated on processes specific to children’s development in the early years, in early school years, adolescence et cetera. It is such an obvious point but so easily forgotten in the sturm und drang of policy, management and practice.4. Recognizing that there are things we can do for all children, including children in need, is another bit of common sense that requires a tour of proven models to properly understand. Improving, for example, the behavior of a typical child can, if done well, reduce the incidence of conduct disorder. This is what we call public health prevention. That is not to say that programs targeted at children with impairments to their health and development are redundant. But a sole focus on the ‘tail’ of the distribution of need carries with it all kinds of handicaps.5. It is now politically correct to be evidence-based, so it is necessary to define this term also. The definition below comes from the Washington State Institute for Public Policy, and it refers to evidence-based programs. They include interventions proven by experimental methods with the results replicated several times over, ideally independent of the program originator. By this definition, few children’s services interventions are ‘evidenced-based’.The phrase “evidence-based” is sometimes used loosely in policy discussions. When the Institute is asked to conduct an evidence-based review, we follow a number of steps to ensure a consistent and rigorous definition.These criteria include:i) We consider all available studies we can locate on a topic rather than selecting only a few studies; that is, we do not “cherry pick” the studies to include in our reviews. We then use formal statistical hypothesis testing procedures—meta-analysis—to determine what the weight of the evidence tells us about effectiveness.ii) To be included in our review, we require that an evaluation’s research design include control or comparison groups. Random assignment studies are preferred, but we allow quasi-experimental or non-experimental studies when the comparison group is well-matched to the treatment group, or when adequate statistical procedures are employed to guard against selection bias. Given the expected direction of selection biases, we discount the findings of less-than-randomized comparison-group trials by a uniform percentage.iii) We prefer evaluation studies that use “real world” samples from actual programs in the field. Evaluations of so-called “model” or “efficacy” programs are included in our reviews, but we discount the effects from these types of studies. Our presumption is that it is difficult to achieve, in actual large-scale operation, the results of model programs; hence, when we conduct our cost-benefit analyses, we discount the statistical results of such studies by a fixed amount. iv) If the researcher of an evaluation is also the developer of the program, we discount the results from the study. Sometimes it is difficult to duplicate the results achieved by highly motivated individuals who originate programs. There may also be potential conflicts of interest if developers evaluate their own programs.6. We are getting much smarter now about how to quantify the amount of impact on children’s health and development attributable to evidence-based programs. The standard rubric is an effect size calculation. An effect size of zero indicates no impact, and prevention scientists begin to get over-excited when it goes as high as 0.6 or 0.7 (as happens with some social and emotional regulation programs delivered as part of the standard curriculum in schools for example). That said, efficient programs generating effect sizes of 0.2 or 0.3 are not to be sniffed at. Policy makers will be dismayed how often the effect size is negative, indicating we are making children’s lives worse not better. Steve Aos at the Washington State Institute for Public Policy has translated many effect sizes in monetary terms, so we know whether the benefits of intervention are exceeding the costs. This science is producing a new challenge for policy makers. Some effective programs are not cost-effective (which is not to say we shouldn’t implement them but….)Building a Portfolio7. Many of our preconceptions about children’s development are plain wrong. For example, most epidemiological studies of children’s emotional and behavioral health indicate that problems are greatest in the first years of primary school. Children are the center of their universe and then they get sent to school where they find themselves orbiting with hundreds of others in a new and strange cosmos. Hardly surprising that they act out. Where do we imagine behavior problems to be highest? Where do we invest most of our school-based behavior improvement resources? In late adolescence. Seems sensible until you look at the evidence.8. Investors in children have to be smart about leverage. To be effective evidence-based programs need to reach the many, and the most efficient route to the many is via mainstream systems. Take a typical community of 5,000 children. Let's say an investor has £1 million to spend. Quite a lot of money, but as it turns out just £200 per child. Lets do the sums another way. Let's reckon that the local authority spends about £5,000 per child per annum, which typically they do. That’s a fat £25 million. And let's suppose that it’s a disadvantaged community and parents are spending another £2,500 per child per annum (typically they too would invest £5,000 per child). That’s another £12.5 million. If the investors one million can change how the £25 million is spent or how the £12.5 million is spent then the impact on children’s lives can be much greater.9. It follows that getting rid of ineffective services is just as important as introducing evidence-based programs. We love to add, but we don’t like to take away. In fact the science of taking away is far less developed than the science of adding. The cost-benefit data helps. In Birmingham, for example, each new evidence-based program is evaluated by experiment that demonstrates costs and cashable benefits to the local authority. A process called ‘benefit realization’ then requires the local authority to identify which resources can be safely cut as a result of the improvements in child outcomes resulting from the evidence-based programs. In Washington one ‘realization’ of the ‘benefit’ was the decision to build one less prison.10. (It takes time to get simple descriptions from program developers but when everything is stripped away…) what is so impressive about evidence based-programs is not only do they have an impact on children’s health and development but that it is clear why they work. It is the logic that underpins these interventions as well as the evidence that they work that sets them apart.11. If programs are not delivered with fidelity they do not have the desired impact. If clinicians do not deliver evidence-based programs in the way that they are intended to be delivered they can do more harm than good. (It would be easier to say that implementing 80% of an evidence-based program does not achieve 80% of the results it achieves none of the results, but also Gil Botvin has worked out that 70% of his Life Skills Training curriculum gets the desired effect). The most sobering moment of the tour for me was the slide prepared by Robert Barnoski (which can be downloaded at the foot of this page) that compares the performance of Masters level social workers additionally trained to deliver Functional Family Therapy. The ‘non-competent’ practitioners are those who don’t deliver the program as intended. The ‘competent’ practitioners have got the hang of it. It’s all very intuitive. One practitioner not following the model does a great job; he or she would probably achieve good results in any context. One practitioner who does everything by the book gets poor results. Probably nothing can be done to improve his or her practice. In general, the best results come from those who follow the model. But here is the sobering bit. The least competent practitioners, those who do their own thing, not only fail to improve children’s health and development, they make it worse than it would otherwise have been without the intervention.12. Systems have to be made ready for programs, and programs have to be made ready for systems. We could dress this up in fancy terms like ecological validity (does the program fit into the real world) or you could plough through the 8,000-word essay I have just written on this subject for the Annie E. Casey Foundation. But easy to say that, with exceptions, systems don’t ‘do’ programs and evidence-based programs are usually designed and delivered outside of the systems that routinely support our children. This is something we can change and need to change.13. When building a portfolio that is based on cost-benefit analysis it is important to think in terms of volume and spread. A program like Nurse Family Partnership is going to cost me $9,118 per family, but I can bank on a return of $26,298, meaning my country is $17,000 better off each time someone is enrolled. The aforementioned Life Skills Training costs me much less ($29 per child) and I get much less back ($746) producing a cost-benefit of just over $700. Measly by comparison. But wait a minute. I know in Birmingham - which I keep mentioning because I know it well - I will be lucky to get 200 families onto Nurse Family Partnership because it's a narrowly targeted prevention program. That’s still going to reap me $3.4m for each year of investment. I can probably get Life Skills Training to about 60,000 kids (that’s two year groups) every year netting me about a million for every year I run it. Now the comparison looks a little more bearable.14. The things we like don’t necessarily work. We love Family Group Conferences. It helps families understand and resolve their own problems. How good is that! And it was designed in New Zealand that is almost a recommendation in itself. And it has been tried with Maoris, so there is the ethnic dimension taken care of. But wait a minute. When evaluated in a randomized control trial with child protection cases it makes matters worse not better. That is extremely inconvenient for us because we like Family Group Conferences, don’t we?Translating evidence to practice15. We saw lots of good methods for connecting communities and systems with evidence-based practice. These are much needed in the UK. David Hawkins uses the work ‘operating systems’ to describe things like Communities that Care and Common Language. There are others like Getting to Outcomes and, in a way, Results-Based Accountability could be added to the list. To David, the operating system is the platform that allows the programs to work. It is the Windows or Mac OS X that gives us access to Excel, Word and PowerPoint. To me it is a shared way of thinking that helps to forge a common purpose and innovation. A good operating system connects people who can help improve the lives of children; and it ensures that they are informed by high quality evidence.16. We also heard the term ‘technical assistance’ many times. We need more of this also. There are skills in knowing about evidence-based programs; who they are for; how much kids benefit; how they work; how much they cost; what training is needed; how to adapt them to meet local conditions; how to fund them; how to support practitioners when they are not working; how to get the politicians onside. Most UK systems don’t bother with technical assistance. But then again they don’t bother to find out if their mangled versions of evidence-based programs make any difference to children’s lives. So no need to worry on that count.17. The well-being of children differs markedly between schools, between neighborhoods and between jurisdictions. These differences do not get picked up by existing census or administrative data, or not least in a way that assists with the targeting of evidence-based programs. Good epidemiology is needed. It was funny that the Washington people got so concerned with a level of monthly alcohol intake that most English people thought normal for a Friday night out, but at least they knew (from the Communities that Care survey). The Together 4 All communities knowledge about conduct and emotional disorders (from the Common Language epidemiology) was fundamental to their choice of evidence-based programs and should be instrumental to any success they are able to achieve.18. There are lots of databases of evidence-based practice, when really we need one (and hopefully funding is being found for just such an entity). But in the meantime I promised to circulate currently publicly available resources.Prevention Action Blueprints Promising Practices Network Coalition of Evidenced-Based Policy National Registry of Evidence-Based Programs and Practices What Works Clearing House Best Evidence Encyclopedia Into the Future19. Better outcomes at zero net cost. Steve Aos’s work is a stepping stone. Now we are having conversations about social impact bonds, municipal bonds, benefit realization …..20. We need a place for the resources to support this work. The databases. Good epidemiological tools. Operating systems. Service design methods. Masters training for prevention specialists. Technical knowledge on the programs. The cost-benefit algorithm …..21. And if you are in any doubt that there is a future in this work:Read the National Research Council and Institute of Medicine of the National Academies, Preventing Mental, Emotional and Behavioral Disorders Among Young People, The National Academies Press, Washington, D.C., 2009. It not light bed time reading but it what we use to ensure we know a little bit more than the people we are talking to.Go to the Society for Prevention Research conference every year in the United States. If you are more practice minded, go to the Blueprints for Violence Prevention conference every other year (next one in 2010) in Denver. 1,000 policy makers, practitioners and scientists in one glorious conference.If the budget won’t stretch to Denver, come to London and hear Del Elliott, founder and director of Blueprints, deliver the annual Social Research Unit lecture at the Commonwealth Club on Northumberland Avenue on 2nd July at 5.30pm. It’s free, and you even get a free drink and food. And if you won’t pay the rail or tube fare to see Del, read all about it on Prevention Action. Its free, your children will be happier and healthier ….

Back to Blogs