• By Dartington SRU
  • Posted on Friday 18th June, 2010

Social Research Unit Annual Lecture: Steve Aos talks about picking them, passing them, and doing them

There are many levels of success in prevention science. Finding an effective evidence-based program is, to use a baseball analogy, like hitting a single. If the program is supported in the State legislature, it’s akin to a double or a triple. The true home run is getting the program fully integrated into mainstream systems. Such was the message delivered yesterday by Steve Aos at the Annual Lecture of the Social Research Unit in London, England.Aos is the Associate Director of the Washington State Institute for Public Policy (WSIPP) in the US. The Institute’s success at bringing hard-nosed economic analysis to Washington State’s investment decisions is known and respected around the world. This sophisticated approach has arguably led to the greatest density of evidence-based programs in the US. Nowhere is this truer than in Washington State’s criminal justice system. New prisons typically cost about $250 million to build and $45 million a year to operate. So in 2005 the Washington legislature asked Aos and his team to examine the economics of a whole range of crime fighting options.They found that although the boom in prison building had helped drive down the crime rate by two-fifths, the cost to the taxpayer had more than doubled. The challenge was to find alternatives that would fight crime but at a lower cost. Aos and his team took a three-step approach to find these programs. The first thing they did was look at the research. ”That’s the starting point,” Aos said. “If you don’t know whether or not it works there is no point doing an economic analysis.”Having reviewed over 500 studies, the team next turned to the economics of implementing the programs in Washington State. The WSIPP model is well respected for its consistent, conservative estimates of the economic benefits of evidence based programs both for government and individuals.“Only when we know what works, and what the costs and benefits are do we go portfolio,” observed Aos. That’s the third step. “The portfolio is the list of policies and programs that can be relied on to deliver an outcome that matters to policy makers, less crime for example.”Aos and his team delivered a portfolio for crime fighting. The result of this analysis was that money slated to build a new prison was instead redirected into programs. Millions were invested in programs like Functional Family Therapy, Multi-systemic Therapy and Nurse Family Partnership.Pre-school education is now funded by Washington, not simply because of its potential impact on children’s intellectual development, but also because its a cost-effictive way to reduce crime.WSIPP’s methods have been celebrated as a major innovation in common-sense government. Less famous, however, are the efforts of the Institute to monitor the policy changes it has recommended. The question is simple: does the portfolio deliver what the economic model said it would?The programs are currently under evaluation from a couple different angles. For instance, is the program reaching the people for whom it’s intended? Aos told the London audience “If we are going to realize some of the financial benefits we anticipate from our model we need to get the right people to the right programs. The ‘right people’ means the high risk populations, those who are candidates for expensive interventions like custody.” Aos said it isn’t enough to put the programs in place; you have to educate everyone about what they do. He talked about one finding “where judges referred non drug-offending youth to drug-offending programs.” Clearly, there were growing pains. An adjustment fixed the problem. Aos said, “Each child who gets convicted is now assessed using a psychometric tool to … ensure that they get the right evidence-based program.”Many of the programs required similar fine-tuning. Functional Family Therapy (FFF) is one of foundations of the WSIPP portfolio. Early evaluations showed that poor implementation of FFF had in some cases exacerbated problem behaviors rather than cured them. So WSIPP recommended improved assessment and supervision systems, and the problems began to resolve themselves. The state legislature also forced the hand of some providers. According to Aos, “The State Legislature said … ‘If you don’t put a fidelity system in place we will cut funding of the program’.” He said it was easy to follow which providers took the threat seriously. “My marker is, do people who aren’t delivering with fidelity get fired or re-assigned? If they don’t, then program providers probably aren’t taking fidelity seriously.”Playing with funding formulas has also been a good way to keep quality up. If agencies have clear financial incentives, they will deliver programs as intended. Local buy-in is also encouraged by giving communities some choice in what programs work best for them. Communities can choose the programs from an approved list. It wasn’t easy, Aos said, but over time the culture of government services in Washington State changed. “In the 1990s I was focusing on juvenile justice and there was great resistance to doing evidence-based programs.” Aos’s focus eventually was directed elsewhere and someone else took over his responsibilites at the Institute. When Aos returned to the issue years later, he said a seismic shift had occurred. “I attended a practitioner team meeting where they were discussing their focus and they listed assessment and evidence-based programs as the number one thing they do. I thought they were kidding me. But it was down to the persistence of the State legislators over the years.”Aos said one reason for WSIPP’s success is its non-partisan approach. “Republicans and Democrats are interested in crime reduction,” observed Aos. “And Democrats and Republicans are both interested in saving taxpayer dollars. Our Institute gives independent advice. There are no political bumper stickers on the cars of our staff.”In conclusion, Aos struck a reflective note. “I have … come to appreciate that the analytics are less than half the challenge. The public policy advice business is a ‘people’ business and we don’t expect perfect use of the information we provide. In fact I think of it as a baseball hitter might think of it. If I can be successful 35 per cent of the time I can be counted as a great hitter. That means 65 per cent of the time I will fail.”The talk generated incredible interest and was attended by a broad spectrum of service providers, government representatives, and academics. Aos and his colleague, Stephanie Lee, are currently at work with the Social Research Unit and a collective of government agencies to see if the WSIPP model could be used in the UK.

Back to Archives