• By Dartington SRU
  • Posted on Monday 16th August, 2010

Lessons from Public Health: Coming to the right conclusion from the wrong results

In the late eighties, many US children with mental health problems received inappropriate services or no services at all. Where services were available, it was believed their effectiveness was compromised by a lack of coordination. So Lenore Behar in North Carolina’s Department of Human Resources persuaded Congress to fund a study to evaluate the ‘system of care’ concept. That study, funded through the Department of the Army, became the Fort Bragg Demonstration Project. Len Bickman at the Center for Mental Health Policy at Vanderbilt University oversaw it.The Fort Bragg Project was designed to improve mental health outcomes for children and adolescents who were referred for treatment. It cost $94 million and made a range of mental health and substance abuse services accessible to over 42,000 child dependants of military personnel. The services formed part of an integrated system of care, which included outpatient therapy, day-treatment, in-home counseling, therapeutic foster homes, specialized group homes, 24-hour crisis management services, and acute hospitalization. The services were provided in civilian facilities.Bickman used a quasi-experimental design involving nearly 1000 families. It started in 1990. He published the results in 1995.The study developed a program theory with project staff. It was predicted, for example, that a single point of entry to services and prompt intake would lead to better access, more clients and increased client satisfaction. Standardised multi-disciplinary assessment was presumed likely to improve diagnosis and treatment planning. Offering a continuum of care was deemed likely to help better match needs and services. Collectively, the hypothesis went, these innovations would contribute to ultimate outcomes such as improved child mental health and lower costs per case.The results were intriguing. Bickman and his colleagues concluded that the system of care model was properly implemented. Providers maintained fidelity to the original model and delivered it with quality. There was increased access for children who needed services, and they were treated in less restrictive environments. Parents and adolescents reported being more satisfied. So far so good.But costs per client were higher in the Fort Bragg demonstration site compared with the control sites, and, critically, there was no difference in outcomes. The added money and the continuum of care was not improving the children’s mental health. The study caused a firestorm. Critics attempted to identify problems in data collection and attacked the quasi-experimental design as spoiling the results. But Bickman’s design and methods withstood the attention. Indeed, the study received numerous accolades.Even so, anxious to test what they had found, Bickman and his team replicated the study in Stark County, Ohio with a different population and system of care. Stark County’s clients were non-military, lower income youth. The system of care was a full, mature system more typical of reform efforts in many communities. The results were repeated. An improved continuum of care did nothing to improve children’s mental health. For the Vanderbilt University team, there was only one plausible explanation. No one could demonstrate that the evaluation was flawed. Implementation was fine. So perhaps the theory underlying the continuum model was wrong?Crucially, the key assumption from the outset was that the services themselves worked. If treatments were effective, there should be a dose-response relationship: more treatment equals better client outcomes. Yet, in both studies the amount of treatment did not affect outcomes. Supported by other reviews of community-based treatment, the evaluators concluded that there was no evidence that the treatments as provided in the real world were effective for children and adolescents. The lesson Bickman and his colleagues took from both Fort Bragg and Ohio was that instead of focusing on system-level issues, policy-makers should focus on improving the training of treatment providers. Ultimately, this should improve the quality of services delivered. Since then, the Vanderbilt team has worked to help make this happen, as tomorrow’s article in Prevention Action will show.ReferencesBickman, L., Guthrie, P. R., Foster, E. M. et al. (1995) Evaluating Managed Mental Health Services: The Fort Bragg Experiment. New York: Plenum.Bickman, L. & Fitzpatrick, J. L. (2002) Evaluation of the Ft. Bragg and Stark County systems of care for children and adolescents: a dialogue with Len Bickman. American Journal of Evaluation, 23 (1), 69-80.Bickman, L., Lambert, E. W., Andrade, A. R. & Penaloza, R. (2000) The Fort Bragg continuum of care for children and adolescents: mental health outcomes over 5 Years. Journal of Consulting and Clinical Psychology, 68 (4), 710-716.Bickman, L., Noser, K. & Summerfelt, W. T. (2000) Long term effects of a system of care on children and adolescents. The Journal of Behavioral Health Services and Research, 26 (2), 185-202.

Back to Archives