The Problems With Evidence-Based Psychotherapy

There has been a tremendous movement toward evidence-based treatment in clinical psychology over the past decade. On its face, this is a good thing — the idea that we should use scientific findings to make sure the types of treatment we’re using in psychotherapy actually work. My own clinical training included a wide array of these empirically based treatments, and I happily use many of their key techniques. I also continue to pay close attention to new clinical research, and frequently review the literature in the course of formulating treatments for my patients.

However, I see some really serious problems with evidence-based psychotherapy, problems that I believe are actually harmful to the patients, and to the credibility of clinical psychology:

The studies use unrealistic exclusion criteria

In an effort to eliminate confounding variables that could interfere with study results, research psychologists try to exclude patients suffering from any ailments outside of the single diagnosis that the treatment protocol is designed for. This is in line with the medical model of psychology, which attempts to isolate discrete psychological disease processes and treat them directly, much the same way that a physician might take a throat culture and then prescribe antibiotics to treat strep throat.

The problem is that psychological issues don’t really work that way. The diagnoses used in clinical psychology are generally not isolated disease processes the way viral or bacterial infections are. Most people don’t present with a single discrete disorder, but rather a spectrum of symptoms, personality characteristics, and social factors which can all be considered a part of the syndrome which has led them to seek treatment. This requires holistic treatment that identifies and addresses the underlying causes.

In many ways, mental illness is similar to metabolic syndrome — another problem for which the medical model is proving inadequate. All day long I see patients with separate diagnoses of diabetes, hypertension, hyperlipidemia, cardiovascular disease, obesity, sleep apnea. Each one of these identified problems are then treated by suppressing the symptoms. For diabetes there are pills to lower blood sugar, for apnea there is the CPAP device, for high blood pressure there are pills to lower the blood pressure. Even the preventive advice that is rendered is symptom-focused: switch to whole grains so the sugar will enter your system more slowly, take less salt to reduce blood volume. But all of the symptoms are actually related. They all result from the same underlying problem of metabolic dysregulation, and until that total syndrome is addressed, all of the symptoms will continue to get worse over time no matter how well they are managed.

Mental illness is like that. There are genetic, social, and cultural predispositions that contribute, but in order for the syndrome to be expressed a disruption in the management of psychological resources — such as emotions, thoughts, internal imagery, relationships, etc. — must take place. A cascade of compensatory actions are then set in motion, resulting in the symptoms we see on the surface: depression, anxiety, nightmares, panic attacks, hallucinations, substance abuse, etc.

I’ve treated substance abuse, and rarely seen a case where depression or anxiety were not also present. I’ve treated PTSD, and it’s the same story. Would you believe that most sufferers of severe, adult-onset, chronic pain are survivors of some form of childhood abuse? Even your basic top-level disorders, anxiety and depression, actually share many symptoms and frequently develop simultaneously. And many patients’ suffering simply doesn’t fit any currently available psychiatric diagnosis.

So what happens when you pick a single disorder, cherry-pick patients who don’t meet criteria for any other problem, and study their response to an intervention? They get better! In fact, they mostly get better even when you just put them on a waiting list. Because these are the easiest, most uncommonly uncomplicated cases. They are not a good representation of the patients that actually come in for treatment. Therefore, the studies are not generalizable; they tell us very little about how well a treatment will work in the real world.

The studies aren’t repeatable

Because there are so many potential confounding factors in research with live humans, you can never read too much in to any one study. Once a study has been repeated a few times, preferably by different researchers under different conditions, then we can start to get a little more confident that we’re seeing a real treatment effect and not just an artifact of some unpredictable condition of the study. But that’s not happening very much in the field of clinical psychology. Most psychotherapy outcome studies are simply never replicated.

Part of the problem is publication bias: for a variety of reasons, papers describing a failure to replicate previous studies don’t get published as often. Many of the findings could not be replicated regardless, because they are false positives — the study designs do not adequately discern between treatment effects, placebo effects, and random effects. There is tremendous pressure on academic researchers to produce positive results, and this leads to both conscious and unconscious bias in the way research is carried out.

Finally, psychological treatments, like medical treatments, are subject to decline effects. Over time, They get less effective. Later studies find lesser effects than earlier studies, even when the research design is sound. This could be due to changes in researcher enthusiasm, changes in the patient populations that the treatments are being tested on, or changes in the cultural context in which treatments are being administered. Regardless, most published research findings are false. It is therefore highly problematic to make those findings the basis of clinical treatment.

Here are some excellent articles about these problems:

The studies often ignore therapist effectiveness & the treatments fail to utilize therapist skill

A large number of controlled and naturalistic studies have found that the therapist providing treatment is more important to the outcome than the type of treatment being provided. This makes a certain kind of natural sense to most people, yet psychologists remain resistant to the idea and the reasons for it have not been adequately studied. We don’t really know what makes the difference between a really great therapist, a mediocre therapist, and an ineffective therapist. Treatment adherence does not explain the difference.

So what do psychological researchers do? Rather than try to isolate the factors that make a therapist great, they try to eliminate them in order to study the effects of specific treatment protocols. The therapists studied must comply with manualized treatments in ways that often contradict their own highly trained clinical instincts, and then statistical methods are used to cover over any remaining differences.

The result is rigid treatment designs that do not provide therapists with information about how to get the greatest clinical effect out of them. In practice, psychotherapy is fluid and must be adapted to the individual client on a moment-by-moment basis. But because these manualized treatments are supposed to be “evidence-based,” clinicians are obligated to go through a process of supervised practice during which they must comply with the manual at the expense of therapeutic effectiveness. Then, once they are certified to administer this evidence-based treatment, they can go back to doing the things they have found actually help people, while adhering only to the basic structure of the manualized treatment. In my experience, clinicians do their best to pick very “easy” and compliant patients as test cases for certification, since these types of patients are the ones who will generally get better no matter what you do with them.

Significant results in studies are not good enough to be considered clinical success

Even if we ignore the methodological problems and take these studies at face value, the results they achieve are generally insufficient to indicate a treatment effect that I would be satisfied with in my own practice. A “statistically significant” effect just means that whatever happened probably didn’t happen by chance. So with this type of result, we know that the treatment had some effect, but we know nothing about how much of an effect it had.

For that, we need to look at effect sizes. Even a “large” effect size (0.8) really only means that the average person in the treatment group ended up better off than 79% of people who didn’t get the treatment. We’re not talking cures here, we’re talking about less than half of the people getting somewhat better than people who got nothing.

Of course, “better” depends on the measures used. In most studies, outcomes are measured using symptom questionnaires that patients are asked to fill out before and after the treatment. The symptoms on the questionnaire will relate only to the specific disorder being treated, so for most studies we never know how well patients ended up in terms of overall quality of life. The broader the disorder, the bigger this problem becomes.

For example, Dialectical Behavior Therapy (DBT) is widely recognized as the best and most empirically supported treatment for Borderline Personality Disorder. And the studies have shown that DBT does, in fact, reduce self-harm and increase treatment compliance in sufferers of borderline personality disorder. They have not, however, shown DBT to cure borderline personality disorder — and not for lack of trying. So here you have a widely promoted intervention that does not bring about a reversal of the diagnosis for which it is prescribed. There are interventions which have been shown to do this, such as Mentalization-Based Treatment. But, for a variety of reasons, that treatment does not seem to be gaining traction in the evidence-based psychology community. As more money gets thrown at DBT studies, the likelihood increases that someone will eventually manage to get a positive result for reversal of borderline personality disorder, and then that finding will be used to solidify DBT as the mandated choice among healthcare systems using evidence-based treatments.

Study funding and treatment implementation tends to be theoretically aligned

Cognitive Behavior Therapy (CBT) is far and away the most widely studied form of psychological treatment. Let’s set aside for a moment the fact that “CBT” has gradually expanded to include the most widely-used interventions from all theoretical frameworks, as well as longer treatment periods for complicated or difficult presentations. It has been shown effective for a wide range of psychiatric disorders, with more added all the time. Based on its impressive base of literature, many researchers, educators, and clinicians assert that it is the most effective form of psychotherapy. The problem is, it’s not.

Studies comparing CBT to other types of therapy have found similar effect sizes between treatments for decades, with psychodynamically-oriented therapies showing more lasting treatment gains. So why is CBT so much better represented in funding and implementation? Essentially, politics. The cognitive-behavioral theoretical camp got a strong foothold at NIMH and held on for dear life. There are more studies showing that CBT is effective simply because there have been more studies funded for this type of treatment.

The same is true with exposure-based therapies for trauma. Therapies which focus on exposure are the most heavily promoted and widely held to have the best empirical base. However, at least five other types of treatment show equal treatment effects, and a couple of them contain no exposure elements whatsoever. There are underlying factors that are getting lost in the politics. As a clinician I have absolutely no interest in the politics — I’m interested in figuring out what works for each patient. From that basis, we may be able to start to extrapolate ingredients that make treatment more generally effective for all patients (this is the idea of practice-based evidence.)

Treatments studied often have poor theoretical integrity

As I mentioned, CBT has been steadily expanding over the years to encompass more techniques, such as relaxation, visualization, and mindfulness meditation. The exploration of core beliefs and schemas has evolved in a direction very similar to modern brief psychodynamic therapies. The only thing that distinguishes CBT from the less- (but still often well-) supported psychotherapies is its theoretical foundation.

However, the same expansion beyond CBT’s theoretical basis has led to the creation of treatment protocols which are essentially eclectic. Researchers have long lamented their difficulty in isolating the specific factors which lead to successful treatment, and these new ostensibly cognitive-behavioral protocols amplify the problem by introducing a technique-driven, kitchen-sink approach that renders a difficult task impossible. Instead of drilling down into the therapies to identify process variables which highly effective therapists incorporate naturally, we are building up theoretically vapid treatments that seem to work about as well as everything else, but target a specific psychiatric disorder (for which there is also little theoretical or empirical integrity.) Are you confused by all this yet?

Symptom-focused EBTs lead to modularization of treatment, increased total treatment time, reduced overall effectiveness

As this newfound glut of grant-funded “evidence-based” psychotherapies are rolled out in large healthcare systems, the result is a bureaucratization of psychotherapy. Patients are shuffled from one empirically supported group to another, and handed off from one therapist to another based on what symptoms each therapist is certified to treat.

For example, a common case is an obese veteran with chronic pain, depression, PTSD, and alcohol dependence. For the obesity, they’ll be referred to a weight loss group. For the chronic pain they’ll be referred for manualized pain management treatment, either in a group or individual format. The alcohol dependence will likely lead to a referral for a substance abuse treatment group, which will be followed by a referral to a manualized “Seeking Safety” group to transition between substance abuse treatment and PTSD treatment. PTSD treatment might be treated with either Cognitive Processing Therapy or Prolonged Exposure Therapy, either of which may lead to some relief from depression, but an additional referral will likely be needed for a depression treatment group. So in this common scenario, the patient is asked to participate in 6 separate 8-12 week, empirically supported treatments, each of which requires orientation into a separate vocabulary and set of expectations, and each of which has about half a chance of producing remission for its particular symptom. This treatment process will average about 60 weeks, most likely with gaps of 1 month or more in between the start of each treatment, meaning that the whole thing will take a year and a half, conservatively. How long do those non-evidence based treatments take again?

Moreover, decades of studies examining the factors related to positive treatment outcomes have shown that one of the most important factors is the therapeutic relationship. The model which I have described, which is currently in use within some major healthcare systems and which is being heavily promoted for implementation everywhere, simply obliterates the therapeutic relationship which would be established over the course of an 18-month, single-provider course of psychotherapy. That relationship is replaced by a series of much shallower treatment relationships with well-meaning therapists, and one major relationship, to the bureaucracy itself.

Medication vs. CBT for Generalized Anxiety Disorder

Citing a relative scarcity of research on the efficacy of CBT for GAD as compared to pharmacotherapy, Kristin Mitte conducted a meta-analysis of 65 controlled studies using a random-effects model to produce results that could be generalized beyond the selected studies. A “trim-and-fill analysis” was also conducted to correct for publication bias, several additional sensitivity analyses were performed to ensure the robustness of the selected studies, and methodological differences were controlled for. Studies utilizing new techniques in CBT such as mindfulness practices and interpersonal interventions were excluded due to insufficient research.

The analysis found CBT to be a highly effective treatment for GAD, “reducing not only the main symptoms of anxiety but also the associated depressive symptoms and subsequently improving quality of life.” Mitte determined that CBT was at least as effective as benzodiazapenes, and approximately as effective as SSRIs and azapirones (such as buspirone) while being far better tolerated than any of these three pharmacological treatments. It is concluded that, although methodological variations make it impossible to determine which of the  GAD treatments considered is the best, CBT is a valuable alternative to pharmacotherapy for treating GAD.

Citation:  Mitte, K. (2005). Meta-analysis of cognitive-behavioral treatments for generalized anxiety disorder : A comparison with pharmacotherapy. Psychological Bulletin, 131(5), 785-795.

Best Practices for Treatment of Anxiety Disorders

The Canadian Psychiatric Association (CPA) cites a number of meta-analyses which it recognizes as having “clearly demonstrated” the efficacy of cognitive-behavioral therapy (CBT) in alleviating anxiety symptoms. According to these Guidelines, the effectiveness of CBT in the treatment of anxiety symptoms in general is on par with that of antidepressant drug treatment. This appears to be the case in both individual and group therapy settings. For Generalized Anxiety Disorder (GAD) specifically, CBT is more effective than either placebo psychological treatment or no treatment at all.

Some of the common problems that have been identified in GAD sufferers, according to the CPA, include intolerance of uncertainty, inadequate approaches to problem-solving, and the belief that worry is an effective way to deal with problems. In response to these cognitive deficits, therapists commonly utilize psychoeducational tactics, cognitive interventions such as reappraisal of unrealistic beliefs, exposure experiences geared towards the development of tolerance for anxiety-provoking situations, emotion-regulation approaches, problem-solving skills development, and preparation for inevitable periods of increased anxiety. The CPA found that a greater number of these components being used in therapy was predictive of a better treatment outcome, while comparisons between individual components showed little difference. They also found that the addition of treatment components focused on increasing the client’s overall sense of psychological well-being is associated with better outcomes.

There is no current evidence to support routine combination of CBT with drug therapy, according to the CPA.

Citation: Canadian Psychiatric Association (2006). Clinical practice guidelines: Management of anxiety disorders. Canadian Journal of Psychiatry, 51(Suppl 2), 51S-55S.

Components of CBT For Anxiety Disorders

Borkovec, Newman, Pincus, and Lytle here cite prior reviews of outcome research as having well established CBT as an effective treatment for GAD with low drop-out rates and treatment gains that “routinely maintained or increased at long-term follow-up.” Regardless of this, CBT still fails to produce highly functional states in a large percentage of clients.

This five-year study was designed to explore potential avenues for increasing the efficacy of CBT by extending its duration, measuring the efficacy of two components—cognitive therapy (CT) and applied relaxation and self-control desensitization (SCD)—of the CBT therapy being used, and by measuring interpersonal factors contributing to the success of failure of the treatment.

Although previous research has demonstrated that CBT is more effective than its individual parts over a short duration, these researchers found that each of the components they studied became as effective as the full CBT over the course of a 16-week treatment. Furthermore, they found no improvement in the treatment outcomes of the group receiving both therapeutic components, indicating that an extended duration is unlikely to improve outcomes.

The study found a strong association between some interpersonal behaviors—such as being domineering, vindictive, or nonassertive—and retention of symptoms at followup. The researchers conclude that complementary interpersonal therapy may improve the efficacy of CBT, and cite some existing research which corroborates this theory.

Citation: Borkovec, T. D., Newman, M. G., Pincus, A. L., & Lytle, R. (2002). A component analysis of cognitive-behavioral therapy for generalized anxiety disorder and the role of interpersonal problems. Journal of Consulting and Clinical Psychology, 70(2), 288-298.

Meta-Analysis of Treatments for Depression, Panic, and GAD

Citation: Westen, D., & Morrison, K. (2001). A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: an empirical examination of the status of empirically supported therapies. Journal of Consulting and Clinical Psychology, 69(6), 875-899.

This meta-analysis draws a distinction between initial response and sustained efficacy and attempts to determine the sustained efficacy of evidence-based treatments. This is particularly useful in their examination of GAD, for which a sizable percentage of clients experience significant improvements even before seeing a therapist for the first time. The analysis included five controlled studies of GAD which generally excluded comorbid conditions and had high completion rates.

The analysis found a clinically meaningful effect size for a majority of clients meeting criteria for GAD, although the majority of these clients retained “mild but clinically significant” symptoms after treatment. Data to determine the sustained efficacy of the examined EBTs, however, are unavailable. The authors point out the serious questions that this lack of data raises about the status of current evidence-based treatments for GAD. Whereas EBTs for panic disorder show promising results, those for depression demonstrate sustained efficacy of only 25-30% over 12-24 months. Until long term follow-up data is available for GAD EBTs, their recommendation as empirically supported interventions remains dubious.

Aging Brains Only Shrink When They’re Sick

It is well known that the average brain size of elderly people is smaller than that of younger ones. Most doctors and scientists have decided that this is a normal part of aging. However, it turns out that this may not be the case.

A new study in the APA journal Neuropsychology used a highly controlled sample from Maastricht University’s long-term Aging Study in Holland. These individuals were screened every three years for a wide range of neurological problems, and divided into groups based on level of functioning. Neuropsychological test batteries were used to determine how well participants maintained neural functioning, and MRI scans were used to measure the physical size of seven brain areas important for aging, including hippocampal and parahippocampal areas as well as the frontal and cingulate cortices.

Study participants showed a loss of brain mass in direct relationship to cognitive decline. The researchers conclude that age-related decreases in brain size likely reflect pathological changes in the brain, rather than natural processes of aging.

This is a great example of the need for more cautious and methodical interpretation of clinical research. In attempting to determine what is normal, it is too easy for clinical researchers to miss the mark on what is optimal.

Citation: Burgmans, S., van Boxtel, M.P.J., Vuurman, E.F.M.P., Smeets, F., Gronenschild, E.H.B.M., Uylings, H.B.M., & Jolles, J. (2009). The prevalence of cortical gray matter atrophy may be overestimated in the healthy aging brain. Neuropsychology, 23 (5), 541-550. [PDF]

Hypnotizability May Be Unrelated To Dissociation & Cognitive Inhibition

The prevailing theories of hypnotic susceptibility hold that the ability to experience hypnotic phenomena is a function of either dissociative capacity or of attentional control. However, an upcoming study in the journal Consciousness And Cognition claims to challenge both of these ideas.

The researchers administered the Waterloo-Stanford Group Scale of Hypnotic Susceptibility (Form C), the Dissociative Experiences Scale (normed for non-clinical populations), and a series of cognitive inhibition tasks to 180 study participants. They conclude, decisively, that “the data ruled out even moderate correlations between hypnotic suggestibility and all the measures of dissociation and cognitive inhibition.”

The implications of these findings are uncertain. After all, the idea that hypnotizability exists as a biological or personality trait at all is controversial. From the perspective of those therapists who suspect that everyone is susceptible to some form of hypnotic experience, it is unsurprising that investigations into hypnotic ability should lead to unpredictable and ultimately meaningless results.

But even if we assume the validity of hypnotic susceptibility as an individual trait, then these findings are of questionable value. With regard to dissociation, this study’s findings are not new. Many conflicting studies have found varying levels of relationship between hypnosis and dissociation, which are now generally considered to be related but separate phenomena.

The study’s findings on cognitive inhibition are interesting, but without the use of a reliable measure of inhibitory cognitive control little can be said other than more research in this area may be warranted.

Citation:  Dienes, Z., Brown, E., Hutton, S., Kirsch, I., Mazzoni, G., & Wright, D.B. (2009) Hypnotic suggestibility, cognitive inhibition, and dissociation. Consciousness And Cognition.