Response to “Does Alcoholics Anonymous Work” published by The Conversation

I came across an article which criticizes AA using the in my opinion now-debunked 2014 book by Dodes. I have already dealt with why the Dodes’ figures are suspect in previous blog entries, but to summarize: The figures have been, in my opinion, downright discredited in Cochrane 2020’s report on AA efficacy.

One thing that article does is point to a study which “proves” that alcoholics can control and moderate their drinking. One thing to keep in mind is that while it’s pretty easy to find a study which shows that out-of-control alcoholics claim they are controlling their drinking, those studies are generally short term and based on self-reported data. The last time we had a real long-term study was the 10-year follow-up in Pendery 1982, a famous (if old) study showing that alcoholics do not successfully control and moderate their drinking over a long time period.

But, interested, I looked at this meta-study supposedly showing controlled drinking among alcoholics. This meta-study included studies going back to the 1970s. Most of the studies had only short-term follow-ups; there were only two studies with follow-ups longer than two years.

Diving deeper, I looked at the two studies with a final follow-up longer than two years used to support the idea that an alcoholic can control and enjoy his drinking:

• One study with a 3.5 year follow up is Graber et. al. 1988 [1]. N=24 (which is a much lower N than the studies used to support Cochrane 2020) Looking at the study, I observed that while the supposed “controlled drinkers” claimed to be drinking less than the group told to abstain from alcohol, the collaterals (usually, the drinker’s loved ones) reported that the “controlled drinkers” were actually drinking a lot more than the abstinent group: Self-reported data among the abstinent group was 21 drinks a week (collaterals reported less); collaterals reported nearly 41 drinks a week for the controlled drinking group (self-reporting was a good deal lower).

• The other study they used was from back in the 1970s: Caddy et. al. 1978 [2]. I started reading this study before metaphorically throwing it across the room. It looks at alcoholics in the infamous Sobell 1973 study. The very same alcoholics they looked at in Pendery 1982! (To be pedantic, Pendery 1982 only looked at the 20 alcoholics in the controlled drinking experimental group, while Caddy 1978 looked at all 70 subjects across four groups, but still) At this point, I stopped bothering to read the meta-study.

We know exactly what happened to that very same CD-E (experimental controlled drinking) group of 20 alcoholics 10 years later: They weren’t controlling and enjoying their drinking! (4 dead, 8 out-of-control drinking, 6 abstinent, 1 moderately drinking, 1 not located) Despite this, the meta-study, without even mentioning the existence of Pendery 1982, claimed this study showed the controlled drinking group did better than the abstinent group. I can not take seriously a meta-study which uses data based on those 20 people in Pendery 1982 to argue an alcoholic can control and enjoy his drinking.

If we throw out the data based on the subjects in Pendery 1982, we are left either with studies whose final follow-up is at most two years after treatment, or with the all of 24 subjects in Graber et. al. 1988 where the “moderate drinking” (“controlled drinking” as they called it back then) group was apparently drinking a good deal more than the abstinent group, and were, based on their collaterals, being more dishonest about their drinking.

In terms of the N used for long term follow-ups, as a point of comparison, the Match study used by the Cochrane group in their 2020 AA report had, at the three year follow-up, 952 subjects.

To address the rest of that article, to use Dodes’ questionable figures to conclude AA has a low success rate, and then compare that to a figure that, as it turns out, considers 12-step fellowships a part of mainstream treatment to imply AA isn’t “mainstream treatment” is straight up bad science.  [3] The article tries really hard to downplay the results of Cochrane’s 2020 report on Alcoholics Anonymous: After claiming TSF (which, yes, gets people in the rooms of AA where the real recovery starts) isn’t more effective than other treatments, the article finally contradicts itself in the next sentence by conceding that 12-step oriented therapies increase abstinence, but hedges this by stating “the goal of 12-step facilitation therapy is always abstinence, while other therapies may support a goal of controlled drinking or harm reduction for some heavy and dependent drinkers”; never mind that the article wasn’t able to provide a compelling case that controlled drinking is a workable long-term solution for alcoholics (it isn’t, as I already detailed earlier in this post). Did I mention that this article doesn’t even try to address the multiple observational studies showing that regular AA meeting attenders are twice as likely to get and stay sober?

(I originally posted this on Reddit)

[1] “Abstinence or controlled drinking goals for problem drinkers: A randomized clinical trial” 1988

[2] “Individualized behavior therapy for alcoholics: a third year independent double-blind follow-up” 1978

 [3] The science in that article gets worse. The article links to AA’s 2014 membership survey pamphlet stating that, since 27% of people in the survey are in their first year, this shows a 73% relapse rate. Actually, what the pamphlet shows is that 27% of members are in their first year, and 73% have one year or more sober—quite the opposite of a 73% relapse rate. Like I said, bad science.

In response to Stanton Peele’s FilterMag criticism of Cochrane 2020

I initially ignored this article, since it was published by one FilterMag, not a particularly mainstream magazine, and one which a fellow Wikipedia editor calls “an openly biased source advocating harm-reduction.”  However, since I’m starting to see this article being used as an attempt to “refute” the Cochrane 2020 review of AA, it is time to address this.

Stanton Peele has a long-standing opposition to AA and rejection of any science showing AA efficacy, posting anti-AA screeds more than 30 years ago and claiming that chronic alcoholics can drink moderately again despite the scientific evidence showing that to be a fool’s errand. Hardly an objective dispassionate source.

Compare this to the peer-reviewed prestigious Cochrane 2020 report on AA. The Cochrane Library is considered by the Wikipedia community to be “generally of high quality” and their 2020 report on AA is an exemplary example of their excellent scientific rigor.

This attempt to refute Cochrane 2020’s report on AA is really really bad.  

Peele claimed that AA overall doesn’t help alcoholics, since, while AA does increase abstinence, Peele alleged that it does not decrease overall percentage of drinking days and other measures of what I would call alcoholism severity. Nick Heather echoed this claim, attributing it to Peele, when writing a letter criticizing the Cochrane Review. Here is the response to Heather from one of the Cochrane writers:

It should be remembered also, that DDD [Drinks per drinking day] and alcohol-related consequences are averages based solely on those individuals who actually drank at all [...] while more individuals in AA/TSF achieved continuous abstinence, those who were not completely abstinent did not drink more heavily, drink more frequently or experience more alcohol-related consequence

Peele mentions that it’s only at the one year mark we see an increase in abstinence large enough to be very statistically significant (i.e. have a low P value). The reason for this is because the only study with enough subjects to decrease the standard deviation and resulting P value was the 1990s Match study at the one-year followup. Studies, including the Match one, show similar increases in abstinence at other time points, but with fewer subjects. There’s no reason to believe the results would be different if we had as many subjects as we do at the one-year mark, but we cannot get a low enough P with the number of subjects we have to consider the data except at the one-year mark, by Cochrane standards, “high certainty” evidence. [1]

Peele also claims there were not any “reliable” studies showing AA effectiveness in the mid-2000s.  Never mind that we had Moos and Moos 2006 and Fiorentine 1999 which showed a strong correlation between abstinence and regular AA attendance, not to mention Project MATCH which should AA was just as effective as psychotherapy (and whose raw data shows greater abstinence among people directed to go to AA meetings).

He then claims it was a “self selecting” population, but all of the studies used randomization of subjects. He then continues to build up a straw man about Cochrane 2020: He claims they only used two studies to show manualized AA-based TSF treatment has better abstinent outcomes, but pages 88 and 89 of the actual Cochrane review show six different studies, all showing better results for therapies which get people in the rooms of AA. Then he claims the Cochrane review shows higher drinking intensity, but page 92 shows that it’s only a single study (McCrady 1996) which saw a higher percentage of days heavy drinking, while two other studies show either the same number of days of heavy drinking, or a slight improvement for AA-based treatments. The idea that there is an “abstinence violation effect”, that since drinks per day and what not are the same between AA-assigned and non-AA-assigned subjects, and the false allegation this indicates AA attenders who still drink have worse drinking consequences has been addressed above.

He then links to studies which did not pass the muster to be part of Cochrane 2020. I can do that too: Moos and Moos 2006, which shows a 67% 16-year success rate for Alcoholics Anonymous. He then complains that the studies only pick good subjects, which isn’t true: The studies pick subjects who only have alcoholism because, you know, AA is not a program for treating mental illness; it’s a program for treating alcoholism.

One Brandon Bergman who helped with the research for the 2020 Cochrane review of AA also has posted a rebuttal to Peele’s article.

Alcoholics Anonymous works for some alcoholics. The fact that there has been, in the last two years, no serious science refuting the Cochrane 2020 review shows just how solid that review is.

[1] See the graph on page 89 of the Cochrane review.

The bogus 5% success rate

Recently, someone asked how the “5% success rate” stat was made up. Here is my answer, edited to give more information and context.

To summarize, the real success rate, based on current science and randomized high quality studies, is 42%. The 5% stat has been discredited. There’s actually two primary places the discredited number comes from, as well as a third figure which I haven’t seen used for a long time.

For people who have long since forgotten their math, a numerator is the number we divide (the top of a fraction); the denominator is the amount we divide by (the bottom of a fraction). For example, 3/4 (.75 or 75%) has a numerator of three and a denominator of four.

The 1990 Triennial Survey

One is a 1990 survey of AA membership. It included a graph which was poorly labeled and confusing to read. People misread it, and thought it showed a 5% “success rate”, when, in fact, the graph shows a 26% one-year retention rate (PDF file).

Don McIntire, in a 2000 paper entitled “How Well Does A.A. Work?”, claimed, without evidence, that there was a 81% first month dropout rate, and that “at the end of 90 days, only 10% of newcomers are still present”. [1] This, despite the fact the original triennial survey directly contradicts him, stating that “about half those who come to A.A. are gone within three months.” There is nothing in the original 1990 triennial survey discussing an 81% one-month dropout rate; the number appears to come from people who didn't understand the graph and decided that's what it showed without any thought of how AA was supposed to measure the people who dropped out in their first month, since AA doesn't keep membership records nor meeting attendance records.

The actual numerator was the number of alcoholics in their 12th month sober; the denominator was all the alcoholics in their first year in the survey.

The Sober Truth

The other big source for the 5% figure comes from a 2014 book by Lance Dodes, The Sober Truth. What Lance Dodes did was look at a then 34-year-old study (and not the 8-year-old Moos and Moos 2006 study he was aware of—he mentions it elsewhere—but which has better numbers for AA success) and decide only people who both had 2.5 years or more sober and were still regularly attending AA were a “success”. Sober, but not regularly going to AA, not a “success”. Regularly going to AA, but only 2 years sober, also not a “success”. Rather disingenuous on Dodes’s part if you ask me, especially since Dodes ignored the fact that 1980 study showed that, the more people went to AA, the more likely they would be sober (e.g. 42% of regular AA attenders were sober one year or more in the graph Dodes used to calculate his 5% figure, compared to 16% for people who never went to AA). [2]

The numerator was the alcoholics in a survey both regularly going to AA and with over 2.5 years sober; the denominator was all of the people in the survey who went to at least one AA meeting.

Reading Vaillant poorly

This hasn’t been attributed to Vaillant since about 2015 or so; it is usually described as being a “Harvard study” which supposedly shows a 95% recidivism rate.

The old Orange Papers site claimed that Vaillant's book The Natural History of Alcoholism Revisited showed a 5% success rate, but that is a myth. The old green-papers site from the same era refuted it quite nicely: http://web.archive.org/web/20150329052802/http://green-papers.org/ “95% of patients had relapsed at some time during the study, even though many of these eventually attained sobriety. [...] It's well known that most severe alcoholics only get sober after many relapses, to the extent that relapses can be considered part of the recovery process. So don't pretend that's a failure. [...] And anyway, this was a study of a health network, not AA.”

The numerator was the number of people who never relapsed, not even once. The denominator was all of the people in the study, regardless of whether they went to AA.

Cochrane 2020: 42% success rate

Now that we have gone over three discredited figures, let’s show some more accurate figures for AA’s success rate. The 2020 Cochrane Report on Alcoholics Anonymous shows a 42% success rate for AA-centered therapies, compared to 35% for the other therapies. 

The numerator here is all of the people who were abstinent from alcohol one year after the survey was started; the denominator is people serious enough about their sobriety to get regular therapy.

Footnotes

 

[1] To be fair here, the triennial survey was initially incorrectly read in Bufe’s 1991 book Cult or Cure (second edition published in 1997; not to be confused with Vaillant’s 2005 paper with the same name) and Fox’s 1993 book Addiction, Change and Choice, both anti-AA polemics published by See Sharp Press, a publisher of books about “anarchism and atheism”, hardly a medical source nor academic journal. McIntire, however, gets the dubious honor of being the first one to publish this erroneous data in an academic journal. McIntire did point out that, if we only count people who stay in AA for 90 days, we get around a 55% one year retention rate. 

[2] Polich, Armor, and Braiker The course of Alcoholism: Four Years after Treatment. Rand 1980. Page 129

Some other recent criticism of AA in the press

I have found other criticism of AA in the mainstream press over the last year which I should address here in this blog.

Naltrexone redux

A recent New York Times opinion piece (archive link) brought out the same old tired “AA (or abstinence) doesn’t work but Naltrexone allows alcoholics to drink like gentlemen” chestnut.

While cherry picked studies show remarkably high success rates for Naltrexone, this is offset by other studies which do not replicate those results. Naltrexone, like AA, appears to be effective for a subset of alcoholics, but, as per Cochrane, it only helps about 11% of alcoholics. 

The most disturbing thing about studies showing Naltrexone showing controlled drinking among alcoholics is that they consistently do not have long term follow-up periods. For example, that New York Times opinion piece links to two different studies which supposedly show alcoholics successfully drinking moderately again. In both cases, the final follow-up is one year after treatment.

As someone who remembers Pendery 1982 all too well, positive results for moderate drinking without long term follow-ups are not very encouraging. Sure, anything someone throws a dead cat, they will hit a study of alcoholism showing alcoholics successfully engaging in “moderate drinking” over the short term, but the few long-term follow-ups I have seen show that moderate drinking is not a long-term solution to chronic alcoholism.

As a point of comparison, the 1990s Project MATCH randomized study showed increased abstinence for people who engaged in AA-centered therapies three years after treatment (as per its data as presented in Cochrane 2020), and Moos and Moos 2006 is a longitudinal study showing that Alcoholics Anonymous keeps alcoholics sober 16 years later.

Another thing: That New York Times opinion piece claims that only 25% of alcoholics will achieve abstinence, but the study it links to supporting that outdated 25% figure doesn’t actually conclude this; it claims a 25% success rate for abstinence, and links to an old article from 2005 using results that are nearly two decades old. More recent figures from Cochrane show a 42% success rate.

Abstinence violation effect

There have been two (open access) different letters (paywall) recently published which make the same criticism of Cochrane 2020: They claim that there is an “abstinence violation effect” where the subjects undergoing AA-based treatment who are not abstinent somehow do worse when they drink again than subjects not undergoing AA-based treatment. In other words, the accusation is that AA may increase abstinence, but overall AA doesn’t improve things for alcoholics, because the alcoholics not abstinent supposedly drink more heavily.

This “abstinence violation effect” is actually a myth, as one of the authors of the 2020 Cochrane review on AA explains. Since this letter is paywalled, I will quote the most important part of that letter here:

while more individuals in AA/TSF achieved continuous abstinence, those who were not completely abstinent did not drink more heavily, drink more frequently or experience more alcohol-related consequences

Point being, AA overall improves things for alcoholics. 42% are straight up abstinent from alcohol a year later, and those who do not achieve abstinence do not, repeat not have worse consequences from their drinking.

The 2020 Cochrane review of AA has been released

The 2020 Cochrane review of AA efficacy has finally been released. Unlike the old 2006 review, which stated that “available experimental studies did not demonstrate the effectiveness of AA or other twelve-step approaches in reducing alcohol use and achieving abstinence compared with other treatments”, the new 2020 Cochrane review has this to say:

clinically-delivered TSF [twelve-step facilitation] interventions designed to increase AA participation usually lead to better outcomes over the subsequent months to years in terms of producing higher rates of continuous abstinence

Looking at the study, the strongest data shows that Alcoholics Anonymous significantly increases abstinent rates: When comparing Alcoholics Anonymous and/or Twelve Step Facilitation to other alcohol use disorder interventions, at the 12-month follow up, studies show a 41.8% abstinent rate for AA/TSF treatments, compared to 34.5% abstinent using non-AA interventions.

The New York Times, in an article about this review, has this to say:

A.A. leads to increased rates and lengths of abstinence compared with other common treatments

While the full review is behind a paywall, its summary is freely available for anyone to read.

This is the final nail in the coffin for the “AA has a 5% success rate” nonsense the media was spreading around in the mid-2010s.