Response to “Does Alcoholics Anonymous Work” published by The Conversation

I came across an article which criticizes AA using the in my opinion now-debunked 2014 book by Dodes. I have already dealt with why the Dodes’ figures are suspect in previous blog entries, but to summarize: The figures have been, in my opinion, downright discredited in Cochrane 2020’s report on AA efficacy.

One thing that article does is point to a study which “proves” that alcoholics can control and moderate their drinking. One thing to keep in mind is that while it’s pretty easy to find a study which shows that out-of-control alcoholics claim they are controlling their drinking, those studies are generally short term and based on self-reported data. The last time we had a real long-term study was the 10-year follow-up in Pendery 1982, a famous (if old) study showing that alcoholics do not successfully control and moderate their drinking over a long time period.

But, interested, I looked at this meta-study supposedly showing controlled drinking among alcoholics. This meta-study included studies going back to the 1970s. Most of the studies had only short-term follow-ups; there were only two studies with follow-ups longer than two years.

Diving deeper, I looked at the two studies with a final follow-up longer than two years used to support the idea that an alcoholic can control and enjoy his drinking:

• One study with a 3.5 year follow up is Graber et. al. 1988 [1]. N=24 (which is a much lower N than the studies used to support Cochrane 2020) Looking at the study, I observed that while the supposed “controlled drinkers” claimed to be drinking less than the group told to abstain from alcohol, the collaterals (usually, the drinker’s loved ones) reported that the “controlled drinkers” were actually drinking a lot more than the abstinent group: Self-reported data among the abstinent group was 21 drinks a week (collaterals reported less); collaterals reported nearly 41 drinks a week for the controlled drinking group (self-reporting was a good deal lower).

• The other study they used was from back in the 1970s: Caddy et. al. 1978 [2]. I started reading this study before metaphorically throwing it across the room. It looks at alcoholics in the infamous Sobell 1973 study. The very same alcoholics they looked at in Pendery 1982! (To be pedantic, Pendery 1982 only looked at the 20 alcoholics in the controlled drinking experimental group, while Caddy 1978 looked at all 70 subjects across four groups, but still) At this point, I stopped bothering to read the meta-study.

We know exactly what happened to that very same CD-E (experimental controlled drinking) group of 20 alcoholics 10 years later: They weren’t controlling and enjoying their drinking! (4 dead, 8 out-of-control drinking, 6 abstinent, 1 moderately drinking, 1 not located) Despite this, the meta-study, without even mentioning the existence of Pendery 1982, claimed this study showed the controlled drinking group did better than the abstinent group. I can not take seriously a meta-study which uses data based on those 20 people in Pendery 1982 to argue an alcoholic can control and enjoy his drinking.

If we throw out the data based on the subjects in Pendery 1982, we are left either with studies whose final follow-up is at most two years after treatment, or with the all of 24 subjects in Graber et. al. 1988 where the “moderate drinking” (“controlled drinking” as they called it back then) group was apparently drinking a good deal more than the abstinent group, and were, based on their collaterals, being more dishonest about their drinking.

In terms of the N used for long term follow-ups, as a point of comparison, the Match study used by the Cochrane group in their 2020 AA report had, at the three year follow-up, 952 subjects.

To address the rest of that article, to use Dodes’ questionable figures to conclude AA has a low success rate, and then compare that to a figure that, as it turns out, considers 12-step fellowships a part of mainstream treatment to imply AA isn’t “mainstream treatment” is straight up bad science.  [3] The article tries really hard to downplay the results of Cochrane’s 2020 report on Alcoholics Anonymous: After claiming TSF (which, yes, gets people in the rooms of AA where the real recovery starts) isn’t more effective than other treatments, the article finally contradicts itself in the next sentence by conceding that 12-step oriented therapies increase abstinence, but hedges this by stating “the goal of 12-step facilitation therapy is always abstinence, while other therapies may support a goal of controlled drinking or harm reduction for some heavy and dependent drinkers”; never mind that the article wasn’t able to provide a compelling case that controlled drinking is a workable long-term solution for alcoholics (it isn’t, as I already detailed earlier in this post). Did I mention that this article doesn’t even try to address the multiple observational studies showing that regular AA meeting attenders are twice as likely to get and stay sober?

(I originally posted this on Reddit)

[1] “Abstinence or controlled drinking goals for problem drinkers: A randomized clinical trial” 1988

[2] “Individualized behavior therapy for alcoholics: a third year independent double-blind follow-up” 1978

 [3] The science in that article gets worse. The article links to AA’s 2014 membership survey pamphlet stating that, since 27% of people in the survey are in their first year, this shows a 73% relapse rate. Actually, what the pamphlet shows is that 27% of members are in their first year, and 73% have one year or more sober—quite the opposite of a 73% relapse rate. Like I said, bad science.

The bogus 5% success rate

Recently, someone asked how the “5% success rate” stat was made up. Here is my answer, edited to give more information and context.

To summarize, the real success rate, based on current science and randomized high quality studies, is 42%. The 5% stat has been discredited. There’s actually two primary places the discredited number comes from, as well as a third figure which I haven’t seen used for a long time.

For people who have long since forgotten their math, a numerator is the number we divide (the top of a fraction); the denominator is the amount we divide by (the bottom of a fraction). For example, 3/4 (.75 or 75%) has a numerator of three and a denominator of four.

The 1990 Triennial Survey

One is a 1990 survey of AA membership. It included a graph which was poorly labeled and confusing to read. People misread it, and thought it showed a 5% “success rate”, when, in fact, the graph shows a 26% one-year retention rate (PDF file).

Don McIntire, in a 2000 paper entitled “How Well Does A.A. Work?”, claimed, without evidence, that there was a 81% first month dropout rate, and that “at the end of 90 days, only 10% of newcomers are still present”. [1] This, despite the fact the original triennial survey directly contradicts him, stating that “about half those who come to A.A. are gone within three months.” There is nothing in the original 1990 triennial survey discussing an 81% one-month dropout rate; the number appears to come from people who didn't understand the graph and decided that's what it showed without any thought of how AA was supposed to measure the people who dropped out in their first month, since AA doesn't keep membership records nor meeting attendance records.

The actual numerator was the number of alcoholics in their 12th month sober; the denominator was all the alcoholics in their first year in the survey.

The Sober Truth

The other big source for the 5% figure comes from a 2014 book by Lance Dodes, The Sober Truth. What Lance Dodes did was look at a then 34-year-old study (and not the 8-year-old Moos and Moos 2006 study he was aware of—he mentions it elsewhere—but which has better numbers for AA success) and decide only people who both had 2.5 years or more sober and were still regularly attending AA were a “success”. Sober, but not regularly going to AA, not a “success”. Regularly going to AA, but only 2 years sober, also not a “success”. Rather disingenuous on Dodes’s part if you ask me, especially since Dodes ignored the fact that 1980 study showed that, the more people went to AA, the more likely they would be sober (e.g. 42% of regular AA attenders were sober one year or more in the graph Dodes used to calculate his 5% figure, compared to 16% for people who never went to AA). [2]

The numerator was the alcoholics in a survey both regularly going to AA and with over 2.5 years sober; the denominator was all of the people in the survey who went to at least one AA meeting.

Reading Vaillant poorly

This hasn’t been attributed to Vaillant since about 2015 or so; it is usually described as being a “Harvard study” which supposedly shows a 95% recidivism rate.

The old Orange Papers site claimed that Vaillant's book The Natural History of Alcoholism Revisited showed a 5% success rate, but that is a myth. The old green-papers site from the same era refuted it quite nicely: http://web.archive.org/web/20150329052802/http://green-papers.org/ “95% of patients had relapsed at some time during the study, even though many of these eventually attained sobriety. [...] It's well known that most severe alcoholics only get sober after many relapses, to the extent that relapses can be considered part of the recovery process. So don't pretend that's a failure. [...] And anyway, this was a study of a health network, not AA.”

The numerator was the number of people who never relapsed, not even once. The denominator was all of the people in the study, regardless of whether they went to AA.

Cochrane 2020: 42% success rate

Now that we have gone over three discredited figures, let’s show some more accurate figures for AA’s success rate. The 2020 Cochrane Report on Alcoholics Anonymous shows a 42% success rate for AA-centered therapies, compared to 35% for the other therapies. 

The numerator here is all of the people who were abstinent from alcohol one year after the survey was started; the denominator is people serious enough about their sobriety to get regular therapy.

Footnotes

 

[1] To be fair here, the triennial survey was initially incorrectly read in Bufe’s 1991 book Cult or Cure (second edition published in 1997; not to be confused with Vaillant’s 2005 paper with the same name) and Fox’s 1993 book Addiction, Change and Choice, both anti-AA polemics published by See Sharp Press, a publisher of books about “anarchism and atheism”, hardly a medical source nor academic journal. McIntire, however, gets the dubious honor of being the first one to publish this erroneous data in an academic journal. McIntire did point out that, if we only count people who stay in AA for 90 days, we get around a 55% one year retention rate. 

[2] Polich, Armor, and Braiker The course of Alcoholism: Four Years after Treatment. Rand 1980. Page 129

How The Sober Truth came up with its bogus 5% figure

While there hasn’t been any mainstream press criticizing AA’s efficacy recently, I think, while Dodes’s figures are now outdated and discredited in light of Cochrane 2020, it would serve the reader to know the exact source data Dodes used to synthesize his bogus 5% “success rate” figure for Alcoholics Anonymous (AA).

A few years ago, Dr. Thomas Beresford had this to say about Lance Dodes’s alleged 5% success rate for Alcoholics Anonymous:

The Atlantic article’s source [i.e. Lance Dodes’s book] multiplies a 25% AA attendance figure by a 22% abstinence figure to arrive at a 5.5% estimate of AA’s effectiveness. Where do these figures come from? Another second-hand source [Note: This blog post will detail that second-hand source below] that also cites the work of others: two publications from the Rand Corporation that examined, among other things, attempts at controlled drinking and offered little focus on AA’s effectiveness. At 4-year follow-up the Rand group identified patients with at least one year abstinence who had been regular members of AA 18 months after the start of treatment: 42% of the regular AA members were abstinent, not the “calculated” 5.5% figure.

It may benefit the reader to dive in to this more deeply. Here is how Lance Dodes describes AA as supposedly having a 5% success rate:

University of California professor Herbert Fingarette cited two [...] statistics: at eighteen months, 25 percent of people still attended AA, and of those who did attend, 22 percent consistently maintained sobriety. [Reference: H. Fingarette, Heavy Drinking: The Myth of Alcoholism as a Disease (Berkeley: University of California Press, 1988)] Taken together, these numbers show that about 5.5 percent of all those who started with AA became sober members.

And here is how the late Herbert Fingarette, back in 1988, described A.A.’s success rate (as cited by Dodes above):

[...] one large-scale study of alcoholics in treatment centers in the U.S. found that 71 percent of their subjects had attended A.A. at some point; but at eighteen-month and subsequent followups only 14 percent to 18 percent were attending. Moreover, the rate of problems was higher for irregular A.A. attenders than for either regular attenders or non-attenders. Among those who were regular A.A. attenders at the time of the initial interview, only 22 percent consistently maintained abstention up through the thirty-month followup interview, and over 33 percent had not only returned to drinking but also showed alcohol-related physical symptoms and life problems.

This paragraph has a single footnote, the 1980 Rand report on alcoholism. The 1980 Rand Report can be readily downloaded. Let’s start of with a summary of a graph on page 154 of that report.

Never attended AA: 23 out of 129 (18%) sober over 6 months.
Some AA attendance: 85 out of 315 (27%) sober over 6 months
Lots of AA attendance: 42 out of 74 (57%)

The Rand report, as skeptical as it is of AA (remember, they were trying to determine if alcoholics could moderately drink again, something we know today is a fool’s errand), admits that the abstinence rate for regular AA attenders is higher than any other group, with a p score of under .05.

The 22% and 18% figures Beresford then Dodes used for the bogus 5.5% figure, are based on table 6.15 at the bottom of page 129 of that 1980 Rand report. This is a four year follow up, with four columns based on 18-month followup AA attendance (never, only in the past, only occasionally, and regular attendance). They also looked at the number of people with over 6 months sober, over one year sober, and with 2.5 years sober. 

As expected, the more people went to AA, the more likely they would be sober. In more detail:

AA attendance	6 months+ sober	1 year+ sober	2.5+ years
Never	24%	16%	11%
Only in past	25%	19%	12%
Occasionally	29%	19%	12%
Regularly	45%	42%	22%

Now, as we can see, overall, not that many people managed to get 2.5 years sober, but those that went to AA were more likely to stay sober. As it turns out, only 18% of the people were regular AA attenders (Dodes, mysteriously, claims the number is 25%); of those, only 22% got sober. 

So, Dodes’s 5.5% number is based on this Rand report, and it is both:

• The people who regularly went to AA meetings. People who did not regularly attend AA meetings were not considered an “AA success” in Dodes’s book, regardless of whether they actually got sober.
• Of those people, only the people who did not have one drink for over 2.5 years were considered a success. If someone had a single drink two years ago, but was sober for over a year, regularly attending AA meetings, they were still considered to be not an “AA success”.

By doing everything to minimize the number of people that are an “AA success”, he was able to come up with a really low number for AA success. Indeed, if we make the bar more reasonable, such as one year or more sober, we get an 8% (instead of 4%) success rate, again not counting the people who didn’t regularly attend AA but got sober anyway.

Why did Dodes use a 1980 study based on 1970s numbers in a 2014 book? The numbers in the 1980 study were remarkably low. If we play the same game with the numbers in Moos and Moos 2006, we get a 17% success rate for AA, again not counting everyone who got sober who didn’t regularly go to AA meetings. 

Even if we use this old 1980 study, if we use the numbers on page 128 (not page 129) of the 1980 Rand report, we get a 8% success rate (if only regular attenders count), or a 24% success rate if anyone who has set foot in AA counts. Actually, we get a 57% success rate if we only use the number of people regularly going to AA as our denominator (instead of all the people in the study).

Point blank: The math Dodes used to come up with a 5.5% AA success rate are based on a study that was 34 years old when he wrote the book, even though another similar 8-year-old study was available, and use a kind of math whose goal appears to be to make AA look as bad as possible.

Dodes’s numbers were disingenuous, plain and simple.

As an aside, since that old 1980 Rand report claimed that some AA attendance resulted in worse drinking consequences than no AA attendance, something Fingarette observed, I will address that. Those numbers were based on self-reporting answers. Based on the fact that the studies used in the 2020 Cochrane report on AA do not see worse drinking consequences for people randomly assigned to AA-centered therapies, the most likely reason that old Rand report saw worse consequences is because people who never have gone to AA are more likely to be dishonest about the extent of negative consequences from their drinking. As I have heard countless people say in AA meetings, doctors could not properly treat us because we were so dishonest about our drinking before going to AA.

Herbert Fingarette; Patheos again

Herbert Fingarette, who died last year, was someone who was always opposed to Alcoholics Anonymous and the disease theory of Alcoholism, and believed the lie that alcoholics can moderately drink again. Indeed, Lance Dodes did some creative math with Fingarette’s numbers when concluding Alcoholics Anonymous has a 5% success rate:

University of California professor Herbert Fingarette cited two [...] statistics: at eighteen months, 25 percent of people still attended AA, and of those who did attend, 22 percent consistently maintained sobriety. [Reference: H. Fingarette, Heavy Drinking: The Myth of Alcoholism as a Disease (Berkeley: University of California Press, 1988)] Taken together, these numbers show that about 5.5 percent of all those who started with AA became sober members.

The New York Times points out that Fingarette never did any alcoholism science himself:

Professor Fingarette acknowledged that he had not conducted any experimental or clinical studies into alcoholism; he reached his conclusions by analyzing scientific literature

Point being, not only did not Dodes multiply unrelated numbers to synthesize an artificially low number for AA’s success, but he used figures from an author who himself never published an alcoholism study.

Patheos

Patheos is one of the few places on the Internet where the same old debunked myths about AA having a “5% success rate” keep coming up again and again (the other two big offenders are Reddit, which has little regard for the actual truth but prefers a good fictional story, and TheFix). I posted this refutation to those old discredited numbers in a recent article there:

AA has, among those who are heavily involved with it, a 75% success rate over the short term, and a 67% rate over the long term (16 years later). These numbers have been consistent among multiple longitudinal studies, and are the same numbers AA members observed in the mid-1950s. To name just one study showing a high success rate among regular AA members, please see PMC2220012 (Moos and Moos 2006)

Indeed, the Surgeon General, in 2016 (during the Obama administration), stated that “Well-supported scientific evidence demonstrates the effectiveness of twelve-step mutual aid groups focused on alcohol and twelve-step facilitation interventions.”

Lance Dodes’s numbers concluding that AA has a 5% success rate are very dubious. Quite frankly, Dodes pulled the numbers out of his rear end: He multiplied unrelated numbers from multiple studies together to synthesize an artificially low number for AA’s success. His figures were so bad, multiple highly regarded treatment experts have criticized him, with one calling his reasoning a “pseudostatistical polemic”

People are welcome to their opinions, but not to their facts. The fact is that AA works among a significant subset of alcoholics (and, yes, atheists, there are agnostic AA meetings, or just use SMART/LifeRing/whatever, which appear to be about as effective as long as abstinence is the goal). Denying facts is what we accuse the superstitious of doing; it would be best if we did not do the same thing.

Good to see the word is getting out

Reddit posted yet another repost of that old 2015 article from The Atlantic about how AA supposedly has a 5% success rate. However, this time someone actually researched the number and posted the following:

From Dodes' book " research indicates that only 5 to 8 percent of the people who go to one or more AA meetings achieve sobriety for longer than one year."
https://en.wikipedia.org/wiki/Effectiveness_of_Alcoholics_Anonymous#The_Sober_Truth
But that number is controversial (and the above wikipedia article gives a good overview of the conflicting views).

Point being, all of the effort I have gone to over the years to find accurate information about AA’s success and making the Wikipedia article on the topic reflect what science actually knows is paying off; people can no longer claim “AA has only a 5% success rate” without the claim being questioned.

The mid-2010s situation where a single doctor, without quoting any actual study, makes up a 5% success rate figure for AA by multiplying numbers from different studies together and not being questioned are behind us. Now, anyone interested in facts can and does find out that Dodes’s numbers were garbage.

AA has a 67-75% success rate among people who regularly go to meetings.  This is the number that science shows us; this number has been replicated across multiple studies and has been consistent since the mid-1950s.

Review: “US of AA” by Joe Miller

I will review “US of AA: How the Twelve Steps Hijacked the Science of Alcoholism” by Joe Miller. Since this blog focuses on the efficacy of AA (67-75%), I will look at how this book looks at AA’s success. To quote the book:

Perhaps the most widely known statistic showing AA's ineffectiveness relative to other treatment methods is the 5 percent success rate -- that served as a centerpiece statistic in The Sober Truth, a 2014 expose by former Harvard psychiatry professor Lance Dodes of the bad science behind AA. In fact, he gathered the statistic from AA’s own surveys. Every three years since 1968 AA has randomly queried several thousands of its members for basic info -- age, career, gender, how they came to AA, and length of sobriety. One of the questions AA asks is the month and year when they first came into AA. In 1990, the AA member who analyzed the results used the data to "show the probability that a member will remain in the Fellowship a given number of months." He calculated that out of every hundred people who came into AA, eighty leave within a month. At the three-month mark, only ten remain. At one year, that number has dwindled to five.

Seems straightforward: AA fails ninety-five percent of the people who come in the door.

In other words, the author has completely disregarded any real research on AA’s success done in the last decade. Indeed, Miller did not correctly read The Sober Truth, since Dodes does concede that that old 1990 AA survey shows a 26% retention rate. Dodes’s book uses another trick to make up the 5% success rate figure, which is also inaccurate (he multiplies numbers from unrelated studies to cook up that artificial 5% figure).

If Miller can not get basic facts about AA’s success rate correct, I see no need to waste my time reading any more of this poorly researched book.

Fortunately, this book did not get the level of press that Dodes’s poorly argued 2014 polemic got.

Charlie Sheen, clean and sober in AA

Charlie Sheen, back in 2011, claimed that Alcoholics Anonymous has only a 5% success rate. Thankfully, over the years, he realized that those numbers are questionable.

He now has one year clean and sober in Alcoholics Anonymous. I'm very happy that he was able to let go of outdated ideas like the idea that science shows AA is ineffective (it doesn't), and instead use the program to become clean and sober.

What is AA's real success rate?

The question is this: What is AA's real success rate? We can never know the true number for two key reasons:

• AA does not keep track of members.
• "AA success" is a poorly defined term, which can have several actual meanings

While AA itself did not keep track of members, Moos and Moos 2006 did. It found the following: Of the people who were serious about AA, 67% of them were sober 16 years later.

But what about the dropouts? Not everyone who did the study did the 16-year followup, so we can't know the true success rate number, but looking at only the people who did follow up gives us a reasonable number (Dodes 2014 claims that Moos/Moos 2006 has completely invalid numbers because of these dropouts, but Dodes appears to have had no interest in an honest assessment of AA's success). Overall, 46% of the people in the Moos and Moos 2006 survey (again, only those who did the 16-year followup) were sober, regardless of their level of AA attendance.

Now we are getting in to the realm of philosophical questions: Do we count someone who was not serious about AA but still stayed sober as a success or failure for AA? Should we count people who weren't serious about meetings when calculating AA's success rate? Polemics like Dodes's The Sober Truth consider every single person who was not serious about AA a failure when making artificially low numbers for AA's success, even if the person got sober after all.

There are a lot of ways we can come up with a "success rate" number based on Moos and Moos 2006:

• There were 461 people in the survey who did a 16-year followup
• Of those, 25% (115) took the AA program seriously. Of those 115, 77 (67% of serious AA members, 17% overall) were sober, and 38 (33% of serious AA members, 8.2% overall) were not sober. We could use these numbers to argue that AA has "only" a 17% success rate, but that argument is dishonest because it discounts the 16% of people (72) who had some level of exposure to AA and still stayed sober.
• Overall, 214 (46%) got sober.  Since pretty much any alcoholic out there at least knows of AA's existence, we could make a case this is a 46% success rate.
• If we look at just people who had a least a little bit of AA treatment, this is 58% (269 of 461 total) of the people in the survey; of those 55% (149) were sober 16 years later. Contrast this with the 42% (192) who didn't try AA at all, 34% or 14% overall (65) were sober 16 years later.

Point being, there are a lot of ways to measure AA success, but to consider people who didn't take AA seriously but still got sober as "failures" (as Dodes and others have done) is downright dishonest.

Alcoholics Anonymous's own success claims
 
Alcoholics Anonymous has, since 1939, made the following claim about AA's success "Rarely have we seen a person fail who has thoroughly followed our path." This claim was expanded in the mid-1950s: "Of alcoholics who came to A.A. and really tried, 50% got sober at once and remained that way; 25% sobered up after some relapses, and among the remainder, those who stayed on with A.A. showed improvement."

Alcoholics Anonymous has never claimed it would help people who did not take the program seriously. Any claim about AA's success should only look at the people who take AA seriously; once we discount the people who did not take AA seriously, the numbers we get are in line with the numbers AA gave us in the mid-1950s (75% success rate among those whose who "really tried" AA).

It is true that AA has a high drop out rate, so I contend this: Have we found a treatment which is effective for alcoholics who, for whatever reason, refuse to work the AA program? Zemore 2019 makes a strong case that people in other programs can also stay clean and sober, as long as the goal is abstinence from alcohol. So, yes, I think people who will not work the AA program can still be clean and sober, but I also think it's dishonest to label all such people "failures" when measuring AA success.

Brandsma 1980 redux

Here’s a paragraph I wrote criticizing Bradsma 1980 when I restored that study to the Effectiveness of Alcoholics Anonymous article (using footnotes):

This study had poor methodology. There was no effort to stop the people in the control group from attending Alcoholics Anonymous meetings, and the "Alcoholics Anonymous" treatment patients underwent in the Brandsma study did not use actual Alcoholics Anonymous meetings. "The control condition allowed for participation in actual AA meetings, while those in the AA condition attended a weekly AA-like meeting administered by the study (that was not an actual AA meeting)"

That might not have been neutral, so another editor tried to make that paragraph more balanced:

Notably, there was no effort to stop the people in the control group from attending Alcoholics Anonymous meetings, and the "Alcoholics Anonymous" treatment patients underwent in the Brandsma study did not use community Alcoholics Anonymous meetings. A later critical analysis noted: "the control condition allowed for participation in actual AA meetings, while those in the AA condition attended a weekly AA-like meeting administered by the study (that was not an actual AA meeting)"

I feel this wording undermines the significant criticism Kaskutas 2009 has about Brandsma 1980, so I reworded it again, hopefully more neutral:

There was no effort to stop the people in the control group from attending Alcoholics Anonymous meetings. The "Alcoholics Anonymous" treatment patients underwent in the Brandsma study did not use community Alcoholics Anonymous meetings; a later analysis says that there are "concerns with the Brandsma trial which call its experimental results into question" because "the control condition allowed for participation in actual AA meetings, while those in the AA condition attended a weekly AA-like meeting administered by the study (that was not an actual AA meeting)"

This study has been heavily abused

I think this is pretty neutral, especially in light of just how problematic Brandsma 1980 is, especially when quoted out of context in anti-AA polemics. As just one example of this, here is how Dodes’s The Sober Truth describes Brandsma 1980:

The investigators found “significantly more binge drinking at the 3-month follow-up” among the people assigned to the AA-oriented meetings. As the year mark approached, the researchers noted, “All of the lay-RBT clients reported drinking less during the last 3 months. This was significantly better than the AA or the control groups at the 0.005 level [meaning the finding was highly statistically significant].” The final data led the researchers to conclude: “In this analysis the AA group was five times more likely to binge than the control group and nine times more likely than the lay-RBT group. The AA group average was 2.4 binges in the last 3 months.”

Brandsma, who passed away in 2008, is probably rolling in his grave that his legacy is this particular study that he did in 1979-1980 being misrepresented. Scott Alexander, in his blog, does not view how this study has been abused very kindly:

Brandsma (1980) is the study beloved of the AA hate groups, since it purports to show that people in Alcoholics Anonymous not only don’t get better, but are nine times more likely to binge drink than people who don’t go into AA at all.

There are a number of problems with this conclusion. First of all, if you actually look at the study, this is one of about fifty different findings. [...]

Second of all, the increased binge drinking was significant at the 6 [this is an error; it was at three months we saw the increased binge drinking] month followup period. It was not significant at the end of treatment, the 3 month [Scott probably means 6-month] followup period, the 9 month followup period, or the 12 month followup period. Remember, taking a single followup result out of the context of the other followup results is a classic piece of Dark Side Statistics and will send you to Science Hell.

Scott also has the same issues with this study Kaskutas 2009 had:

Brandsma didn’t use a real AA group, because the real AA groups make people be anonymous which makes it inconvenient to research stuff. He just sort of started his own non-anonymous group, let’s call it A, with no help from the rest of the fellowship, and had it do Alcoholics Anonymous-like stuff. On the other hand, many members of his control group went out into the community and…attended a real Alcoholics Anonymous, because Brandsma can’t exactly ethically tell them not to. So technically, there were more people in AA in the no-AA group than in the AA group.

Point being, this study has been criticized for its methodological problems, abused by people more interested in attacking AA than in objective facts, and has possibly done more harm than good in finding out the truth of Alcoholics Anonymous’s efficacy.

The 1990 triennial survey: 26%, not 5% one year retention rate

There is an old myth that a 1990 AA Triennial survey shows a 5% one-year retention rate for Alcoholics Anonymous. This is a false belief: That graph, frequently brought up by anti-steppers with little interest in objective truth, actually shows a 26% retention rate, as pointed out in the article "Alcoholics Anonymous (AA) Recovery Outcome Rates".

The graph is not a membership retention graph; it is a simple count of, among the members in their first year, how many were around a given number of months.  If AA had a 100% retention rate, the graph would have shown 8.3% (i.e. 100% divided by 12) of members (instead of the 5% it shows) in their 12th month of sobriety.

The wording in the survey itself says it is a graph of "the number of members that have been around a given number of months"; it was not a graph of per-month membership retention.

This graph does not show an 80% first month drop out rate and it does not show a 5% one-year retention rate.  It shows a 70% first-month retention rate and a 26% one-year retention rate.

Here are a number of books and papers which claimed the 1990 Triennial survey showed a 5% retention rate before "Alcoholic Anonymous Recovery Outcome Rates" was published:

• McIntire, Don (2000). "How Well Does A.A. Work?". Alcoholism Treatment Quarterly. 18 (4): 1. doi:10.1300/J020v18n04_01. "at 12 months the figures are 5% remaining and 95% departed."
• Carl G. Lukefield, "Behavioral Therapy for Rural Substance Abusers", 2000 has this quote: "By 12 months, 90 percent have dropped out (McCrady & Miller, 1993)"
• Charles Bufe, AA: Cult or Cure 1997 has this quote Quote: "AA produced a large monograph, “Comments on A.A.’s Triennial Surveys,” that analyzed the results of all five surveys done to that point. [...] AA has a 95% new-member dropout rate during the first year of attendance."

However, after "Alcoholic Anonymous Recovery Outcome Rates" was published, a number of sources all of a sudden decided that the 1990 Membership retention survey showed a 26% one-year retention rate, which just happens to be the number in that paper. Even anti-AA polemics like Dodes' (poorly-argued) The Sober Truth now use the 26% figure. For example:

• Lance Dodes, M.D.; Zachary Dodes (2014). The Sober Truth: Debunking the Bad Science Behind 12-Step Programs and the Rehab Industry. ISBN 978-0807033159. "AA itself has published a comparable figure in a set of comments on its own thirteen-year internal survey, stating that only 26 percent of people who attend AA stay for longer than one year"
• Adrian, Manuella (2012). "Can Failure Carefully Observed Become a Springboard to Success?". Substance Use & Misuse. 47 (13–14): 1384. doi:10.3109/10826084.2012.706178. "the Alcoholics Anonymous triennial surveys from 1977 through 1989 found that one quarter (26%) of those who first attend an AA meeting are still attending after 1 year"

Point being, the conclusion in "Alcoholic Anonymous Recovery Outcome Rates" that those old surveys showed a 26% one-year retention rate are considered reliable among treatment experts.

As Slate Star Codex puts it: "Almost everyone’s belief about AA’s retention rate is off by a factor of five because one person long ago misread a really confusing graph and everyone else copied them without double-checking."

Dodes 2014: Bad science

Lance Dodes' book The Sober Truth is, bottom line, bad science. I have read it and compared what it says to the papers Dodes cites; in multiple cases, Dodes makes fundamental errors in his research. For example, Dodes claims that Fiorentine 1999 showed a 40% sobriety rate among active AA members; however, reading this study shows that it actually demonstrates a 75% sobriety rate. As another example, Dodes claims that Moos and Moos 2006 didn't look at how many active AA members stayed sober; actually Moos and Moos 2006 reports that 67% (or 77 people) were sober 16 years later.

I am not the only one who finds Dodes' reasoning flawed. The New York Times calls it a "polemical and deeply flawed book." The esteemed doctors Chad Emrick and Thomas Beresford, who find Dodes entire reasoning process questionable, looking at the second way Dodes tries to claim AA has a 5% success rate and say that "Albeit harsh to say, it appears that Dodes and Dodes did not let facts get in the way of drawing their second conclusion about AA’s effectiveness." The psychiatrists Jeffrey D. Roth and Edward J. Khantzian, in their review of The Sober Truth called Dodes' reasoning "pseudostatistical polemic." Addiction experts John Kelly and Gene Beresin also found Dodes' reasoning flawed.

For the record, the scientists criticizing Dodes' work come from the best schools: John Kelly, Eugene Beresin, and Edward J. Khantzian are all from Harvard. In addition, Jeffrey Roth graduated from Yale, and Thomas Beresford graduated from Stanford. I see a lot of very prestigious doctors strongly criticizing Dodes' work.

The outrage press (NPR, The Atlantic, Salon) placed far too much weight on the opinion of one doctor and his son even though so many other experts have looked at his reasoning and found it deeply flawed.

Dodes and Moos and Moos 2006

Since Moos and Moos 2006 invalidates a central thesis of Dodes' false claim that Alcoholics Anonymous has a 5% success rate, Dodes goes to some effort to try to discredit this key study.

After discussing how the people who were successful in Moos and Moos 2006 were compliant (which is just another way to word the "self selection bias" problem), Dodes writes this:

The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.

Dodes is flat out inaccurate here. The multiple Moos studies which use the same subjects, including Moos and Moos 2006, had a total of 628 people who said they would participate in the studies. Moos and Moos 2006 looks at 461 of them. We have a drop out rate (in other words, people who elected to be in the study but didn't follow up) of under 27%, yet Dodes is stating Moos/Moos 2006 had a drop out rate of 40%. It would appear Dodes is confusing the number of people who stopped going to meetings with the number of people who didn't do two or more follow-ups; while these are both "drop outs", they are very different figures.

What Moos and Moos measured was this: Of the 115 people who were heavily involved with AA in the first year, and who did two or more follow ups, 77 of them (67%) were sober 16 years later.

What about those 167 subjects in Moos and Moos 2006 who didn't do two or more follow ups? We can go back to Timko and Moos 1993, the one-year follow-up study which used the same subjects. Here, they look at 515 of those 628 people. Even when looking at a larger subset of subjects, we get the same level of people who tried AA: Moos and Moos 2006 has 269 of the 461 subjects studied who tried AA in their first year (that's 58%).  Timko and Moos 2003 has 294 of the 515 subjects trying AA in their first year (57%). It would appear that the drop outs who did not finish two or more surveys did not significantly affect the results.

Dodes continues:

The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up and also stayed in AA for longer than six months -- that is, the group on which the authors’ major findings are based -- was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group "had better outcomes."

Dodes, again, is not correctly reading Moos and Moos 2006. First of all, there were 115 (not 107) people heavily involved with AA in the first year. Moos and Moos 2006, in addition, looked very closely at the number of people who were sober 16 years later: "Individuals who received 27 weeks or more of [Alcoholics Anonymous] treatment in the first year were more likely to be abstinent and less likely to have drinking problems at 16 years than were individuals who remained untreated in the first year." In addition, Moos and Moos 2006 tells us exactly how many people in that group were sober 16 years later in Table 3: 67% of 115. I will save Dr. Dodes the bother of bringing out a calculator: That's 77 people. I can not see how Dodes can look at Moos and Moos 2006 and claim it did not state how many people who took AA seriously were sober 16 years later.

In terms of other factors, table 2 of Moos and Moos 2006 shows a strong correlation between AA treatment and improvement in both self-efficacy and social functioning. To quote Moos/Moos 2006: "a longer duration of AA was independently related to a higher likelihood of 16-year abstinence, enhanced self-efficacy, and good social functioning, and less likelihood of 16-year drinking problems."

As far as I can tell, Dodes did not even fully read Moos and Moos 2006 before writing up a polemic trying to discredit its numbers. I expected better.

Let's get some real figures for AA success:  Moos and Moos 2006 showed that 67% of the people who took AA seriously were sober 16 years later. If the self section issue is problematic and you want experimental studies, both Litt et al. 2009 and Walitzer 2009 show that correctly getting subjects in the rooms of 12-step meetings work better than other addiction treatments.  Bottom line, AA works.

Fiorentine 1999: 75% success rate

I just added this to Effectiveness of Alcoholics Anonymous:

Fiorentine 1999 was a 24-month longitudinal study measuring the effectiveness of AA and Narcotics Anonymous. Like other longitudinal studies, it shows a strong correlation between 12-step attendance and being both clean from drugs and sober from alcohol: People who went to 12-step meetings in the study had about a 75% success rate.

In more detail, at the 24 month follow up, 77.7% of people who went to one or more meetings a week self-reported being clean; urinalysis was very close to that figure, showing some 76.4% of the regular attenders begin clean. For participants who did not regularly go to meetings, the self reported figure for being clean was 56% and the drug test showed 57.9% being clean. The self-reported figures for alcohol abuse were similar: 74.8% of regular 12-step attenders self-reported being completely sober, but only 40% of people who did not go to 12-step meetings claimed to be sober. Urinalysis showed 96.6% of people regularly going to meetings as sober, in contrast to the 88.9% of people who didn't go to meetings once a week or more whose urine sample was alcohol-free.

The report then sees if 12-steps meetings have a specific effect (in other words, if there is causation or merely correlation) by using statistical analysis to compare participants self-reported level of motivation, 12-step attendance, and successfully getting clean and sober. The result of this observation was that "Weekly or more frequent 12-step participants are 1.59 times more likely than less-than-weekly participants to maintain abstinence after controlling for the differences in recovery motivation measured by the scale."

---

Not added to that article: Lance Dodes falsely claims that Fiorentine 1999 states that "approximately 40 percent of individuals categorized as having continued active participation in AA maintained high rates of abstinence." and then multiplies an unrelated figure to falsely claim that AA has an 8% success rate. Fiorentine 1999 makes no such claim. No where in this paper does the above quote appear. I was unable to find this quote. Thomas Beresford was unable to find this quote. As far as either of us can tell, this quote is a figment of Dodes' imagination.

Surprise! An article in a stoner magazine doesn't like Alcoholics Anonymous

Surprise! An article in a stoner magazine doesn't like Alcoholics Anonymous.

This article uses the following two anti-AA tropes to claim that AA does not work:

• Lance Dodes' claim that AA has a 5-8% success rate, which I have have dealt with four years ago in this blog, and which has been fully refuted in the article "Alcoholics Anonymous and The Atlantic: A Call For Better Science"
• That table in Handbook of Alcoholism Treatment Approaches which anti-steppers (such as stoners lying to themselves) love to bring out, which is also quite inaccurate (yes, I have already blogged about this table). Did I mention that its author, William R. Miller, has recently written a book telling alcoholics how to moderate their drinking which is, quite bluntly, a fool's errand.

The article also claims the AA Big Book is inaccurate when it claims a 75% success rate. Actually, I have seen multiple studies with this figure or similar figures; e.g. Moos and Moos 2006 shows, among alcoholics who took AA seriously, 67% of them were still sober 16 years later.

Salon's posts on AA

As I mentioned in my previous blog entry, with one exception, all of Salon's articles about AA have been negative. Let's look at Salon's list of AA articles:

• Pseudo-science and Bible-thumping, the worst of both worlds: The problem with rehab, AA and calling addiction a disease Negative, obviously
• Addiction is not a disease: How AA and 12-step programs erect barriers while attempting to relieve suffering Another negative one
• The drunkard’s loop: How one writer fought her way back from blackout alcoholism While this article mentions AA, it is not an article about AA
• AA is not a fraud: Why haters are afraid to listen Salon's one and only positive article about AA
• Debunking Alcoholics Anonymous: Behind the myths of recovery Here is one of the articles from Lance Dodes Salon published
• The pseudo-science of Alcoholics Anonymous: There’s a better way to treat addiction Salon likes Lance Dodes' bovine excrement about AA so much, they published two articles from him
• Is my friend suicidal? This article mentions AA but is not about AA
• My sober conversion to atheism While not as negative as, say, anything Lance Dodes ever wrote, this article is pretty negative about the AA program
• “Inside Rehab”: How it could work better, and why it doesn’t Surprisingly neutral when mentioning AA, but this is not an article about AA per se
• “Workaholism” is real This article only mentions AA in passing
• I was a sober college student Not directly about AA, but describes AA positively
• I didn’t even want these kids! Not directly about AA, but describes the 12-step program positively

In 2012 Salon was neutral, maybe even positive towards AA. By 2014, they took the Orange Papers kool-aid and were posting negative anti-AA polemics. If Salon is going to survive, they are going to have to be more positive about AA. They posted Lance Dodes' bovine excrement -- why don't they post Thomas Beresford's articles showing that Alcoholics Anonymous really works? 

"The Insane Idea"

Here is a very long -- probably too long -- article which goes in to, among other things, the fact that the science shows, as we say after the serenity prayer, "it works if you work it."

http://thepointmag.com/2016/examined-life/the-insane-idea Mirror: https://archive.is/u2ttt

One point this article makes is that it is an objective scientific fact that "the longer people attend twelve-step meetings, the more likely they are to achieve long-term sobriety."

The article, however, while acknowledging the countless studies showing a correlation between AA attendance and long-term sobriety, and correctly points out that "It is [a] gerrymandered set of data that leads Dodes to the conclusion that just “5 to 8 percent of the total population of people who enter AA are able to achieve and maintain sobriety for longer than a year”," simply quotes Dodes' assertion that AA's success rate may be because of self selection, without looking at PMC4285560 which indicates that it's not just self-selection which causes people who regularly go to meetings to stay sober.

A somewhat disappointing article; while it engages Glaser and Dodes for their inaccurate polemics, the author did not look at the recent research showing that AA is effective before writing this extended rambling piece.

the longer people attend twelve-step meetings, the more likely they are to achieve long-term sobriety - See more at: http://thepointmag.com/2016/examined-life/the-insane-idea#sthash.v6jBN2HS.dpuf

the longer people attend twelve-step meetings, the more likely they are to achieve long-term sobriety - See more at: http://thepointmag.com/2016/examined-life/the-insane-idea#sthash.v6jBN2HS.dpuf

the longer people attend twelve-step meetings, the more likely they are to achieve long-term sobriety - See more at: http://thepointmag.com/2016/examined-life/the-insane-idea#sthash.v6jBN2HS.dpuf

the longer people attend twelve-step meetings, the more likely they are to achieve long-term sobriety - See more at: http://thepointmag.com/2016/examined-life/the-insane-idea#sthash.v6jBN2HS.dpuf

the longer people attend twelve-step meetings, the more likely they are to achieve long-term sobriety - See more at: http://thepointmag.com/2016/examined-life/the-insane-idea#sthash.v6jBN2HS.dpuf

AA has a higher success rate than 5%

There is a frequently quoted 5% success rate figure for AA's success. This is a popular figure which anti-steppers love to bring up in the comments sections of articles about AA, almost always without citation. This number comes from no less than three different sources, all of which use incorrect methods to derive the 5% number:

• The 1990 Triennial Survey which the anti-steppers love to bring out had, of the population of people in their first year, 5% in their 12th month. To say this shows a 5% success rate shows a profound ignorance of statistics, as a HindsFoot article clearly shows.
• Agent Orange started a rumor that Vallant in The Natural History of Alcoholism Revisited showed a 5% success rate. The old green-papers site refuted it quite nicely; I summarize those findings in the Wikipedia talk page for Valliant's book. The actual AA success figures in that book, for people who attended 300 or more meetings over a ten-year period, is on page 197. The "this many people who attended 300 or more meetings" numbers are not in that table, but easily enough determined with some simple math: 74% of the people who went to 300+ meetings had stable remission, 21% of the 300+ meetings attenders had intermittent alcoholism, and only 5% of people who went to 300 or more meetings were still chronic alcoholics -- numbers, that, interestingly enough, agree with the figures in the preface to the second edition of AA's own Big Book (50% got sober right away, 25% got sober after relapsing, and the rest showed improvement).
• Lance Dodes used some questionable math in The Sober Truth to come up with this same figure; Gabrielle Glaser, in her 2015 hit piece, parroted Dodes' figures. Thomas Beresford, writing for the National Council on Alcoholism and Drug Dependence, directly refutes Dodes' bogus 5% figure in one article and shows some actual success figures for AA in another article.

June 2015 round-up

OK, I'm back, but just to summarize a couple of articles and comments I have found.

First of all,  a recent study concluded that "increasing AA attendance leads to short- and long-term decreases in alcohol consumption that cannot be attributed to self-selection." (PMC4285560). Austin Frakt wrote a New York Times piece about it: http://www.nytimes.com/2015/04/07/upshot/alcoholics-anonymous-and-the-challenge-of-evidence-based-medicine.html 

This study is pretty key: It's been well-established by the science that, as we say in AA, "meeting makers make it"; the question now being asked is "do meeting makers make it because those that go to meetings are more motivated to get sober, or because AA helps keep alcoholics sober?" This study concludes it's the actual program which keeps people sober.

* * *

Here's another good refutation of Lance Dodes's book: http://www.amazon.com/gp/cdp/member-reviews/A3HLE5G8L7A4OQ For example, here's an excerpt from one study (PMC2746426; I have taken the liberty to remove references to other parts of the study placed in parenthesis in the original):

Rates of abstinence are about twice as high among those who attend AA; higher levels of attendance are related to higher rates of abstinence; these relationships are found for different samples and follow-up periods; prior AA attendance is predictive of subsequent abstinence; and mechanisms of action predicted by theories of behavior change are present in AA. However, rigorous experimental evidence establishing the specificity of an effect for AA or Twelve Step Facilitation/TSF is mixed, with 2 trials finding a positive effect for AA, 1 trial finding a negative effect for AA, and 1 trial finding a null effect. Studies addressing specificity using statistical approaches have had two contradictory findings, and to that reported significant effects for AA after adjusting for potential confounders such as motivation to change.

This is how Dodes quotes that in his book:

Rigorous experimental evidence establishing the specificity of an effect for AA or 12 Step Facilitation is mixed, with two trials finding a positive effect for AA, one trial finding a negative effect for AA, and one trial finding a null effect. Studies addressing specificity using statistical approaches have had two contradictory findings, and to that reported significant effects for AA after adjusting for potential confounders such as motivation to change.

Only someone with a strong anti-12-step bias would take a study that states that "rates of abstinence are about twice as high among those who attend AA" and conclude that the study is saying that "the strong evidence that one would expect if AA were clearly effective is simply not present."

What Dodes is doing here is a dishonest: He is not arguing that meeting makers make it, although he tries to imply that using unrelated data. The evidence that "meeting makers make it" is so well-established in the addiction science academic community that Dodes does not even try to directly argue that adage. What he is instead arguing is that perhaps meeting makers make it because they were more motivated to get sober at the start, or because of AA's social network. 

It's a strictly academic question, and does not affect the fact that AA is the only alcohol support group in the world with thousands upon thousands of free meetings. Even here, the studies are at worst inconclusive, only because AA is so successful and universal that anyone seeking treatment for alcoholism has been exposed to AA. To go from "it's really hard to decide whether someone going to AA is successful because of the program itself or because of other factors" to "AA doesn't help alcoholics" is a really large, and downright intellectually dishonest, leap.

Anyway, studies like PMC4285560 have tried really hard to eliminate other factors, and conclude that there's something about the AA program itself that is helping keep people sober. While not directly looked at in this study, I say it is the 12 steps as written in the first 164 pages of the Big Book that are keeping people sober.

As we were talking about at a meeting yesterday, the problem with the press is that the only news that attracts the press is bad news. The numerous studies giving us more and more evidence that AA works are ignored by the press; it was Lance Dodes's very biased, inaccurate, and negative book which got so much mainstream press attention last year.

More comments defending AA against Dodes' attacks

More comments defending AA against Dodes' attacks:

http://umano.me/c/5LmEp/debunking-alcoholics-anonymous

As one person put it "the real research shows the positive greatly outweighs the negatives."

Lance Dodes refuted

Lance Dodes has been claiming that AA is roughly as effective as spontaneous remission (10%).  However, the science does not back up this claim, as discussed here:

http://commonhealth.wbur.org/2014/04/defense-12-step-addiction

To summarize that article:

• A 1991 study that Dodes quotes actually shows that AA is about 20% effective when people just go to meetings, and 40% effective when people go to 30-day 12-step based treatment and then go to AA meetings.
• The Cochrane study does not say AA is ineffective.  It actually says that AA is at least as effective as other treatment: "AA may help patients to accept treatment and keep patients in treatment more than alternative treatments" and "Other studies reported similar retention rates regardless of treatment group"
• There are other scientific studies showing that AA is an effective treatment which Dodes completely ignored.