The bogus 5% success rate

Recently, someone asked how the “5% success rate” stat was made up. Here is my answer, edited to give more information and context.

To summarize, the real success rate, based on current science and randomized high quality studies, is 42%. The 5% stat has been discredited. There’s actually two primary places the discredited number comes from, as well as a third figure which I haven’t seen used for a long time.

For people who have long since forgotten their math, a numerator is the number we divide (the top of a fraction); the denominator is the amount we divide by (the bottom of a fraction). For example, 3/4 (.75 or 75%) has a numerator of three and a denominator of four.

The 1990 Triennial Survey

One is a 1990 survey of AA membership. It included a graph which was poorly labeled and confusing to read. People misread it, and thought it showed a 5% “success rate”, when, in fact, the graph shows a 26% one-year retention rate (PDF file).

Don McIntire, in a 2000 paper entitled “How Well Does A.A. Work?”, claimed, without evidence, that there was a 81% first month dropout rate, and that “at the end of 90 days, only 10% of newcomers are still present”. [1] This, despite the fact the original triennial survey directly contradicts him, stating that “about half those who come to A.A. are gone within three months.” There is nothing in the original 1990 triennial survey discussing an 81% one-month dropout rate; the number appears to come from people who didn't understand the graph and decided that's what it showed without any thought of how AA was supposed to measure the people who dropped out in their first month, since AA doesn't keep membership records nor meeting attendance records.

The actual numerator was the number of alcoholics in their 12th month sober; the denominator was all the alcoholics in their first year in the survey.

The Sober Truth

The other big source for the 5% figure comes from a 2014 book by Lance Dodes, The Sober Truth. What Lance Dodes did was look at a then 34-year-old study (and not the 8-year-old Moos and Moos 2006 study he was aware of—he mentions it elsewhere—but which has better numbers for AA success) and decide only people who both had 2.5 years or more sober and were still regularly attending AA were a “success”. Sober, but not regularly going to AA, not a “success”. Regularly going to AA, but only 2 years sober, also not a “success”. Rather disingenuous on Dodes’s part if you ask me, especially since Dodes ignored the fact that 1980 study showed that, the more people went to AA, the more likely they would be sober (e.g. 42% of regular AA attenders were sober one year or more in the graph Dodes used to calculate his 5% figure, compared to 16% for people who never went to AA). [2]

The numerator was the alcoholics in a survey both regularly going to AA and with over 2.5 years sober; the denominator was all of the people in the survey who went to at least one AA meeting.

Reading Vaillant poorly

This hasn’t been attributed to Vaillant since about 2015 or so; it is usually described as being a “Harvard study” which supposedly shows a 95% recidivism rate.

The old Orange Papers site claimed that Vaillant's book The Natural History of Alcoholism Revisited showed a 5% success rate, but that is a myth. The old green-papers site from the same era refuted it quite nicely: http://web.archive.org/web/20150329052802/http://green-papers.org/ “95% of patients had relapsed at some time during the study, even though many of these eventually attained sobriety. [...] It's well known that most severe alcoholics only get sober after many relapses, to the extent that relapses can be considered part of the recovery process. So don't pretend that's a failure. [...] And anyway, this was a study of a health network, not AA.”

The numerator was the number of people who never relapsed, not even once. The denominator was all of the people in the study, regardless of whether they went to AA.

Cochrane 2020: 42% success rate

Now that we have gone over three discredited figures, let’s show some more accurate figures for AA’s success rate. The 2020 Cochrane Report on Alcoholics Anonymous shows a 42% success rate for AA-centered therapies, compared to 35% for the other therapies. 

The numerator here is all of the people who were abstinent from alcohol one year after the survey was started; the denominator is people serious enough about their sobriety to get regular therapy.

Footnotes

 

[1] To be fair here, the triennial survey was initially incorrectly read in Bufe’s 1991 book Cult or Cure (second edition published in 1997; not to be confused with Vaillant’s 2005 paper with the same name) and Fox’s 1993 book Addiction, Change and Choice, both anti-AA polemics published by See Sharp Press, a publisher of books about “anarchism and atheism”, hardly a medical source nor academic journal. McIntire, however, gets the dubious honor of being the first one to publish this erroneous data in an academic journal. McIntire did point out that, if we only count people who stay in AA for 90 days, we get around a 55% one year retention rate. 

[2] Polich, Armor, and Braiker The course of Alcoholism: Four Years after Treatment. Rand 1980. Page 129

How The Sober Truth came up with its bogus 5% figure

While there hasn’t been any mainstream press criticizing AA’s efficacy recently, I think, while Dodes’s figures are now outdated and discredited in light of Cochrane 2020, it would serve the reader to know the exact source data Dodes used to synthesize his bogus 5% “success rate” figure for Alcoholics Anonymous (AA).

A few years ago, Dr. Thomas Beresford had this to say about Lance Dodes’s alleged 5% success rate for Alcoholics Anonymous:

The Atlantic article’s source [i.e. Lance Dodes’s book] multiplies a 25% AA attendance figure by a 22% abstinence figure to arrive at a 5.5% estimate of AA’s effectiveness. Where do these figures come from? Another second-hand source [Note: This blog post will detail that second-hand source below] that also cites the work of others: two publications from the Rand Corporation that examined, among other things, attempts at controlled drinking and offered little focus on AA’s effectiveness. At 4-year follow-up the Rand group identified patients with at least one year abstinence who had been regular members of AA 18 months after the start of treatment: 42% of the regular AA members were abstinent, not the “calculated” 5.5% figure.

It may benefit the reader to dive in to this more deeply. Here is how Lance Dodes describes AA as supposedly having a 5% success rate:

University of California professor Herbert Fingarette cited two [...] statistics: at eighteen months, 25 percent of people still attended AA, and of those who did attend, 22 percent consistently maintained sobriety. [Reference: H. Fingarette, Heavy Drinking: The Myth of Alcoholism as a Disease (Berkeley: University of California Press, 1988)] Taken together, these numbers show that about 5.5 percent of all those who started with AA became sober members.

And here is how the late Herbert Fingarette, back in 1988, described A.A.’s success rate (as cited by Dodes above):

[...] one large-scale study of alcoholics in treatment centers in the U.S. found that 71 percent of their subjects had attended A.A. at some point; but at eighteen-month and subsequent followups only 14 percent to 18 percent were attending. Moreover, the rate of problems was higher for irregular A.A. attenders than for either regular attenders or non-attenders. Among those who were regular A.A. attenders at the time of the initial interview, only 22 percent consistently maintained abstention up through the thirty-month followup interview, and over 33 percent had not only returned to drinking but also showed alcohol-related physical symptoms and life problems.

This paragraph has a single footnote, the 1980 Rand report on alcoholism. The 1980 Rand Report can be readily downloaded. Let’s start of with a summary of a graph on page 154 of that report.

Never attended AA: 23 out of 129 (18%) sober over 6 months.
Some AA attendance: 85 out of 315 (27%) sober over 6 months
Lots of AA attendance: 42 out of 74 (57%)

The Rand report, as skeptical as it is of AA (remember, they were trying to determine if alcoholics could moderately drink again, something we know today is a fool’s errand), admits that the abstinence rate for regular AA attenders is higher than any other group, with a p score of under .05.

The 22% and 18% figures Beresford then Dodes used for the bogus 5.5% figure, are based on table 6.15 at the bottom of page 129 of that 1980 Rand report. This is a four year follow up, with four columns based on 18-month followup AA attendance (never, only in the past, only occasionally, and regular attendance). They also looked at the number of people with over 6 months sober, over one year sober, and with 2.5 years sober. 

As expected, the more people went to AA, the more likely they would be sober. In more detail:

AA attendance	6 months+ sober	1 year+ sober	2.5+ years
Never	24%	16%	11%
Only in past	25%	19%	12%
Occasionally	29%	19%	12%
Regularly	45%	42%	22%

Now, as we can see, overall, not that many people managed to get 2.5 years sober, but those that went to AA were more likely to stay sober. As it turns out, only 18% of the people were regular AA attenders (Dodes, mysteriously, claims the number is 25%); of those, only 22% got sober. 

So, Dodes’s 5.5% number is based on this Rand report, and it is both:

• The people who regularly went to AA meetings. People who did not regularly attend AA meetings were not considered an “AA success” in Dodes’s book, regardless of whether they actually got sober.
• Of those people, only the people who did not have one drink for over 2.5 years were considered a success. If someone had a single drink two years ago, but was sober for over a year, regularly attending AA meetings, they were still considered to be not an “AA success”.

By doing everything to minimize the number of people that are an “AA success”, he was able to come up with a really low number for AA success. Indeed, if we make the bar more reasonable, such as one year or more sober, we get an 8% (instead of 4%) success rate, again not counting the people who didn’t regularly attend AA but got sober anyway.

Why did Dodes use a 1980 study based on 1970s numbers in a 2014 book? The numbers in the 1980 study were remarkably low. If we play the same game with the numbers in Moos and Moos 2006, we get a 17% success rate for AA, again not counting everyone who got sober who didn’t regularly go to AA meetings. 

Even if we use this old 1980 study, if we use the numbers on page 128 (not page 129) of the 1980 Rand report, we get a 8% success rate (if only regular attenders count), or a 24% success rate if anyone who has set foot in AA counts. Actually, we get a 57% success rate if we only use the number of people regularly going to AA as our denominator (instead of all the people in the study).

Point blank: The math Dodes used to come up with a 5.5% AA success rate are based on a study that was 34 years old when he wrote the book, even though another similar 8-year-old study was available, and use a kind of math whose goal appears to be to make AA look as bad as possible.

Dodes’s numbers were disingenuous, plain and simple.

As an aside, since that old 1980 Rand report claimed that some AA attendance resulted in worse drinking consequences than no AA attendance, something Fingarette observed, I will address that. Those numbers were based on self-reporting answers. Based on the fact that the studies used in the 2020 Cochrane report on AA do not see worse drinking consequences for people randomly assigned to AA-centered therapies, the most likely reason that old Rand report saw worse consequences is because people who never have gone to AA are more likely to be dishonest about the extent of negative consequences from their drinking. As I have heard countless people say in AA meetings, doctors could not properly treat us because we were so dishonest about our drinking before going to AA.

The 2020 Cochrane review of AA has been released

The 2020 Cochrane review of AA efficacy has finally been released. Unlike the old 2006 review, which stated that “available experimental studies did not demonstrate the effectiveness of AA or other twelve-step approaches in reducing alcohol use and achieving abstinence compared with other treatments”, the new 2020 Cochrane review has this to say:

clinically-delivered TSF [twelve-step facilitation] interventions designed to increase AA participation usually lead to better outcomes over the subsequent months to years in terms of producing higher rates of continuous abstinence

Looking at the study, the strongest data shows that Alcoholics Anonymous significantly increases abstinent rates: When comparing Alcoholics Anonymous and/or Twelve Step Facilitation to other alcohol use disorder interventions, at the 12-month follow up, studies show a 41.8% abstinent rate for AA/TSF treatments, compared to 34.5% abstinent using non-AA interventions.

The New York Times, in an article about this review, has this to say:

A.A. leads to increased rates and lengths of abstinence compared with other common treatments

While the full review is behind a paywall, its summary is freely available for anyone to read.

This is the final nail in the coffin for the “AA has a 5% success rate” nonsense the media was spreading around in the mid-2010s.

Herbert Fingarette; Patheos again

Herbert Fingarette, who died last year, was someone who was always opposed to Alcoholics Anonymous and the disease theory of Alcoholism, and believed the lie that alcoholics can moderately drink again. Indeed, Lance Dodes did some creative math with Fingarette’s numbers when concluding Alcoholics Anonymous has a 5% success rate:

University of California professor Herbert Fingarette cited two [...] statistics: at eighteen months, 25 percent of people still attended AA, and of those who did attend, 22 percent consistently maintained sobriety. [Reference: H. Fingarette, Heavy Drinking: The Myth of Alcoholism as a Disease (Berkeley: University of California Press, 1988)] Taken together, these numbers show that about 5.5 percent of all those who started with AA became sober members.

The New York Times points out that Fingarette never did any alcoholism science himself:

Professor Fingarette acknowledged that he had not conducted any experimental or clinical studies into alcoholism; he reached his conclusions by analyzing scientific literature

Point being, not only did not Dodes multiply unrelated numbers to synthesize an artificially low number for AA’s success, but he used figures from an author who himself never published an alcoholism study.

Patheos

Patheos is one of the few places on the Internet where the same old debunked myths about AA having a “5% success rate” keep coming up again and again (the other two big offenders are Reddit, which has little regard for the actual truth but prefers a good fictional story, and TheFix). I posted this refutation to those old discredited numbers in a recent article there:

AA has, among those who are heavily involved with it, a 75% success rate over the short term, and a 67% rate over the long term (16 years later). These numbers have been consistent among multiple longitudinal studies, and are the same numbers AA members observed in the mid-1950s. To name just one study showing a high success rate among regular AA members, please see PMC2220012 (Moos and Moos 2006)

Indeed, the Surgeon General, in 2016 (during the Obama administration), stated that “Well-supported scientific evidence demonstrates the effectiveness of twelve-step mutual aid groups focused on alcohol and twelve-step facilitation interventions.”

Lance Dodes’s numbers concluding that AA has a 5% success rate are very dubious. Quite frankly, Dodes pulled the numbers out of his rear end: He multiplied unrelated numbers from multiple studies together to synthesize an artificially low number for AA’s success. His figures were so bad, multiple highly regarded treatment experts have criticized him, with one calling his reasoning a “pseudostatistical polemic”

People are welcome to their opinions, but not to their facts. The fact is that AA works among a significant subset of alcoholics (and, yes, atheists, there are agnostic AA meetings, or just use SMART/LifeRing/whatever, which appear to be about as effective as long as abstinence is the goal). Denying facts is what we accuse the superstitious of doing; it would be best if we did not do the same thing.

Good to see the word is getting out

Reddit posted yet another repost of that old 2015 article from The Atlantic about how AA supposedly has a 5% success rate. However, this time someone actually researched the number and posted the following:

From Dodes' book " research indicates that only 5 to 8 percent of the people who go to one or more AA meetings achieve sobriety for longer than one year."
https://en.wikipedia.org/wiki/Effectiveness_of_Alcoholics_Anonymous#The_Sober_Truth
But that number is controversial (and the above wikipedia article gives a good overview of the conflicting views).

Point being, all of the effort I have gone to over the years to find accurate information about AA’s success and making the Wikipedia article on the topic reflect what science actually knows is paying off; people can no longer claim “AA has only a 5% success rate” without the claim being questioned.

The mid-2010s situation where a single doctor, without quoting any actual study, makes up a 5% success rate figure for AA by multiplying numbers from different studies together and not being questioned are behind us. Now, anyone interested in facts can and does find out that Dodes’s numbers were garbage.

AA has a 67-75% success rate among people who regularly go to meetings.  This is the number that science shows us; this number has been replicated across multiple studies and has been consistent since the mid-1950s.

Review: “US of AA” by Joe Miller

I will review “US of AA: How the Twelve Steps Hijacked the Science of Alcoholism” by Joe Miller. Since this blog focuses on the efficacy of AA (67-75%), I will look at how this book looks at AA’s success. To quote the book:

Perhaps the most widely known statistic showing AA's ineffectiveness relative to other treatment methods is the 5 percent success rate -- that served as a centerpiece statistic in The Sober Truth, a 2014 expose by former Harvard psychiatry professor Lance Dodes of the bad science behind AA. In fact, he gathered the statistic from AA’s own surveys. Every three years since 1968 AA has randomly queried several thousands of its members for basic info -- age, career, gender, how they came to AA, and length of sobriety. One of the questions AA asks is the month and year when they first came into AA. In 1990, the AA member who analyzed the results used the data to "show the probability that a member will remain in the Fellowship a given number of months." He calculated that out of every hundred people who came into AA, eighty leave within a month. At the three-month mark, only ten remain. At one year, that number has dwindled to five.

Seems straightforward: AA fails ninety-five percent of the people who come in the door.

In other words, the author has completely disregarded any real research on AA’s success done in the last decade. Indeed, Miller did not correctly read The Sober Truth, since Dodes does concede that that old 1990 AA survey shows a 26% retention rate. Dodes’s book uses another trick to make up the 5% success rate figure, which is also inaccurate (he multiplies numbers from unrelated studies to cook up that artificial 5% figure).

If Miller can not get basic facts about AA’s success rate correct, I see no need to waste my time reading any more of this poorly researched book.

Fortunately, this book did not get the level of press that Dodes’s poorly argued 2014 polemic got.

As one Reddit user put it

Here is a Reddit post which nicely summarizes the “AA does not work” nonsense out there:

Partial truth & twisting of facts is however representative of those with a bias and agenda against 12 Step recovery modalities. They expect the gullible to just accept their reporting as the entire truth and not take the time to check out the actual facts. Those with a predilection against 12 step or have a perceived resentment of course will jump on the partial truths and outright lies of those with a similar bias.

How else can you describe a mindset where people believe AA has a 5% success rate even though no peer reviewed study has ever concluded this. Instead, the studies show a 67%-75% success rate among alcoholics who get serious about the program; people in denial, however, don’t let facts gets in the way of their flawed world view.

AA has a 67-75% success rate

Critics of the 12-step programs attempt to discredit the longitudinal studies, which show a 67-75% success rate for AA:

• They don’t like the fact that those high success rate numbers only come from people who actively go to AA meetings.  However, this is consistent with AA’s own claim that “Rarely have we seen a person fail who has thoroughly followed our path” (emphasis mine)
• They claim the numbers are invalid because of “self selection bias”; in other words, they believe the people who are clean and sober because of AA would have gotten clean and sober anyway in an alternate universe where AA doesn’t exist. A 2014 paper demonstrates this actually isn’t the case.
• They complain that the studies do not include dropouts.  However, the drop out rate is small enough to not significantly skew the numbers, especially since, most of the time, the dropouts undoubtedly 1) Didn’t work the AA program and 2) Didn’t get sober. 
• They make other complaints about the methodology of the studies. However, this ignores the fact that these numbers have been pretty consistent across multiple longitudinal studies. Fiorentine 1999: 74.8% success rate. Moos and Moos 2006: 67% success rate 16 years later. The Natural History of Alcoholism Revisited by George E. Vaillant has a table on page 197 showing that 19 out of 100 alcoholics attended 300 or more meetings over a 10-year period, of those 19, 14 were sober: 74% success rate. Witbrodt 2012 saw similar numbers: “high 12-step attendance (abstinence rates averaging 75%)”. AA’s own Big Book has the same abstinence rate using 1955 numbers: “Of alcoholics who came to A.A. and really tried, 50% got sober at once and remained that way; 25% sobered up after some relapses”

The fact is this: We have multiple studies showing the same number: 75% success rate among people who keep coming back to AA. The science is undeniable: Meeting makers make it.
 

More people are realizing that AA is effective

Just yesterday, someone wrote an article for a small regional magazine (which comes off as like those magazines which were free on every street corner back in the days before the Internet took over publishing) regurgitation the same old “AA only has a 5% success rate” nonsense which has long been refuted.

However, instead of the comments agreeing with the bogus 5% success figures, the comments pointed out how inaccurate those figures are. At the beginning of this year, with the exact same online magazine, I was the only commentator with accurate figures for AA’s success.

This indicates that the tide is really starting to turn; the days of being able to post an article claiming AA has only a 5% success rate without getting devoured in the comments section are behind us.

Charlie Sheen, clean and sober in AA

Charlie Sheen, back in 2011, claimed that Alcoholics Anonymous has only a 5% success rate. Thankfully, over the years, he realized that those numbers are questionable.

He now has one year clean and sober in Alcoholics Anonymous. I'm very happy that he was able to let go of outdated ideas like the idea that science shows AA is ineffective (it doesn't), and instead use the program to become clean and sober.

What is AA's real success rate?

The question is this: What is AA's real success rate? We can never know the true number for two key reasons:

• AA does not keep track of members.
• "AA success" is a poorly defined term, which can have several actual meanings

While AA itself did not keep track of members, Moos and Moos 2006 did. It found the following: Of the people who were serious about AA, 67% of them were sober 16 years later.

But what about the dropouts? Not everyone who did the study did the 16-year followup, so we can't know the true success rate number, but looking at only the people who did follow up gives us a reasonable number (Dodes 2014 claims that Moos/Moos 2006 has completely invalid numbers because of these dropouts, but Dodes appears to have had no interest in an honest assessment of AA's success). Overall, 46% of the people in the Moos and Moos 2006 survey (again, only those who did the 16-year followup) were sober, regardless of their level of AA attendance.

Now we are getting in to the realm of philosophical questions: Do we count someone who was not serious about AA but still stayed sober as a success or failure for AA? Should we count people who weren't serious about meetings when calculating AA's success rate? Polemics like Dodes's The Sober Truth consider every single person who was not serious about AA a failure when making artificially low numbers for AA's success, even if the person got sober after all.

There are a lot of ways we can come up with a "success rate" number based on Moos and Moos 2006:

• There were 461 people in the survey who did a 16-year followup
• Of those, 25% (115) took the AA program seriously. Of those 115, 77 (67% of serious AA members, 17% overall) were sober, and 38 (33% of serious AA members, 8.2% overall) were not sober. We could use these numbers to argue that AA has "only" a 17% success rate, but that argument is dishonest because it discounts the 16% of people (72) who had some level of exposure to AA and still stayed sober.
• Overall, 214 (46%) got sober.  Since pretty much any alcoholic out there at least knows of AA's existence, we could make a case this is a 46% success rate.
• If we look at just people who had a least a little bit of AA treatment, this is 58% (269 of 461 total) of the people in the survey; of those 55% (149) were sober 16 years later. Contrast this with the 42% (192) who didn't try AA at all, 34% or 14% overall (65) were sober 16 years later.

Point being, there are a lot of ways to measure AA success, but to consider people who didn't take AA seriously but still got sober as "failures" (as Dodes and others have done) is downright dishonest.

Alcoholics Anonymous's own success claims
 
Alcoholics Anonymous has, since 1939, made the following claim about AA's success "Rarely have we seen a person fail who has thoroughly followed our path." This claim was expanded in the mid-1950s: "Of alcoholics who came to A.A. and really tried, 50% got sober at once and remained that way; 25% sobered up after some relapses, and among the remainder, those who stayed on with A.A. showed improvement."

Alcoholics Anonymous has never claimed it would help people who did not take the program seriously. Any claim about AA's success should only look at the people who take AA seriously; once we discount the people who did not take AA seriously, the numbers we get are in line with the numbers AA gave us in the mid-1950s (75% success rate among those whose who "really tried" AA).

It is true that AA has a high drop out rate, so I contend this: Have we found a treatment which is effective for alcoholics who, for whatever reason, refuse to work the AA program? Zemore 2019 makes a strong case that people in other programs can also stay clean and sober, as long as the goal is abstinence from alcohol. So, yes, I think people who will not work the AA program can still be clean and sober, but I also think it's dishonest to label all such people "failures" when measuring AA success.

The 1990 triennial survey: 26%, not 5% one year retention rate

There is an old myth that a 1990 AA Triennial survey shows a 5% one-year retention rate for Alcoholics Anonymous. This is a false belief: That graph, frequently brought up by anti-steppers with little interest in objective truth, actually shows a 26% retention rate, as pointed out in the article "Alcoholics Anonymous (AA) Recovery Outcome Rates".

The graph is not a membership retention graph; it is a simple count of, among the members in their first year, how many were around a given number of months.  If AA had a 100% retention rate, the graph would have shown 8.3% (i.e. 100% divided by 12) of members (instead of the 5% it shows) in their 12th month of sobriety.

The wording in the survey itself says it is a graph of "the number of members that have been around a given number of months"; it was not a graph of per-month membership retention.

This graph does not show an 80% first month drop out rate and it does not show a 5% one-year retention rate.  It shows a 70% first-month retention rate and a 26% one-year retention rate.

Here are a number of books and papers which claimed the 1990 Triennial survey showed a 5% retention rate before "Alcoholic Anonymous Recovery Outcome Rates" was published:

• McIntire, Don (2000). "How Well Does A.A. Work?". Alcoholism Treatment Quarterly. 18 (4): 1. doi:10.1300/J020v18n04_01. "at 12 months the figures are 5% remaining and 95% departed."
• Carl G. Lukefield, "Behavioral Therapy for Rural Substance Abusers", 2000 has this quote: "By 12 months, 90 percent have dropped out (McCrady & Miller, 1993)"
• Charles Bufe, AA: Cult or Cure 1997 has this quote Quote: "AA produced a large monograph, “Comments on A.A.’s Triennial Surveys,” that analyzed the results of all five surveys done to that point. [...] AA has a 95% new-member dropout rate during the first year of attendance."

However, after "Alcoholic Anonymous Recovery Outcome Rates" was published, a number of sources all of a sudden decided that the 1990 Membership retention survey showed a 26% one-year retention rate, which just happens to be the number in that paper. Even anti-AA polemics like Dodes' (poorly-argued) The Sober Truth now use the 26% figure. For example:

• Lance Dodes, M.D.; Zachary Dodes (2014). The Sober Truth: Debunking the Bad Science Behind 12-Step Programs and the Rehab Industry. ISBN 978-0807033159. "AA itself has published a comparable figure in a set of comments on its own thirteen-year internal survey, stating that only 26 percent of people who attend AA stay for longer than one year"
• Adrian, Manuella (2012). "Can Failure Carefully Observed Become a Springboard to Success?". Substance Use & Misuse. 47 (13–14): 1384. doi:10.3109/10826084.2012.706178. "the Alcoholics Anonymous triennial surveys from 1977 through 1989 found that one quarter (26%) of those who first attend an AA meeting are still attending after 1 year"

Point being, the conclusion in "Alcoholic Anonymous Recovery Outcome Rates" that those old surveys showed a 26% one-year retention rate are considered reliable among treatment experts.

As Slate Star Codex puts it: "Almost everyone’s belief about AA’s retention rate is off by a factor of five because one person long ago misread a really confusing graph and everyone else copied them without double-checking."

Dodes 2014: Bad science

Lance Dodes' book The Sober Truth is, bottom line, bad science. I have read it and compared what it says to the papers Dodes cites; in multiple cases, Dodes makes fundamental errors in his research. For example, Dodes claims that Fiorentine 1999 showed a 40% sobriety rate among active AA members; however, reading this study shows that it actually demonstrates a 75% sobriety rate. As another example, Dodes claims that Moos and Moos 2006 didn't look at how many active AA members stayed sober; actually Moos and Moos 2006 reports that 67% (or 77 people) were sober 16 years later.

I am not the only one who finds Dodes' reasoning flawed. The New York Times calls it a "polemical and deeply flawed book." The esteemed doctors Chad Emrick and Thomas Beresford, who find Dodes entire reasoning process questionable, looking at the second way Dodes tries to claim AA has a 5% success rate and say that "Albeit harsh to say, it appears that Dodes and Dodes did not let facts get in the way of drawing their second conclusion about AA’s effectiveness." The psychiatrists Jeffrey D. Roth and Edward J. Khantzian, in their review of The Sober Truth called Dodes' reasoning "pseudostatistical polemic." Addiction experts John Kelly and Gene Beresin also found Dodes' reasoning flawed.

For the record, the scientists criticizing Dodes' work come from the best schools: John Kelly, Eugene Beresin, and Edward J. Khantzian are all from Harvard. In addition, Jeffrey Roth graduated from Yale, and Thomas Beresford graduated from Stanford. I see a lot of very prestigious doctors strongly criticizing Dodes' work.

The outrage press (NPR, The Atlantic, Salon) placed far too much weight on the opinion of one doctor and his son even though so many other experts have looked at his reasoning and found it deeply flawed.

Dodes and Moos and Moos 2006

Since Moos and Moos 2006 invalidates a central thesis of Dodes' false claim that Alcoholics Anonymous has a 5% success rate, Dodes goes to some effort to try to discredit this key study.

After discussing how the people who were successful in Moos and Moos 2006 were compliant (which is just another way to word the "self selection bias" problem), Dodes writes this:

The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.

Dodes is flat out inaccurate here. The multiple Moos studies which use the same subjects, including Moos and Moos 2006, had a total of 628 people who said they would participate in the studies. Moos and Moos 2006 looks at 461 of them. We have a drop out rate (in other words, people who elected to be in the study but didn't follow up) of under 27%, yet Dodes is stating Moos/Moos 2006 had a drop out rate of 40%. It would appear Dodes is confusing the number of people who stopped going to meetings with the number of people who didn't do two or more follow-ups; while these are both "drop outs", they are very different figures.

What Moos and Moos measured was this: Of the 115 people who were heavily involved with AA in the first year, and who did two or more follow ups, 77 of them (67%) were sober 16 years later.

What about those 167 subjects in Moos and Moos 2006 who didn't do two or more follow ups? We can go back to Timko and Moos 1993, the one-year follow-up study which used the same subjects. Here, they look at 515 of those 628 people. Even when looking at a larger subset of subjects, we get the same level of people who tried AA: Moos and Moos 2006 has 269 of the 461 subjects studied who tried AA in their first year (that's 58%).  Timko and Moos 2003 has 294 of the 515 subjects trying AA in their first year (57%). It would appear that the drop outs who did not finish two or more surveys did not significantly affect the results.

Dodes continues:

The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up and also stayed in AA for longer than six months -- that is, the group on which the authors’ major findings are based -- was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group "had better outcomes."

Dodes, again, is not correctly reading Moos and Moos 2006. First of all, there were 115 (not 107) people heavily involved with AA in the first year. Moos and Moos 2006, in addition, looked very closely at the number of people who were sober 16 years later: "Individuals who received 27 weeks or more of [Alcoholics Anonymous] treatment in the first year were more likely to be abstinent and less likely to have drinking problems at 16 years than were individuals who remained untreated in the first year." In addition, Moos and Moos 2006 tells us exactly how many people in that group were sober 16 years later in Table 3: 67% of 115. I will save Dr. Dodes the bother of bringing out a calculator: That's 77 people. I can not see how Dodes can look at Moos and Moos 2006 and claim it did not state how many people who took AA seriously were sober 16 years later.

In terms of other factors, table 2 of Moos and Moos 2006 shows a strong correlation between AA treatment and improvement in both self-efficacy and social functioning. To quote Moos/Moos 2006: "a longer duration of AA was independently related to a higher likelihood of 16-year abstinence, enhanced self-efficacy, and good social functioning, and less likelihood of 16-year drinking problems."

As far as I can tell, Dodes did not even fully read Moos and Moos 2006 before writing up a polemic trying to discredit its numbers. I expected better.

Let's get some real figures for AA success:  Moos and Moos 2006 showed that 67% of the people who took AA seriously were sober 16 years later. If the self section issue is problematic and you want experimental studies, both Litt et al. 2009 and Walitzer 2009 show that correctly getting subjects in the rooms of 12-step meetings work better than other addiction treatments.  Bottom line, AA works.

Another bogus 5% success rate claim found in the wild

I have found another bogus 5% success rate claim in the wild, in the comments for this post:

http://www.patheos.com/blogs/secularoutpost/2016/11/04/pseudo-random-thoughts-on-aa-and-free-will/

The claim comes from a misreading of this passage in "Alcoholics Anonymous Effectiveness: Faith Meets Science":

As stated at the outset, the experimental evidence for AA effectiveness(addressing specificity) is the weakest among the six criteria considered crucial for establishing causation. Only two studies provided strong proof of a specific AA or TSF effect: the outpatient arm of Project MATCH (with effects at 1 and 3 years), and the intensive referral condition in Timko’s trial (with effects for abstinence at 6 months and 1 year).

The effect sizes were similar, with the TSF/Intensive referral
conditions having a 5-10% advantage in abstinence rates.

(Emphasis mine)

This passage uses a lot of specialized terminology and I can see why someone would misread it to mean that AA only has a 5-10% success rate. I would render the above paragraphs like this:

As we mentioned before, the evidence for the AA program helping alcoholics (and not other factors, such as the fact that people who take AA's program seriously simply being the people who want to get sober the most) is weak. Only two studies (Note: This was written before Humphreys 2014) provided strong proof that the AA program itself, or doctors trying to get alcoholics engaged in the AA program via TSF, is helping alcoholics.

The amount of effect we saw the program helping people was similar; when doctors tried really hard to get their patients to work the AA program (TSF), the patients were 5-10% more likely to stay sober.

In other words, whether or not an alcoholic works the AA program is a decision only the alcoholic can make. If a doctor tries really hard to get someone to work the program, the alcoholic is 5-10% more likely to get sober. This figure is not an overall success rate for AA. We know that, if the alcoholic himself decides to work the program, they have, depending on the study, between a 75% (Vaillant 1995; Fiorentine 1999) and 67% (Moos and Moos 2006) chance of staying sober.

A FAQ

A brief FAQ

• 1 Effectiveness of Alcoholics Anonymous
• 2 Other data that is inaccurate
  • 2.1 Randomized controlled trials
  • 2.2 Brandsma 1980
  • 2.3 AA is actually not rated #37 (sometimes #38) compared to other treatments
  • 2.4 5% success rate myth #1: The Sober Truth
  • 2.5 5% success rate myth #2: AA triennial survey
  • 2.6 5% success rate myth #3: Reading Vaillant poorly
• 3 Alcoholics can not moderate their drinking
  • 3.1 How the moderation myth tainted studies about AA
• 4 References

Effectiveness of Alcoholics Anonymous

For the 20%^[1][2][3] or so of alcoholics who become highly engaged with the program, Alcoholics Anonymous has about a 75%^[4][5] effectiveness rate.^[6]
For people who do not take AA as seriously, the number of alcoholics who achieve long-term sobriety is about 20%^[7], with the amount of people achieving sobriety directly correlated to AA participation^[8].
While some of the positive effects of the AA program keeping people sober can be attributed to self-selection bias, one 2014 study makes the case that AA's effectiveness shown in these studies does not come from self-selection^[9]; a 2016 study demonstrates that some of the specific effect is the prayers AA members say.^[10]

Other data that is inaccurate

Randomized controlled trials

Early randomized controlled trials were ineffective because the control group was not really a control group (people who wanted to stay sober went to AA anyway, even if they were not assigned to AA, and people assigned to AA did not necessarily take the program seriously), so their numbers can not be trusted.^[11]

Brandsma 1980

Anti-AA advocates claim that Brandsma 1980^[12] shows increased binge drinking among AA members. However 1) The study looked at about three dozen different findings at multiple follow-up periods, but this result was only seen in this one finding at one follow-up period 2) This result has not been replicated in recent studies^[13] even though 3) The study is 36 years old.^[14]

AA is actually not rated #37 (sometimes #38) compared to other treatments

Another common talking point popular with the anti-AA crowd is a table from Handbook of Alcoholism Treatment Approaches which ranks AA fairly lowly compared to other treatments. That table is, shall we say, weird: It rates acupuncture as one of the top 20 treatments of alcoholism, and gives motivational enhancement (#2 in the table), cognitive therapy (#13), and AA (#37) very different ratings, when the 2006 Cochrane study rate those as being about equal^[15]; other studies rate AA and/or TSF (Twelve Step Facilitation: teaching patients how to go to meetings and be a part of the AA culture) as being superior.^[16] No other reliably published meta-analysis comparing AA to other treatments has rated AA as being inferior.^[17]

5% success rate myth #1: The Sober Truth

Lance Dodes' The Sober Truth is a recent attempt to cook numbers to claim AA has a 5% success rate (for some reason, anti-AAs have a real obsession with coming up with a 5% success rate figure, facts be ignored). While this book was widely reported in the press in 2014, its claimed 5-8% success rate for AA does not come from any peer-reviewed studies, but is instead derived by multiplying numbers from different studies together. Beresford looks at the math Dodes performed to come up with these bogus figures, and explains why the math is invalid at https://www.ncadd.org/blogs/research-update/alcoholics-anonymous-call-for-better-science.

5% success rate myth #2: AA triennial survey

There is a chart in the 1990 AA triennial survey showing that 5% of the members in their first year happened to be in their 12th month, instead of the 8.3% we would expect if AA had a 100% success rate. This has frequently been misread to indicate that only 5% of members of AA stay sober one year. An actual analysis of the graph shows a 26% retention rate.^[3]

5% success rate myth #3: Reading Vaillant poorly

Some claim that Vaillant's book The Natural History of Alcoholism showed a 5% success rate, but that is a myth. The old green-papers site refuted it quite nicely: http://web.archive.org/web/20150329052802/http://green-papers.org/ "95% of patients had relapsed at some time during the study, even though many of these eventually attained sobriety. [...] It's well known that most severe alcoholics only get sober after many relapses, to the extent that relapses can be considered part of the recovery process. So don't pretend that's a failure. [...] And anyway, this was a study of a health network, not AA."

 

Alcoholics can not moderate their drinking

While not directly about AA's effectiveness, there is no reliable scientific evidence that alcoholics can drink moderately again. The studies that claim alcoholics can do so are based on self-reported information from alcoholics over period of a couple years or less. What is happening is that alcoholics have a pattern of being dishonest about their drinking, to the point that surveys using self-reported data from alcoholics are inaccurate. This has been seen time and time again: The Sobell Study from the early 1970s was shown to be a sham in 1982; the Rand report from 1976 was shown to be wrong in the 1980 follow-up; the NESARC study from 2001-2002 was a two-year study which never had a follow-up; and while Moderation Management's effectiveness has never been scientifically analyzed, its founder realized she could not control her drinking, but it was too late: Her runaway drinking resulted in her killing two people while driving drunk; she eventually killed herself.^[18]

How the moderation myth tainted studies about AA

A couple of surveys from the 1970s and early 1980s erroneously reported that AA makes people who drink again have worse relapses. The reason why these surveys were inaccurate is probably because they were based on self-reported data. Since people in AA are less likely to be dishonest about their drinking, they will self-report any drinking they engage in as being more chronic, while people not in AA are more likely to dishonestly report their drinking as being "moderate" or "under control" (e.g. lying about the number of drinks they have when filling out the survey).

References

1. The Natural History of Alcoholism Revisited by George E. Vaillant has a table showing that 19 out of 100 alcoholics attended 300 or more meetings over a 10-year period. The table is on page 197, and can be downloaded from http://web.archive.org/web/20150329052802/http://www.green-papers.org/Vaillant-Nat_hist_of_alcoholism_revisited-excerpts.pdf
2. This article shows a roughly 80% drop out rate in Brazil: Terra, Mauro Barbosa; Barros, Helena Maria Tannhauser; Stein, Airton Tetelbom; Figueira, Ivan; Palermo, Luiz Henrique; Athayde, Luciana Dias; Gonçalves, Marcelo de Souza; Da Silveira, Dartiu Xavier (2008). "Do Alcoholics Anonymous Groups Really Work? Factors of Adherence in a Brazilian Sample of Hospitalized Alcohol Dependents". American Journal on Addictions 17 (1): 48–53. doi:10.1080/10550490701756393. PMID 18214722.  "AA adherence was below 20%"
3. http://hindsfoot.org/recout01.pdf, while independently published and not peer-reviewed, performs statistical analysis of the unpublished 1990 triennial survey which anti-AA advocates frequently quote (since unsophisticated people reading it have incorrectly concluded it shows a 5% retention rate among AA members), showing that 26% of members surveyed make it from their first month to their twelfth month. If you insist on these figures in a more mainstream source, Dodes parrots that 26% figure in The Sober Truth.
4. That table in The Natural History of Alcoholism Revisited has, of those 19 alcoholics who took AA seriously, 14 who achieved long-term sobriety. While the actual 14 people figure is not on this table, 29 people * 48% is 14.
5. Beresford, Thomas (2016), Alcoholics Anonymous and The Atlantic: A Call For Better Science, National Council on Alcoholism and Drug Dependence, retrieved 2016-04-15  This page refers to Fiorentine 1999, pointing out that "the data Fiorentine presents is as follows: 77.7% of individuals who attended AA 12-step meetings at least weekly reported being free of drug use for 6 months prior to a 24-month follow-up, a finding corroborated by urinalysis at the time of the interview, and 74.8 % reported being free of alcohol use during the same time period"
6. Another study, which measures something slightly different, has similar high success figures for people who were highly engaged with Alcoholics Anonymous: Moos, Rudolf H.; Moos, BS (June 2006). "Participation in Treatment and Alcoholics Anonymous: A 16-Year Follow-Up of Initially Untreated Individuals". Journal of Clinical Psychology 62 (6): 735–750. doi:10.1002/jclp.20259. PMC 2220012. PMID 16538654. 
7. Of the 81 alcoholic subjects in The Natural History of Alcoholism Revisited who did not attend 300 AA meetings over a 10-year period, 15 achieved long-term sobriety
8. See the graphs in Kaskutas, Lee Ann (2009). "Alcoholics Anonymous Effectiveness: Faith Meets Science". Journal of Addictive Diseases 28 (2): 145–157. doi:10.1080/10550880902772464. PMC 2746426. PMID 19340677. 
9. Humphreys, Blodgett, Wagner (2014). "Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials". Alcoholism: Clinical and Experimental Research 38 (11): 2688–94. doi:10.1111/acer.12557. PMC 4285560. PMID 25421504.  CS1 maint: Multiple names: authors list (link)
10. Marc Galanter, Zoran Josipovic, Helen Dermatis, Jochen Weber, and Mary Alice Millard (2016-03-25). "An initial fMRI study on neural correlates of prayer in members of Alcoholics Anonymous". The American Journal of Drug and Alcohol Abuse. doi:10.3109/00952990.2016.1141912. Lay summary. 
11. There is some discussion of this in Kaskutas, Lee Ann (2009). "Alcoholics Anonymous Effectiveness: Faith Meets Science". Journal of Addictive Diseases 28 (2): 145–157. doi:10.1080/10550880902772464. PMC 2746426. PMID 19340677. 
12. Here's the proper citation for Brandsma 1980 Brandsma, Jeffery M; Maultsby, Maxie C; Welsh, Richard J (1980). Outpatient treatment of alcoholism: a review and comparative study. Baltimore, MD: University Park Press. ISBN 0-8391-1393-5. OCLC 5219646. 
13. While I haven't seen this cited by an anti-stepper in the wild, I should note this: Ogborne AC, Bornet A (1982). "Abstinence and abusive drinking among affiliates of Alcoholics Anonymous: are these the only alternatives?". Addict Behav. PMID 7102452.  This study, like Brandsma 1980, is a museum piece; the study uses data from the "Rand report", which is available for download and a table showing AA effectiveness there is on top of page 98 (page 114 in the PDF), showing fewer AA members who engage in "normal drinking." The notion that the Rand report showed alcoholics drinking normally again has since been discredited, see the blog entry at https://elplatt.com/return-moderate-drinking-still-lie for discussion; what that 1982 paper probably saw was that AA members are less likely to report their drinking as "normal drinking" and more likely to be honest about the negative consequences of their drinking
14. While not a reliable source, the best discussion I have found about Brandsma 1980 is at http://slatestarcodex.com/2014/10/26/alcoholics-anonymous-much-more-than-you-wanted-to-know/, in the beginning of section III. Note that this blog dismisses longitudinal studies too easily, but, then again, it was published before Humphreys, Blodgett, Wagner 2014 demonstrated a specific effect for AA, and before Galanter 2016 showed that prayer is part of that specific effect.
15. Ferri, Marcia; Amato, Laura; Davoli, Marina (19 July 2006). "Alcoholics Anonymous and other 12-step programmes for alcohol dependence". Cochrane Database of Systematic Reviews (3): CD005032. doi:10.1002/14651858.CD005032.pub2. PMID 16856072. 
16. Humphreys, Keith; Moos, Rudolf (May 2001). "Can encouraging substance abuse patients to participate in self-help groups reduce demand for health care? A quasi-experimental study". Alcoholism: Clinical and Experimental Research 1 (5): 711–716. doi:10.1111/j.1530-0277.2001.tb02271.x. PMID 11371720. 12-step patients had higher rates of abstinence at follow-up (45.7% versus 36.2% for patients from CB [cognitive-behavioral] programs, p < 0.001 
17. While not published by a mainstream press nor peer-reviewed, the blog entry at http://slatestarcodex.com/2014/10/26/alcoholics-anonymous-much-more-than-you-wanted-to-know/ devotes an entire section to this table and why it is inaccurate. Scroll down to section VI.
18. While, yes, a blog, a good discussion with references of how alcoholics can not moderate their drinking and why any study claiming otherwise is wrong is at the blog entry https://elplatt.com/return-moderate-drinking-still-lie

Salon's posts on AA

As I mentioned in my previous blog entry, with one exception, all of Salon's articles about AA have been negative. Let's look at Salon's list of AA articles:

• Pseudo-science and Bible-thumping, the worst of both worlds: The problem with rehab, AA and calling addiction a disease Negative, obviously
• Addiction is not a disease: How AA and 12-step programs erect barriers while attempting to relieve suffering Another negative one
• The drunkard’s loop: How one writer fought her way back from blackout alcoholism While this article mentions AA, it is not an article about AA
• AA is not a fraud: Why haters are afraid to listen Salon's one and only positive article about AA
• Debunking Alcoholics Anonymous: Behind the myths of recovery Here is one of the articles from Lance Dodes Salon published
• The pseudo-science of Alcoholics Anonymous: There’s a better way to treat addiction Salon likes Lance Dodes' bovine excrement about AA so much, they published two articles from him
• Is my friend suicidal? This article mentions AA but is not about AA
• My sober conversion to atheism While not as negative as, say, anything Lance Dodes ever wrote, this article is pretty negative about the AA program
• “Inside Rehab”: How it could work better, and why it doesn’t Surprisingly neutral when mentioning AA, but this is not an article about AA per se
• “Workaholism” is real This article only mentions AA in passing
• I was a sober college student Not directly about AA, but describes AA positively
• I didn’t even want these kids! Not directly about AA, but describes the 12-step program positively

In 2012 Salon was neutral, maybe even positive towards AA. By 2014, they took the Orange Papers kool-aid and were posting negative anti-AA polemics. If Salon is going to survive, they are going to have to be more positive about AA. They posted Lance Dodes' bovine excrement -- why don't they post Thomas Beresford's articles showing that Alcoholics Anonymous really works?