The bogus 5% success rate

Recently, someone asked how the “5% success rate” stat was made up. Here is my answer, edited to give more information and context.

To summarize, the real success rate, based on current science and randomized high quality studies, is 42%. The 5% stat has been discredited. There’s actually two primary places the discredited number comes from, as well as a third figure which I haven’t seen used for a long time.

For people who have long since forgotten their math, a numerator is the number we divide (the top of a fraction); the denominator is the amount we divide by (the bottom of a fraction). For example, 3/4 (.75 or 75%) has a numerator of three and a denominator of four.

The 1990 Triennial Survey

One is a 1990 survey of AA membership. It included a graph which was poorly labeled and confusing to read. People misread it, and thought it showed a 5% “success rate”, when, in fact, the graph shows a 26% one-year retention rate (PDF file).

Don McIntire, in a 2000 paper entitled “How Well Does A.A. Work?”, claimed, without evidence, that there was a 81% first month dropout rate, and that “at the end of 90 days, only 10% of newcomers are still present”. [1] This, despite the fact the original triennial survey directly contradicts him, stating that “about half those who come to A.A. are gone within three months.” There is nothing in the original 1990 triennial survey discussing an 81% one-month dropout rate; the number appears to come from people who didn't understand the graph and decided that's what it showed without any thought of how AA was supposed to measure the people who dropped out in their first month, since AA doesn't keep membership records nor meeting attendance records.

The actual numerator was the number of alcoholics in their 12th month sober; the denominator was all the alcoholics in their first year in the survey.

The Sober Truth

The other big source for the 5% figure comes from a 2014 book by Lance Dodes, The Sober Truth. What Lance Dodes did was look at a then 34-year-old study (and not the 8-year-old Moos and Moos 2006 study he was aware of—he mentions it elsewhere—but which has better numbers for AA success) and decide only people who both had 2.5 years or more sober and were still regularly attending AA were a “success”. Sober, but not regularly going to AA, not a “success”. Regularly going to AA, but only 2 years sober, also not a “success”. Rather disingenuous on Dodes’s part if you ask me, especially since Dodes ignored the fact that 1980 study showed that, the more people went to AA, the more likely they would be sober (e.g. 42% of regular AA attenders were sober one year or more in the graph Dodes used to calculate his 5% figure, compared to 16% for people who never went to AA). [2]

The numerator was the alcoholics in a survey both regularly going to AA and with over 2.5 years sober; the denominator was all of the people in the survey who went to at least one AA meeting.

Reading Vaillant poorly

This hasn’t been attributed to Vaillant since about 2015 or so; it is usually described as being a “Harvard study” which supposedly shows a 95% recidivism rate.

The old Orange Papers site claimed that Vaillant's book The Natural History of Alcoholism Revisited showed a 5% success rate, but that is a myth. The old green-papers site from the same era refuted it quite nicely: http://web.archive.org/web/20150329052802/http://green-papers.org/ “95% of patients had relapsed at some time during the study, even though many of these eventually attained sobriety. [...] It's well known that most severe alcoholics only get sober after many relapses, to the extent that relapses can be considered part of the recovery process. So don't pretend that's a failure. [...] And anyway, this was a study of a health network, not AA.”

The numerator was the number of people who never relapsed, not even once. The denominator was all of the people in the study, regardless of whether they went to AA.

Cochrane 2020: 42% success rate

Now that we have gone over three discredited figures, let’s show some more accurate figures for AA’s success rate. The 2020 Cochrane Report on Alcoholics Anonymous shows a 42% success rate for AA-centered therapies, compared to 35% for the other therapies. 

The numerator here is all of the people who were abstinent from alcohol one year after the survey was started; the denominator is people serious enough about their sobriety to get regular therapy.

Footnotes

 

[1] To be fair here, the triennial survey was initially incorrectly read in Bufe’s 1991 book Cult or Cure (second edition published in 1997; not to be confused with Vaillant’s 2005 paper with the same name) and Fox’s 1993 book Addiction, Change and Choice, both anti-AA polemics published by See Sharp Press, a publisher of books about “anarchism and atheism”, hardly a medical source nor academic journal. McIntire, however, gets the dubious honor of being the first one to publish this erroneous data in an academic journal. McIntire did point out that, if we only count people who stay in AA for 90 days, we get around a 55% one year retention rate. 

[2] Polich, Armor, and Braiker The course of Alcoholism: Four Years after Treatment. Rand 1980. Page 129

How The Sober Truth came up with its bogus 5% figure

While there hasn’t been any mainstream press criticizing AA’s efficacy recently, I think, while Dodes’s figures are now outdated and discredited in light of Cochrane 2020, it would serve the reader to know the exact source data Dodes used to synthesize his bogus 5% “success rate” figure for Alcoholics Anonymous (AA).

A few years ago, Dr. Thomas Beresford had this to say about Lance Dodes’s alleged 5% success rate for Alcoholics Anonymous:

The Atlantic article’s source [i.e. Lance Dodes’s book] multiplies a 25% AA attendance figure by a 22% abstinence figure to arrive at a 5.5% estimate of AA’s effectiveness. Where do these figures come from? Another second-hand source [Note: This blog post will detail that second-hand source below] that also cites the work of others: two publications from the Rand Corporation that examined, among other things, attempts at controlled drinking and offered little focus on AA’s effectiveness. At 4-year follow-up the Rand group identified patients with at least one year abstinence who had been regular members of AA 18 months after the start of treatment: 42% of the regular AA members were abstinent, not the “calculated” 5.5% figure.

It may benefit the reader to dive in to this more deeply. Here is how Lance Dodes describes AA as supposedly having a 5% success rate:

University of California professor Herbert Fingarette cited two [...] statistics: at eighteen months, 25 percent of people still attended AA, and of those who did attend, 22 percent consistently maintained sobriety. [Reference: H. Fingarette, Heavy Drinking: The Myth of Alcoholism as a Disease (Berkeley: University of California Press, 1988)] Taken together, these numbers show that about 5.5 percent of all those who started with AA became sober members.

And here is how the late Herbert Fingarette, back in 1988, described A.A.’s success rate (as cited by Dodes above):

[...] one large-scale study of alcoholics in treatment centers in the U.S. found that 71 percent of their subjects had attended A.A. at some point; but at eighteen-month and subsequent followups only 14 percent to 18 percent were attending. Moreover, the rate of problems was higher for irregular A.A. attenders than for either regular attenders or non-attenders. Among those who were regular A.A. attenders at the time of the initial interview, only 22 percent consistently maintained abstention up through the thirty-month followup interview, and over 33 percent had not only returned to drinking but also showed alcohol-related physical symptoms and life problems.

This paragraph has a single footnote, the 1980 Rand report on alcoholism. The 1980 Rand Report can be readily downloaded. Let’s start of with a summary of a graph on page 154 of that report.

Never attended AA: 23 out of 129 (18%) sober over 6 months.
Some AA attendance: 85 out of 315 (27%) sober over 6 months
Lots of AA attendance: 42 out of 74 (57%)

The Rand report, as skeptical as it is of AA (remember, they were trying to determine if alcoholics could moderately drink again, something we know today is a fool’s errand), admits that the abstinence rate for regular AA attenders is higher than any other group, with a p score of under .05.

The 22% and 18% figures Beresford then Dodes used for the bogus 5.5% figure, are based on table 6.15 at the bottom of page 129 of that 1980 Rand report. This is a four year follow up, with four columns based on 18-month followup AA attendance (never, only in the past, only occasionally, and regular attendance). They also looked at the number of people with over 6 months sober, over one year sober, and with 2.5 years sober. 

As expected, the more people went to AA, the more likely they would be sober. In more detail:

AA attendance	6 months+ sober	1 year+ sober	2.5+ years
Never	24%	16%	11%
Only in past	25%	19%	12%
Occasionally	29%	19%	12%
Regularly	45%	42%	22%

Now, as we can see, overall, not that many people managed to get 2.5 years sober, but those that went to AA were more likely to stay sober. As it turns out, only 18% of the people were regular AA attenders (Dodes, mysteriously, claims the number is 25%); of those, only 22% got sober. 

So, Dodes’s 5.5% number is based on this Rand report, and it is both:

• The people who regularly went to AA meetings. People who did not regularly attend AA meetings were not considered an “AA success” in Dodes’s book, regardless of whether they actually got sober.
• Of those people, only the people who did not have one drink for over 2.5 years were considered a success. If someone had a single drink two years ago, but was sober for over a year, regularly attending AA meetings, they were still considered to be not an “AA success”.

By doing everything to minimize the number of people that are an “AA success”, he was able to come up with a really low number for AA success. Indeed, if we make the bar more reasonable, such as one year or more sober, we get an 8% (instead of 4%) success rate, again not counting the people who didn’t regularly attend AA but got sober anyway.

Why did Dodes use a 1980 study based on 1970s numbers in a 2014 book? The numbers in the 1980 study were remarkably low. If we play the same game with the numbers in Moos and Moos 2006, we get a 17% success rate for AA, again not counting everyone who got sober who didn’t regularly go to AA meetings. 

Even if we use this old 1980 study, if we use the numbers on page 128 (not page 129) of the 1980 Rand report, we get a 8% success rate (if only regular attenders count), or a 24% success rate if anyone who has set foot in AA counts. Actually, we get a 57% success rate if we only use the number of people regularly going to AA as our denominator (instead of all the people in the study).

Point blank: The math Dodes used to come up with a 5.5% AA success rate are based on a study that was 34 years old when he wrote the book, even though another similar 8-year-old study was available, and use a kind of math whose goal appears to be to make AA look as bad as possible.

Dodes’s numbers were disingenuous, plain and simple.

As an aside, since that old 1980 Rand report claimed that some AA attendance resulted in worse drinking consequences than no AA attendance, something Fingarette observed, I will address that. Those numbers were based on self-reporting answers. Based on the fact that the studies used in the 2020 Cochrane report on AA do not see worse drinking consequences for people randomly assigned to AA-centered therapies, the most likely reason that old Rand report saw worse consequences is because people who never have gone to AA are more likely to be dishonest about the extent of negative consequences from their drinking. As I have heard countless people say in AA meetings, doctors could not properly treat us because we were so dishonest about our drinking before going to AA.