The bogus 5% success rate

Recently, someone asked how the “5% success rate” stat was made up. Here is my answer, edited to give more information and context.

To summarize, the real success rate, based on current science and randomized high quality studies, is 42%. The 5% stat has been discredited. There’s actually two primary places the discredited number comes from, as well as a third figure which I haven’t seen used for a long time.

For people who have long since forgotten their math, a numerator is the number we divide (the top of a fraction); the denominator is the amount we divide by (the bottom of a fraction). For example, 3/4 (.75 or 75%) has a numerator of three and a denominator of four.

The 1990 Triennial Survey

One is a 1990 survey of AA membership. It included a graph which was poorly labeled and confusing to read. People misread it, and thought it showed a 5% “success rate”, when, in fact, the graph shows a 26% one-year retention rate (PDF file).

Don McIntire, in a 2000 paper entitled “How Well Does A.A. Work?”, claimed, without evidence, that there was a 81% first month dropout rate, and that “at the end of 90 days, only 10% of newcomers are still present”. [1] This, despite the fact the original triennial survey directly contradicts him, stating that “about half those who come to A.A. are gone within three months.” There is nothing in the original 1990 triennial survey discussing an 81% one-month dropout rate; the number appears to come from people who didn't understand the graph and decided that's what it showed without any thought of how AA was supposed to measure the people who dropped out in their first month, since AA doesn't keep membership records nor meeting attendance records.

The actual numerator was the number of alcoholics in their 12th month sober; the denominator was all the alcoholics in their first year in the survey.

The Sober Truth

The other big source for the 5% figure comes from a 2014 book by Lance Dodes, The Sober Truth. What Lance Dodes did was look at a then 34-year-old study (and not the 8-year-old Moos and Moos 2006 study he was aware of—he mentions it elsewhere—but which has better numbers for AA success) and decide only people who both had 2.5 years or more sober and were still regularly attending AA were a “success”. Sober, but not regularly going to AA, not a “success”. Regularly going to AA, but only 2 years sober, also not a “success”. Rather disingenuous on Dodes’s part if you ask me, especially since Dodes ignored the fact that 1980 study showed that, the more people went to AA, the more likely they would be sober (e.g. 42% of regular AA attenders were sober one year or more in the graph Dodes used to calculate his 5% figure, compared to 16% for people who never went to AA). [2]

The numerator was the alcoholics in a survey both regularly going to AA and with over 2.5 years sober; the denominator was all of the people in the survey who went to at least one AA meeting.

Reading Vaillant poorly

This hasn’t been attributed to Vaillant since about 2015 or so; it is usually described as being a “Harvard study” which supposedly shows a 95% recidivism rate.

The old Orange Papers site claimed that Vaillant's book The Natural History of Alcoholism Revisited showed a 5% success rate, but that is a myth. The old green-papers site from the same era refuted it quite nicely: http://web.archive.org/web/20150329052802/http://green-papers.org/ “95% of patients had relapsed at some time during the study, even though many of these eventually attained sobriety. [...] It's well known that most severe alcoholics only get sober after many relapses, to the extent that relapses can be considered part of the recovery process. So don't pretend that's a failure. [...] And anyway, this was a study of a health network, not AA.”

The numerator was the number of people who never relapsed, not even once. The denominator was all of the people in the study, regardless of whether they went to AA.

Cochrane 2020: 42% success rate

Now that we have gone over three discredited figures, let’s show some more accurate figures for AA’s success rate. The 2020 Cochrane Report on Alcoholics Anonymous shows a 42% success rate for AA-centered therapies, compared to 35% for the other therapies. 

The numerator here is all of the people who were abstinent from alcohol one year after the survey was started; the denominator is people serious enough about their sobriety to get regular therapy.

Footnotes

 

[1] To be fair here, the triennial survey was initially incorrectly read in Bufe’s 1991 book Cult or Cure (second edition published in 1997; not to be confused with Vaillant’s 2005 paper with the same name) and Fox’s 1993 book Addiction, Change and Choice, both anti-AA polemics published by See Sharp Press, a publisher of books about “anarchism and atheism”, hardly a medical source nor academic journal. McIntire, however, gets the dubious honor of being the first one to publish this erroneous data in an academic journal. McIntire did point out that, if we only count people who stay in AA for 90 days, we get around a 55% one year retention rate. 

[2] Polich, Armor, and Braiker The course of Alcoholism: Four Years after Treatment. Rand 1980. Page 129

How The Sober Truth came up with its bogus 5% figure

While there hasn’t been any mainstream press criticizing AA’s efficacy recently, I think, while Dodes’s figures are now outdated and discredited in light of Cochrane 2020, it would serve the reader to know the exact source data Dodes used to synthesize his bogus 5% “success rate” figure for Alcoholics Anonymous (AA).

A few years ago, Dr. Thomas Beresford had this to say about Lance Dodes’s alleged 5% success rate for Alcoholics Anonymous:

The Atlantic article’s source [i.e. Lance Dodes’s book] multiplies a 25% AA attendance figure by a 22% abstinence figure to arrive at a 5.5% estimate of AA’s effectiveness. Where do these figures come from? Another second-hand source [Note: This blog post will detail that second-hand source below] that also cites the work of others: two publications from the Rand Corporation that examined, among other things, attempts at controlled drinking and offered little focus on AA’s effectiveness. At 4-year follow-up the Rand group identified patients with at least one year abstinence who had been regular members of AA 18 months after the start of treatment: 42% of the regular AA members were abstinent, not the “calculated” 5.5% figure.

It may benefit the reader to dive in to this more deeply. Here is how Lance Dodes describes AA as supposedly having a 5% success rate:

University of California professor Herbert Fingarette cited two [...] statistics: at eighteen months, 25 percent of people still attended AA, and of those who did attend, 22 percent consistently maintained sobriety. [Reference: H. Fingarette, Heavy Drinking: The Myth of Alcoholism as a Disease (Berkeley: University of California Press, 1988)] Taken together, these numbers show that about 5.5 percent of all those who started with AA became sober members.

And here is how the late Herbert Fingarette, back in 1988, described A.A.’s success rate (as cited by Dodes above):

[...] one large-scale study of alcoholics in treatment centers in the U.S. found that 71 percent of their subjects had attended A.A. at some point; but at eighteen-month and subsequent followups only 14 percent to 18 percent were attending. Moreover, the rate of problems was higher for irregular A.A. attenders than for either regular attenders or non-attenders. Among those who were regular A.A. attenders at the time of the initial interview, only 22 percent consistently maintained abstention up through the thirty-month followup interview, and over 33 percent had not only returned to drinking but also showed alcohol-related physical symptoms and life problems.

This paragraph has a single footnote, the 1980 Rand report on alcoholism. The 1980 Rand Report can be readily downloaded. Let’s start of with a summary of a graph on page 154 of that report.

Never attended AA: 23 out of 129 (18%) sober over 6 months.
Some AA attendance: 85 out of 315 (27%) sober over 6 months
Lots of AA attendance: 42 out of 74 (57%)

The Rand report, as skeptical as it is of AA (remember, they were trying to determine if alcoholics could moderately drink again, something we know today is a fool’s errand), admits that the abstinence rate for regular AA attenders is higher than any other group, with a p score of under .05.

The 22% and 18% figures Beresford then Dodes used for the bogus 5.5% figure, are based on table 6.15 at the bottom of page 129 of that 1980 Rand report. This is a four year follow up, with four columns based on 18-month followup AA attendance (never, only in the past, only occasionally, and regular attendance). They also looked at the number of people with over 6 months sober, over one year sober, and with 2.5 years sober. 

As expected, the more people went to AA, the more likely they would be sober. In more detail:

AA attendance	6 months+ sober	1 year+ sober	2.5+ years
Never	24%	16%	11%
Only in past	25%	19%	12%
Occasionally	29%	19%	12%
Regularly	45%	42%	22%

Now, as we can see, overall, not that many people managed to get 2.5 years sober, but those that went to AA were more likely to stay sober. As it turns out, only 18% of the people were regular AA attenders (Dodes, mysteriously, claims the number is 25%); of those, only 22% got sober. 

So, Dodes’s 5.5% number is based on this Rand report, and it is both:

• The people who regularly went to AA meetings. People who did not regularly attend AA meetings were not considered an “AA success” in Dodes’s book, regardless of whether they actually got sober.
• Of those people, only the people who did not have one drink for over 2.5 years were considered a success. If someone had a single drink two years ago, but was sober for over a year, regularly attending AA meetings, they were still considered to be not an “AA success”.

By doing everything to minimize the number of people that are an “AA success”, he was able to come up with a really low number for AA success. Indeed, if we make the bar more reasonable, such as one year or more sober, we get an 8% (instead of 4%) success rate, again not counting the people who didn’t regularly attend AA but got sober anyway.

Why did Dodes use a 1980 study based on 1970s numbers in a 2014 book? The numbers in the 1980 study were remarkably low. If we play the same game with the numbers in Moos and Moos 2006, we get a 17% success rate for AA, again not counting everyone who got sober who didn’t regularly go to AA meetings. 

Even if we use this old 1980 study, if we use the numbers on page 128 (not page 129) of the 1980 Rand report, we get a 8% success rate (if only regular attenders count), or a 24% success rate if anyone who has set foot in AA counts. Actually, we get a 57% success rate if we only use the number of people regularly going to AA as our denominator (instead of all the people in the study).

Point blank: The math Dodes used to come up with a 5.5% AA success rate are based on a study that was 34 years old when he wrote the book, even though another similar 8-year-old study was available, and use a kind of math whose goal appears to be to make AA look as bad as possible.

Dodes’s numbers were disingenuous, plain and simple.

As an aside, since that old 1980 Rand report claimed that some AA attendance resulted in worse drinking consequences than no AA attendance, something Fingarette observed, I will address that. Those numbers were based on self-reporting answers. Based on the fact that the studies used in the 2020 Cochrane report on AA do not see worse drinking consequences for people randomly assigned to AA-centered therapies, the most likely reason that old Rand report saw worse consequences is because people who never have gone to AA are more likely to be dishonest about the extent of negative consequences from their drinking. As I have heard countless people say in AA meetings, doctors could not properly treat us because we were so dishonest about our drinking before going to AA.

Some other recent criticism of AA in the press

I have found other criticism of AA in the mainstream press over the last year which I should address here in this blog.

Naltrexone redux

A recent New York Times opinion piece (archive link) brought out the same old tired “AA (or abstinence) doesn’t work but Naltrexone allows alcoholics to drink like gentlemen” chestnut.

While cherry picked studies show remarkably high success rates for Naltrexone, this is offset by other studies which do not replicate those results. Naltrexone, like AA, appears to be effective for a subset of alcoholics, but, as per Cochrane, it only helps about 11% of alcoholics. 

The most disturbing thing about studies showing Naltrexone showing controlled drinking among alcoholics is that they consistently do not have long term follow-up periods. For example, that New York Times opinion piece links to two different studies which supposedly show alcoholics successfully drinking moderately again. In both cases, the final follow-up is one year after treatment.

As someone who remembers Pendery 1982 all too well, positive results for moderate drinking without long term follow-ups are not very encouraging. Sure, anything someone throws a dead cat, they will hit a study of alcoholism showing alcoholics successfully engaging in “moderate drinking” over the short term, but the few long-term follow-ups I have seen show that moderate drinking is not a long-term solution to chronic alcoholism.

As a point of comparison, the 1990s Project MATCH randomized study showed increased abstinence for people who engaged in AA-centered therapies three years after treatment (as per its data as presented in Cochrane 2020), and Moos and Moos 2006 is a longitudinal study showing that Alcoholics Anonymous keeps alcoholics sober 16 years later.

Another thing: That New York Times opinion piece claims that only 25% of alcoholics will achieve abstinence, but the study it links to supporting that outdated 25% figure doesn’t actually conclude this; it claims a 25% success rate for abstinence, and links to an old article from 2005 using results that are nearly two decades old. More recent figures from Cochrane show a 42% success rate.

Abstinence violation effect

There have been two (open access) different letters (paywall) recently published which make the same criticism of Cochrane 2020: They claim that there is an “abstinence violation effect” where the subjects undergoing AA-based treatment who are not abstinent somehow do worse when they drink again than subjects not undergoing AA-based treatment. In other words, the accusation is that AA may increase abstinence, but overall AA doesn’t improve things for alcoholics, because the alcoholics not abstinent supposedly drink more heavily.

This “abstinence violation effect” is actually a myth, as one of the authors of the 2020 Cochrane review on AA explains. Since this letter is paywalled, I will quote the most important part of that letter here:

while more individuals in AA/TSF achieved continuous abstinence, those who were not completely abstinent did not drink more heavily, drink more frequently or experience more alcohol-related consequences

Point being, AA overall improves things for alcoholics. 42% are straight up abstinent from alcohol a year later, and those who do not achieve abstinence do not, repeat not have worse consequences from their drinking.

I will only respond to articles published by the mainstream media

Moving forward, I will only respond to new articles published by the mainstream media which falsely state Alcoholics Anonymous does not work, or link to sources that falsely state that Alcoholics Anonymous is not effective. For example, the Vice article I criticized yesterday was published by a mainstream publisher, and while it did acknowledge Cochrane 2020, it also linked directly to sources which falsely claim AA doesn’t help alcoholics. 

My criteria is: “Is this a source notable enough to have its article linked to by mainstream news aggregators?” In other words, stuff published by mainstream newspapers and magazines. Sources which do not meet that bar will be ignored: social media postings, self-published blogs, postings in comment sections (here in the 2020s, comment sections are not as popular as they were a decade ago; good riddance to bad rubbish), and so on will be cheerfully ignored. 

If a new peer reviewed study making very negative claims about AA’s efficacy were to be published, I would address it here. Remember: Metastudies from top-notch publisher like the Cochrane 2020 review on AA’s effectiveness trump a single study, even if it’s peer-reviewed.

I am aware and acknowledge AA is not for everyone, and that other treatments appear to be just as effective as AA, but, for most alcoholics, AA is the first place they will go seeking treatment and trying to get better, and the goal of this blog is to make sure that alcoholic still suffering from the disease is properly informed that AA can very well help them get sober and better.

I now have a Twitter account, which I will primarily use to engage with authors of pieces which unfairly criticize AA. 

Goodbye for now. This account will only be made active again the when and if I see a new anti-AA article from a reliable and mainstream source.

Vice posted an anti-AA screed

Over a year ago, I retired this blog. When I retired the blog, I noted “I would restart this blog again if inaccurate claims about AA efficacy made it back to the mainstream press”.

Well, wouldn’t you know it, Vice has posted an anti-AA screed (archive link) in the UK edition of their magazine.

First of all, Vice is considered, as I type this, a source of questionable reliability in the Wikipedia. Concerning the quality of the article they posted about AA, I can see why their reliability is not top notch.

Near the beginning of the article, they link to two anti-AA Facebook groups; one has, as I type this, about 1,600 members; the other has around 1,300 members. Compare that to the one AA support group with about 41,500 members and another unofficial group simply called Alcoholics Anonymous with about 39,300 members, we see that the there are 24-30 people getting better in AA for each person who wastes their time complaining about the program and not improving. Considering how much more popular the AA-positive support groups are, it’s undue weight to focus only on the anti-AA groups.

They then look at the AA documentary The 13th Step without any critical evaluation of its contents. This “documentary” (which was a complete flop without little to no notability) makes the accusation that women will be forced to endure sexual harassment if they wish to get sober in AA. That may be true of a few isolated fellowships (such as the infamous Midtown fellowship well over a decade ago), but the general rule is this:

• Alcoholics Anonymous has safe spaces for women: A number of meetings are meetings which only women can attend.
• There is a strong cultural rule that men stick with men and women stick with women in the AA fellowship, in order to minimize women getting unwanted sexual advances.
• Many sponsors and people in the rooms of AA suggest people stay out of relationships (i.e. have sex) in their first year of sobriety; this again is for the protection of newcomer women.
• The fellowships I have been to will confront men who make inappropriate advances towards women.
  

Women have plenty of options to avoid men and harassment in the fellowships I have been to.

Next, the article goes on to mentions a book written by one Steven Slate, who makes a number of dubious claims about AA’s effectiveness. For example, he quotes a study published over 40 years ago to make negative claims about AA’s efficacy, without acknowledging that the study has been discredited, something I have extensively discussed on this blog. As another example, Mr. Slate has not even acknowledged the existence of the Cochrane 2020 study which shows that Alcoholics Anonymous is more effective at achieving abstinence for alcoholics than other treatments.

Indeed, the Vice article does acknowledge that “AA definitely works” for a significant subset of alcoholics, but only in the final paragraph.

The fact of the matter is this: AA works most of the time for people who work it. However, alcoholics are a stubborn lot, and many of them will find excuses for not going to meetings, not working the steps, and not getting better: Complaining about sexual harassment (so why not only go to women’s AA meetings) or falsely claiming that AA does not help alcoholics.

It’s unfortunate that Vice media has chosen to publish an article which encourages alcoholics to come up with more excuses to not go to AA and more excuses to not get better.

Update: The author the this Vice article has responded on Twitter: “Hi! Thanks for responding. This article isn’t meant to downplay AAs significance in helping millions get sober. It’s just an alternative perspective that explains why some people left during the pandemic. It’s good to offer alternative solutions so such complex problems!”