How The Sober Truth came up with its bogus 5% figure

While there hasn’t been any mainstream press criticizing AA’s efficacy recently, I think, while Dodes’s figures are now outdated and discredited in light of Cochrane 2020, it would serve the reader to know the exact source data Dodes used to synthesize his bogus 5% “success rate” figure for Alcoholics Anonymous (AA).

A few years ago, Dr. Thomas Beresford had this to say about Lance Dodes’s alleged 5% success rate for Alcoholics Anonymous:

The Atlantic article’s source [i.e. Lance Dodes’s book] multiplies a 25% AA attendance figure by a 22% abstinence figure to arrive at a 5.5% estimate of AA’s effectiveness. Where do these figures come from? Another second-hand source [Note: This blog post will detail that second-hand source below] that also cites the work of others: two publications from the Rand Corporation that examined, among other things, attempts at controlled drinking and offered little focus on AA’s effectiveness. At 4-year follow-up the Rand group identified patients with at least one year abstinence who had been regular members of AA 18 months after the start of treatment: 42% of the regular AA members were abstinent, not the “calculated” 5.5% figure.

It may benefit the reader to dive in to this more deeply. Here is how Lance Dodes describes AA as supposedly having a 5% success rate:

University of California professor Herbert Fingarette cited two [...] statistics: at eighteen months, 25 percent of people still attended AA, and of those who did attend, 22 percent consistently maintained sobriety. [Reference: H. Fingarette, Heavy Drinking: The Myth of Alcoholism as a Disease (Berkeley: University of California Press, 1988)] Taken together, these numbers show that about 5.5 percent of all those who started with AA became sober members.

And here is how the late Herbert Fingarette, back in 1988, described A.A.’s success rate (as cited by Dodes above):

[...] one large-scale study of alcoholics in treatment centers in the U.S. found that 71 percent of their subjects had attended A.A. at some point; but at eighteen-month and subsequent followups only 14 percent to 18 percent were attending. Moreover, the rate of problems was higher for irregular A.A. attenders than for either regular attenders or non-attenders. Among those who were regular A.A. attenders at the time of the initial interview, only 22 percent consistently maintained abstention up through the thirty-month followup interview, and over 33 percent had not only returned to drinking but also showed alcohol-related physical symptoms and life problems.

This paragraph has a single footnote, the 1980 Rand report on alcoholism. The 1980 Rand Report can be readily downloaded. Let’s start of with a summary of a graph on page 154 of that report.

Never attended AA: 23 out of 129 (18%) sober over 6 months.
Some AA attendance: 85 out of 315 (27%) sober over 6 months
Lots of AA attendance: 42 out of 74 (57%)

The Rand report, as skeptical as it is of AA (remember, they were trying to determine if alcoholics could moderately drink again, something we know today is a fool’s errand), admits that the abstinence rate for regular AA attenders is higher than any other group, with a p score of under .05.

The 22% and 18% figures Beresford then Dodes used for the bogus 5.5% figure, are based on table 6.15 at the bottom of page 129 of that 1980 Rand report. This is a four year follow up, with four columns based on 18-month followup AA attendance (never, only in the past, only occasionally, and regular attendance). They also looked at the number of people with over 6 months sober, over one year sober, and with 2.5 years sober. 

As expected, the more people went to AA, the more likely they would be sober. In more detail:

AA attendance	6 months+ sober	1 year+ sober	2.5+ years
Never	24%	16%	11%
Only in past	25%	19%	12%
Occasionally	29%	19%	12%
Regularly	45%	42%	22%

Now, as we can see, overall, not that many people managed to get 2.5 years sober, but those that went to AA were more likely to stay sober. As it turns out, only 18% of the people were regular AA attenders (Dodes, mysteriously, claims the number is 25%); of those, only 22% got sober. 

So, Dodes’s 5.5% number is based on this Rand report, and it is both:

• The people who regularly went to AA meetings. People who did not regularly attend AA meetings were not considered an “AA success” in Dodes’s book, regardless of whether they actually got sober.
• Of those people, only the people who did not have one drink for over 2.5 years were considered a success. If someone had a single drink two years ago, but was sober for over a year, regularly attending AA meetings, they were still considered to be not an “AA success”.

By doing everything to minimize the number of people that are an “AA success”, he was able to come up with a really low number for AA success. Indeed, if we make the bar more reasonable, such as one year or more sober, we get an 8% (instead of 4%) success rate, again not counting the people who didn’t regularly attend AA but got sober anyway.

Why did Dodes use a 1980 study based on 1970s numbers in a 2014 book? The numbers in the 1980 study were remarkably low. If we play the same game with the numbers in Moos and Moos 2006, we get a 17% success rate for AA, again not counting everyone who got sober who didn’t regularly go to AA meetings. 

Even if we use this old 1980 study, if we use the numbers on page 128 (not page 129) of the 1980 Rand report, we get a 8% success rate (if only regular attenders count), or a 24% success rate if anyone who has set foot in AA counts. Actually, we get a 57% success rate if we only use the number of people regularly going to AA as our denominator (instead of all the people in the study).

Point blank: The math Dodes used to come up with a 5.5% AA success rate are based on a study that was 34 years old when he wrote the book, even though another similar 8-year-old study was available, and use a kind of math whose goal appears to be to make AA look as bad as possible.

Dodes’s numbers were disingenuous, plain and simple.

As an aside, since that old 1980 Rand report claimed that some AA attendance resulted in worse drinking consequences than no AA attendance, something Fingarette observed, I will address that. Those numbers were based on self-reporting answers. Based on the fact that the studies used in the 2020 Cochrane report on AA do not see worse drinking consequences for people randomly assigned to AA-centered therapies, the most likely reason that old Rand report saw worse consequences is because people who never have gone to AA are more likely to be dishonest about the extent of negative consequences from their drinking. As I have heard countless people say in AA meetings, doctors could not properly treat us because we were so dishonest about our drinking before going to AA.

Some other recent criticism of AA in the press

I have found other criticism of AA in the mainstream press over the last year which I should address here in this blog.

Naltrexone redux

A recent New York Times opinion piece (archive link) brought out the same old tired “AA (or abstinence) doesn’t work but Naltrexone allows alcoholics to drink like gentlemen” chestnut.

While cherry picked studies show remarkably high success rates for Naltrexone, this is offset by other studies which do not replicate those results. Naltrexone, like AA, appears to be effective for a subset of alcoholics, but, as per Cochrane, it only helps about 11% of alcoholics. 

The most disturbing thing about studies showing Naltrexone showing controlled drinking among alcoholics is that they consistently do not have long term follow-up periods. For example, that New York Times opinion piece links to two different studies which supposedly show alcoholics successfully drinking moderately again. In both cases, the final follow-up is one year after treatment.

As someone who remembers Pendery 1982 all too well, positive results for moderate drinking without long term follow-ups are not very encouraging. Sure, anything someone throws a dead cat, they will hit a study of alcoholism showing alcoholics successfully engaging in “moderate drinking” over the short term, but the few long-term follow-ups I have seen show that moderate drinking is not a long-term solution to chronic alcoholism.

As a point of comparison, the 1990s Project MATCH randomized study showed increased abstinence for people who engaged in AA-centered therapies three years after treatment (as per its data as presented in Cochrane 2020), and Moos and Moos 2006 is a longitudinal study showing that Alcoholics Anonymous keeps alcoholics sober 16 years later.

Another thing: That New York Times opinion piece claims that only 25% of alcoholics will achieve abstinence, but the study it links to supporting that outdated 25% figure doesn’t actually conclude this; it claims a 25% success rate for abstinence, and links to an old article from 2005 using results that are nearly two decades old. More recent figures from Cochrane show a 42% success rate.

Abstinence violation effect

There have been two (open access) different letters (paywall) recently published which make the same criticism of Cochrane 2020: They claim that there is an “abstinence violation effect” where the subjects undergoing AA-based treatment who are not abstinent somehow do worse when they drink again than subjects not undergoing AA-based treatment. In other words, the accusation is that AA may increase abstinence, but overall AA doesn’t improve things for alcoholics, because the alcoholics not abstinent supposedly drink more heavily.

This “abstinence violation effect” is actually a myth, as one of the authors of the 2020 Cochrane review on AA explains. Since this letter is paywalled, I will quote the most important part of that letter here:

while more individuals in AA/TSF achieved continuous abstinence, those who were not completely abstinent did not drink more heavily, drink more frequently or experience more alcohol-related consequences

Point being, AA overall improves things for alcoholics. 42% are straight up abstinent from alcohol a year later, and those who do not achieve abstinence do not, repeat not have worse consequences from their drinking.

AA has a 67-75% success rate

Critics of the 12-step programs attempt to discredit the longitudinal studies, which show a 67-75% success rate for AA:

• They don’t like the fact that those high success rate numbers only come from people who actively go to AA meetings.  However, this is consistent with AA’s own claim that “Rarely have we seen a person fail who has thoroughly followed our path” (emphasis mine)
• They claim the numbers are invalid because of “self selection bias”; in other words, they believe the people who are clean and sober because of AA would have gotten clean and sober anyway in an alternate universe where AA doesn’t exist. A 2014 paper demonstrates this actually isn’t the case.
• They complain that the studies do not include dropouts.  However, the drop out rate is small enough to not significantly skew the numbers, especially since, most of the time, the dropouts undoubtedly 1) Didn’t work the AA program and 2) Didn’t get sober. 
• They make other complaints about the methodology of the studies. However, this ignores the fact that these numbers have been pretty consistent across multiple longitudinal studies. Fiorentine 1999: 74.8% success rate. Moos and Moos 2006: 67% success rate 16 years later. The Natural History of Alcoholism Revisited by George E. Vaillant has a table on page 197 showing that 19 out of 100 alcoholics attended 300 or more meetings over a 10-year period, of those 19, 14 were sober: 74% success rate. Witbrodt 2012 saw similar numbers: “high 12-step attendance (abstinence rates averaging 75%)”. AA’s own Big Book has the same abstinence rate using 1955 numbers: “Of alcoholics who came to A.A. and really tried, 50% got sober at once and remained that way; 25% sobered up after some relapses”

The fact is this: We have multiple studies showing the same number: 75% success rate among people who keep coming back to AA. The science is undeniable: Meeting makers make it.
 

Charlie Sheen, clean and sober in AA

Charlie Sheen, back in 2011, claimed that Alcoholics Anonymous has only a 5% success rate. Thankfully, over the years, he realized that those numbers are questionable.

He now has one year clean and sober in Alcoholics Anonymous. I'm very happy that he was able to let go of outdated ideas like the idea that science shows AA is ineffective (it doesn't), and instead use the program to become clean and sober.

What is AA's real success rate?

The question is this: What is AA's real success rate? We can never know the true number for two key reasons:

• AA does not keep track of members.
• "AA success" is a poorly defined term, which can have several actual meanings

While AA itself did not keep track of members, Moos and Moos 2006 did. It found the following: Of the people who were serious about AA, 67% of them were sober 16 years later.

But what about the dropouts? Not everyone who did the study did the 16-year followup, so we can't know the true success rate number, but looking at only the people who did follow up gives us a reasonable number (Dodes 2014 claims that Moos/Moos 2006 has completely invalid numbers because of these dropouts, but Dodes appears to have had no interest in an honest assessment of AA's success). Overall, 46% of the people in the Moos and Moos 2006 survey (again, only those who did the 16-year followup) were sober, regardless of their level of AA attendance.

Now we are getting in to the realm of philosophical questions: Do we count someone who was not serious about AA but still stayed sober as a success or failure for AA? Should we count people who weren't serious about meetings when calculating AA's success rate? Polemics like Dodes's The Sober Truth consider every single person who was not serious about AA a failure when making artificially low numbers for AA's success, even if the person got sober after all.

There are a lot of ways we can come up with a "success rate" number based on Moos and Moos 2006:

• There were 461 people in the survey who did a 16-year followup
• Of those, 25% (115) took the AA program seriously. Of those 115, 77 (67% of serious AA members, 17% overall) were sober, and 38 (33% of serious AA members, 8.2% overall) were not sober. We could use these numbers to argue that AA has "only" a 17% success rate, but that argument is dishonest because it discounts the 16% of people (72) who had some level of exposure to AA and still stayed sober.
• Overall, 214 (46%) got sober.  Since pretty much any alcoholic out there at least knows of AA's existence, we could make a case this is a 46% success rate.
• If we look at just people who had a least a little bit of AA treatment, this is 58% (269 of 461 total) of the people in the survey; of those 55% (149) were sober 16 years later. Contrast this with the 42% (192) who didn't try AA at all, 34% or 14% overall (65) were sober 16 years later.

Point being, there are a lot of ways to measure AA success, but to consider people who didn't take AA seriously but still got sober as "failures" (as Dodes and others have done) is downright dishonest.

Alcoholics Anonymous's own success claims
 
Alcoholics Anonymous has, since 1939, made the following claim about AA's success "Rarely have we seen a person fail who has thoroughly followed our path." This claim was expanded in the mid-1950s: "Of alcoholics who came to A.A. and really tried, 50% got sober at once and remained that way; 25% sobered up after some relapses, and among the remainder, those who stayed on with A.A. showed improvement."

Alcoholics Anonymous has never claimed it would help people who did not take the program seriously. Any claim about AA's success should only look at the people who take AA seriously; once we discount the people who did not take AA seriously, the numbers we get are in line with the numbers AA gave us in the mid-1950s (75% success rate among those whose who "really tried" AA).

It is true that AA has a high drop out rate, so I contend this: Have we found a treatment which is effective for alcoholics who, for whatever reason, refuse to work the AA program? Zemore 2019 makes a strong case that people in other programs can also stay clean and sober, as long as the goal is abstinence from alcohol. So, yes, I think people who will not work the AA program can still be clean and sober, but I also think it's dishonest to label all such people "failures" when measuring AA success.

Dodes and Moos and Moos 2006

Since Moos and Moos 2006 invalidates a central thesis of Dodes' false claim that Alcoholics Anonymous has a 5% success rate, Dodes goes to some effort to try to discredit this key study.

After discussing how the people who were successful in Moos and Moos 2006 were compliant (which is just another way to word the "self selection bias" problem), Dodes writes this:

The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.

Dodes is flat out inaccurate here. The multiple Moos studies which use the same subjects, including Moos and Moos 2006, had a total of 628 people who said they would participate in the studies. Moos and Moos 2006 looks at 461 of them. We have a drop out rate (in other words, people who elected to be in the study but didn't follow up) of under 27%, yet Dodes is stating Moos/Moos 2006 had a drop out rate of 40%. It would appear Dodes is confusing the number of people who stopped going to meetings with the number of people who didn't do two or more follow-ups; while these are both "drop outs", they are very different figures.

What Moos and Moos measured was this: Of the 115 people who were heavily involved with AA in the first year, and who did two or more follow ups, 77 of them (67%) were sober 16 years later.

What about those 167 subjects in Moos and Moos 2006 who didn't do two or more follow ups? We can go back to Timko and Moos 1993, the one-year follow-up study which used the same subjects. Here, they look at 515 of those 628 people. Even when looking at a larger subset of subjects, we get the same level of people who tried AA: Moos and Moos 2006 has 269 of the 461 subjects studied who tried AA in their first year (that's 58%).  Timko and Moos 2003 has 294 of the 515 subjects trying AA in their first year (57%). It would appear that the drop outs who did not finish two or more surveys did not significantly affect the results.

Dodes continues:

The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up and also stayed in AA for longer than six months -- that is, the group on which the authors’ major findings are based -- was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group "had better outcomes."

Dodes, again, is not correctly reading Moos and Moos 2006. First of all, there were 115 (not 107) people heavily involved with AA in the first year. Moos and Moos 2006, in addition, looked very closely at the number of people who were sober 16 years later: "Individuals who received 27 weeks or more of [Alcoholics Anonymous] treatment in the first year were more likely to be abstinent and less likely to have drinking problems at 16 years than were individuals who remained untreated in the first year." In addition, Moos and Moos 2006 tells us exactly how many people in that group were sober 16 years later in Table 3: 67% of 115. I will save Dr. Dodes the bother of bringing out a calculator: That's 77 people. I can not see how Dodes can look at Moos and Moos 2006 and claim it did not state how many people who took AA seriously were sober 16 years later.

In terms of other factors, table 2 of Moos and Moos 2006 shows a strong correlation between AA treatment and improvement in both self-efficacy and social functioning. To quote Moos/Moos 2006: "a longer duration of AA was independently related to a higher likelihood of 16-year abstinence, enhanced self-efficacy, and good social functioning, and less likelihood of 16-year drinking problems."

As far as I can tell, Dodes did not even fully read Moos and Moos 2006 before writing up a polemic trying to discredit its numbers. I expected better.

Let's get some real figures for AA success:  Moos and Moos 2006 showed that 67% of the people who took AA seriously were sober 16 years later. If the self section issue is problematic and you want experimental studies, both Litt et al. 2009 and Walitzer 2009 show that correctly getting subjects in the rooms of 12-step meetings work better than other addiction treatments.  Bottom line, AA works.