Dodes 2014: Bad science

Lance Dodes' book The Sober Truth is, bottom line, bad science. I have read it and compared what it says to the papers Dodes cites; in multiple cases, Dodes makes fundamental errors in his research. For example, Dodes claims that Fiorentine 1999 showed a 40% sobriety rate among active AA members; however, reading this study shows that it actually demonstrates a 75% sobriety rate. As another example, Dodes claims that Moos and Moos 2006 didn't look at how many active AA members stayed sober; actually Moos and Moos 2006 reports that 67% (or 77 people) were sober 16 years later.

I am not the only one who finds Dodes' reasoning flawed. The New York Times calls it a "polemical and deeply flawed book." The esteemed doctors Chad Emrick and Thomas Beresford, who find Dodes entire reasoning process questionable, looking at the second way Dodes tries to claim AA has a 5% success rate and say that "Albeit harsh to say, it appears that Dodes and Dodes did not let facts get in the way of drawing their second conclusion about AA’s effectiveness." The psychiatrists Jeffrey D. Roth and Edward J. Khantzian, in their review of The Sober Truth called Dodes' reasoning "pseudostatistical polemic." Addiction experts John Kelly and Gene Beresin also found Dodes' reasoning flawed.

For the record, the scientists criticizing Dodes' work come from the best schools: John Kelly, Eugene Beresin, and Edward J. Khantzian are all from Harvard. In addition, Jeffrey Roth graduated from Yale, and Thomas Beresford graduated from Stanford. I see a lot of very prestigious doctors strongly criticizing Dodes' work.

The outrage press (NPR, The Atlantic, Salon) placed far too much weight on the opinion of one doctor and his son even though so many other experts have looked at his reasoning and found it deeply flawed.

Dodes and Moos and Moos 2006

Since Moos and Moos 2006 invalidates a central thesis of Dodes' false claim that Alcoholics Anonymous has a 5% success rate, Dodes goes to some effort to try to discredit this key study.

After discussing how the people who were successful in Moos and Moos 2006 were compliant (which is just another way to word the "self selection bias" problem), Dodes writes this:

The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.

Dodes is flat out inaccurate here. The multiple Moos studies which use the same subjects, including Moos and Moos 2006, had a total of 628 people who said they would participate in the studies. Moos and Moos 2006 looks at 461 of them. We have a drop out rate (in other words, people who elected to be in the study but didn't follow up) of under 27%, yet Dodes is stating Moos/Moos 2006 had a drop out rate of 40%. It would appear Dodes is confusing the number of people who stopped going to meetings with the number of people who didn't do two or more follow-ups; while these are both "drop outs", they are very different figures.

What Moos and Moos measured was this: Of the 115 people who were heavily involved with AA in the first year, and who did two or more follow ups, 77 of them (67%) were sober 16 years later.

What about those 167 subjects in Moos and Moos 2006 who didn't do two or more follow ups? We can go back to Timko and Moos 1993, the one-year follow-up study which used the same subjects. Here, they look at 515 of those 628 people. Even when looking at a larger subset of subjects, we get the same level of people who tried AA: Moos and Moos 2006 has 269 of the 461 subjects studied who tried AA in their first year (that's 58%).  Timko and Moos 2003 has 294 of the 515 subjects trying AA in their first year (57%). It would appear that the drop outs who did not finish two or more surveys did not significantly affect the results.

Dodes continues:

The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up and also stayed in AA for longer than six months -- that is, the group on which the authors’ major findings are based -- was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group "had better outcomes."

Dodes, again, is not correctly reading Moos and Moos 2006. First of all, there were 115 (not 107) people heavily involved with AA in the first year. Moos and Moos 2006, in addition, looked very closely at the number of people who were sober 16 years later: "Individuals who received 27 weeks or more of [Alcoholics Anonymous] treatment in the first year were more likely to be abstinent and less likely to have drinking problems at 16 years than were individuals who remained untreated in the first year." In addition, Moos and Moos 2006 tells us exactly how many people in that group were sober 16 years later in Table 3: 67% of 115. I will save Dr. Dodes the bother of bringing out a calculator: That's 77 people. I can not see how Dodes can look at Moos and Moos 2006 and claim it did not state how many people who took AA seriously were sober 16 years later.

In terms of other factors, table 2 of Moos and Moos 2006 shows a strong correlation between AA treatment and improvement in both self-efficacy and social functioning. To quote Moos/Moos 2006: "a longer duration of AA was independently related to a higher likelihood of 16-year abstinence, enhanced self-efficacy, and good social functioning, and less likelihood of 16-year drinking problems."

As far as I can tell, Dodes did not even fully read Moos and Moos 2006 before writing up a polemic trying to discredit its numbers. I expected better.

Let's get some real figures for AA success:  Moos and Moos 2006 showed that 67% of the people who took AA seriously were sober 16 years later. If the self section issue is problematic and you want experimental studies, both Litt et al. 2009 and Walitzer 2009 show that correctly getting subjects in the rooms of 12-step meetings work better than other addiction treatments.  Bottom line, AA works.

Cochrane 2006: Out of date

The Cochrane 2006 review of Alcoholics Anonymous (Ferri 2006), which concluded that no experimental studies show the effectiveness of Alcoholics Anonymous compared to other treatments, is out of date. There have been multiple experimental studies showing twelve step facilitation (which is when a doctor on counselor helps an alcoholic go to AA meetings and become a part of AA's culture) effectiveness which are more recent than this old review.  I just added two of those studies to the Wikipedia:

Litt et al. 2009
Litt et al. 2009 randomly assigned to patients to one of three treatments: Network support, network support and contingency management, or a control condition (case management). The network support condition, which was "designed to use AA as an efficient means to engage patients in a supportive abstinence-oriented social network", had significantly better abstinence rates compared to the control and the combined treatments. This study is PMC2661035.

Walitzer 2009
Walitzer 2009 is an experimental study which randomly assigned patients to one of three treatments. Two of the treatments were Twelve Step Facilitation (TSF) treatments: Treatments which helped patients become involved with Alcoholics Anonymous and other twelve step programs.

One of the two TSF treatments, the 12-Step-based directive approach, resulted in increased number of days abstinent and other positive outcomes compared to the other two treatments. This study is PMC2802221.

Both of these studies are free to read; I encourage anyone who believes that the old Cochrane study still shows that Alcoholics Anonymous does not work to read these post-Cochrane studies.

Fiorentine 1999: 75% success rate

I just added this to Effectiveness of Alcoholics Anonymous:

Fiorentine 1999 was a 24-month longitudinal study measuring the effectiveness of AA and Narcotics Anonymous. Like other longitudinal studies, it shows a strong correlation between 12-step attendance and being both clean from drugs and sober from alcohol: People who went to 12-step meetings in the study had about a 75% success rate.

In more detail, at the 24 month follow up, 77.7% of people who went to one or more meetings a week self-reported being clean; urinalysis was very close to that figure, showing some 76.4% of the regular attenders begin clean. For participants who did not regularly go to meetings, the self reported figure for being clean was 56% and the drug test showed 57.9% being clean. The self-reported figures for alcohol abuse were similar: 74.8% of regular 12-step attenders self-reported being completely sober, but only 40% of people who did not go to 12-step meetings claimed to be sober. Urinalysis showed 96.6% of people regularly going to meetings as sober, in contrast to the 88.9% of people who didn't go to meetings once a week or more whose urine sample was alcohol-free.

The report then sees if 12-steps meetings have a specific effect (in other words, if there is causation or merely correlation) by using statistical analysis to compare participants self-reported level of motivation, 12-step attendance, and successfully getting clean and sober. The result of this observation was that "Weekly or more frequent 12-step participants are 1.59 times more likely than less-than-weekly participants to maintain abstinence after controlling for the differences in recovery motivation measured by the scale."

---

Not added to that article: Lance Dodes falsely claims that Fiorentine 1999 states that "approximately 40 percent of individuals categorized as having continued active participation in AA maintained high rates of abstinence." and then multiplies an unrelated figure to falsely claim that AA has an 8% success rate. Fiorentine 1999 makes no such claim. No where in this paper does the above quote appear. I was unable to find this quote. Thomas Beresford was unable to find this quote. As far as either of us can tell, this quote is a figment of Dodes' imagination.