Sunday, May 27, 2018

Dodes 2014: Bad science

Lance Dodes' book The Sober Truth is, bottom line, bad science. I have read it and compared what it says to the papers Dodes cites; in multiple cases, Dodes makes fundamental errors in his research. For example, Dodes claims that Fiorentine 1999 showed a 40% sobriety rate among active AA members; however, reading this study shows that it actually demonstrates a 75% sobriety rate. As another example, Dodes claims that Moos and Moos 2006 didn't look at how many active AA members stayed sober; actually Moos and Moos 2006 reports that 67% (or 77 people) were sober 16 years later.

I am not the only one who finds Dodes' reasoning flawed. The New York Times calls it a "polemical and deeply flawed book." The esteemed doctors Chad Emrick and Thomas Beresford, who find Dodes entire reasoning process questionable, looking at the second way Dodes tries to claim AA has a 5% success rate and say that "Albeit harsh to say, it appears that Dodes and Dodes did not let facts get in the way of drawing their second conclusion about AA’s effectiveness." The psychiatrists Jeffrey D. Roth and Edward J. Khantzian, in their review of The Sober Truth called Dodes' reasoning "pseudostatistical polemic." Addiction experts John Kelly and Gene Beresin also found Dodes' reasoning flawed.

For the record, the scientists criticizing Dodes' work come from the best schools: John Kelly, Eugene Beresin, and Edward J. Khantzian are all from Harvard. In addition, Jeffrey Roth graduated from Yale, and Thomas Beresford graduated from Stanford. I see a lot of very prestigious doctors strongly criticizing Dodes' work.

The outrage press (NPR, The Atlantic, Salon) placed far too much weight on the opinion of one doctor and his son even though so many other experts have looked at his reasoning and found it deeply flawed.

Saturday, May 26, 2018

Dodes and Moos and Moos 2006

Since Moos and Moos 2006 invalidates a central thesis of Dodes' false claim that Alcoholics Anonymous has a 5% success rate, Dodes goes to some effort to try to discredit this key study.

After discussing how the people who were successful in Moos and Moos 2006 were compliant (which is just another way to word the "self selection bias" problem), Dodes writes this:
The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.
Dodes is flat out inaccurate here. The multiple Moos studies which use the same subjects, including Moos and Moos 2006, had a total of 628 people who said they would participate in the studies. Moos and Moos 2006 looks at 461 of them. We have a drop out rate (in other words, people who elected to be in the study but didn't follow up) of under 27%, yet Dodes is stating Moos/Moos 2006 had a drop out rate of 40%. It would appear Dodes is confusing the number of people who stopped going to meetings with the number of people who didn't do two or more follow-ups; while these are both "drop outs", they are very different figures.

What Moos and Moos measured was this: Of the 115 people who were heavily involved with AA in the first year, and who did two or more follow ups, 77 of them (67%) were sober 16 years later.

What about those 167 subjects in Moos and Moos 2006 who didn't do two or more follow ups? We can go back to Timko and Moos 2003, the one-year follow-up study which used the same subjects. Here, they look at 515 of those 628 people. Even when looking at a larger subset of subjects, we get the same level of people who tried AA: Moos and Moos 2006 has 269 of the 461 subjects studied who tried AA in their first year (that's 58%).  Timko and Moos 2003 has 294 of the 515 subjects trying AA in their first year (57%). It would appear that the drop outs who did not finish two or more surveys did not significantly affect the results.

Dodes continues:
The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up and also stayed in AA for longer than six months -- that is, the group on which the authors’ major findings are based -- was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group "had better outcomes."
Dodes, again, is not correctly reading Moos and Moos 2006. First of all, there were 115 (not 107) people heavily involved with AA in the first year. Moos and Moos 2006, in addition, looked very closely at the number of people who were sober 16 years later: "Individuals who received 27 weeks or more of [Alcoholics Anonymous] treatment in the first year were more likely to be abstinent and less likely to have drinking problems at 16 years than were individuals who remained untreated in the first year." In addition, Moos and Moos 2006 tells us exactly how many people in that group were sober 16 years later in Table 3: 67% of 115. I will save Dr. Dodes the bother of bringing out a calculator: That's 77 people. I can not see how Dodes can look at Moos and Moos 2006 and claim it did not state how many people who took AA seriously were sober 16 years later.

In terms of other factors, table 2 of Moos and Moos 2006 shows a strong correlation between AA treatment and improvement in both self-efficacy and social functioning. To quote Moos/Moos 2006: "a longer duration of AA was independently related to a higher likelihood of 16-year abstinence, enhanced self-efficacy, and good social functioning, and less likelihood of 16-year drinking problems."

As far as I can tell, Dodes did not even fully read Moos and Moos 2006 before writing up a polemic trying to discredit its numbers. I expected better.

Let's get some real figures for AA success:  Moos and Moos 2006 showed that 67% of the people who took AA seriously were sober 16 years later. If the self section issue is problematic and you want experimental studies, both Litt et al. 2009 and Walitzer 2009 show that correctly getting subjects in the rooms of 12-step meetings work better than other addiction treatments.  Bottom line, AA works.

Wednesday, May 23, 2018

Cochrane 2006: Out of date

The Cochrane 2006 review of Alcoholics Anonymous (Ferri 2006), which concluded that no experimental studies show the effectiveness of Alcoholics Anonymous compared to other treatments, is out of date. There have been multiple experimental studies showing twelve step facilitation (which is when a doctor on counselor helps an alcoholic go to AA meetings and become a part of AA's culture) effectiveness which are more recent than this old review.  I just added two of those studies to the Wikipedia:

Litt et al. 2009
Litt et al. 2009 randomly assigned to patients to one of three treatments: Network support, network support and contingency management, or a control condition (case management). The network support condition, which was "designed to use AA as an efficient means to engage patients in a supportive abstinence-oriented social network", had significantly better abstinence rates compared to the control and the combined treatments. This study is PMC2661035.

Walitzer 2009
Walitzer 2009 is an experimental study which randomly assigned patients to one of three treatments. Two of the treatments were Twelve Step Facilitation (TSF) treatments: Treatments which helped patients become involved with Alcoholics Anonymous and other twelve step programs.

One of the two TSF treatments, the 12-Step-based directive approach, resulted in increased number of days abstinent and other positive outcomes compared to the other two treatments. This study is PMC2802221.

Both of these studies are free to read; I encourage anyone who believes that the old Cochrane study still shows that Alcoholics Anonymous does not work to read these post-Cochrane studies.

Tuesday, May 22, 2018

Fiorentine 1999: 75% success rate

I just added this to Effectiveness of Alcoholics Anonymous:

Fiorentine 1999 was a 24-month longitudinal study measuring the effectiveness of AA and Narcotics Anonymous. Like other longitudinal studies, it shows a strong correlation between 12-step attendance and being both clean from drugs and sober from alcohol: People who went to 12-step meetings in the study had about a 75% success rate.

In more detail, at the 24 month follow up, 77.7% of people who went to one or more meetings a week self-reported being clean; urinalysis was very close to that figure, showing some 76.4% of the regular attenders begin clean. For participants who did not regularly go to meetings, the self reported figure for being clean was 56% and the drug test showed 57.9% being clean. The self-reported figures for alcohol abuse were similar: 74.8% of regular 12-step attenders self-reported being completely sober, but only 40% of people who did not go to 12-step meetings claimed to be sober. Urinalysis showed 96.6% of people regularly going to meetings as sober, in contrast to the 88.9% of people who didn't go to meetings once a week or more whose urine sample was alcohol-free.

The report then sees if 12-steps meetings have a specific effect (in other words, if there is causation or merely correlation) by using statistical analysis to compare participants self-reported level of motivation, 12-step attendance, and successfully getting clean and sober. The result of this observation was that "Weekly or more frequent 12-step participants are 1.59 times more likely than less-than-weekly participants to maintain abstinence after controlling for the differences in recovery motivation measured by the scale."


Not added to that article: Lance Dodes falsely claims that Fiorentine 1999 states that "approximately 40 percent of individuals categorized as having continued active participation in AA maintained high rates of abstinence." and then multiplies an unrelated figure to falsely claim that AA has an 8% success rate. Fiorentine 1999 makes no such claim. No where in this paper does the above quote appear. I was unable to find this quote. Thomas Beresford was unable to find this quote. As far as either of us can tell, this quote is a figment of Dodes' imagination.

Wednesday, April 18, 2018

AA has no opinion on whether God exists

From another of my recent Disqus postings:

The existence of God is a theological question, not a question that can be answered by science. However, we can measure how much AA helps alcoholics stay sober.

We know that the more often people go to meetings (especially in the first year), the more likely they are to stay sober. Moos and Moos 2006 shows an undeniable correlation. Humpreys 2014 makes a strong case that the AA program is the cause for meeting makers making it. Galanter 2016 goes even further: It shows objective scientific evidence that praying helps successful AA members stay sober.

Now, we don't have peer reviewed science asking people “did you work the steps as written in the first 164 pages to the best of your ability?”, but we do have Gabriel Segal's essay "How an addict's power of choice is lost and can be regained" which makes a very strong case that 99% of alcoholics who get serious about the program stay sober.

Monday, April 16, 2018

Moderate drinking redux; also more on the Sinclair Method

Let me just post some of my recent disqus postings:

The fact is that the Sobell 1973 study [a study which claimed alcoholics could moderately drink again] was so bad, a fraud investigation was opened. As a 1984 New York Times article put it (look for “Panel finds no fraud by alcohol researchers”):

>>The panel also said it had found ''several errors'' and ''use of ambiguous terminology'' in scientific publications by the Sobells, which indicated that they were ''careless in preparing their manuscripts for publication.'' But the panel said it had seen no evidence that these errors and ambiguities ''represent scientific misconduct - defined as 'fraudulent or highly irregular research practices.' ''<<

Anyway, facts are facts. Those “moderate” drinkers from the Sobell 1973 study were either drinking alcoholically again, or were dead from drinking, or were abstaining from alcohol when a 10-year-followup was performed.


Sobell 1973 and the moderate drinkers: Four of those “moderate drinkers” died. Also: Six were abstinent, nine of the “moderate” drinkers were engaging in out of control drinking. There was only one person who, ten years later, could be considered a “moderate” drinker. Please read Pendery 1982 or any of the numerous books which correctly cite it.

Sobell 1973 in comparison to the non-moderate drinkers: This is cherry picking data. The Sobells only compared the fatalities of the “moderate” drinkers with the non-moderate drinkers. I have not seen the abstinent or out-of-control drinking figures for this other group; and it is telling that the Sobells never shared (or, it would seem, even bother to get) those figures.

Since there were only 20 subjects in each study, whether six or four died is an error of approximation, nothing more.

Davies 1962: These “moderate” drinkers also reverted to serious drinking again. Reference: G. Edwards. "D.L. Davies and 'Normal drinking in recovered alcohol addicts': the genesis of a paper" Drug and Alcohol Dependence 35.3 (1994)

Rand report: The 1980 four-year followup shows many of the 1976 Rand report moderate drinkers relapsing.

Moderation management: No peer-reviewed science was ever done with this group, but it’s very telling that its founder drank heavily, to the point police took her to detox in handcuffs while she was still the leader of Moderation Management. She soon afterwards killed two people while driving drunk and eventually killed herself [1].

Point being, pretty much any study showing successful “moderate” drinking, when followed up, shows those same “moderate” drinkers drinking heavily again.

[1] There are claims that the Moderation Management founder was a member of AA at the time of her crash, but as per her autobiography, she did not work the 12-step program in any meaningful way. Instead, she was coming up with excuses for not going to meetings.


In response to a claim that the “moderate” drinkers in Sobell 1973 had better success rates than people trying abstinence:

In terms of more recent studies, people who choose to work the AA program have between a 75% (Vaillant 1995) and 67% (Moos and Moos 2006) success rate. The Sobell moderation group had, by comparison, a 30% abstinence success rate and a 5% moderation success rate.


In response to a claim that the Sinclair method works better when trying complete abstinence, and works better the longer it is applied:

OK, let me go back to PMC3970823. They use a ”magic number” called “g” which is summarized near the end: “the aggregated Hedges’ g for naltrexone and acamprosate compared to placebo was 0.209 (CI: 0.157 – 0.262) – indicating a small but significant effect. In comparison to other medications prescribed for mental health (e.g., for depression), the effects of naltrexone and acamprosate are somewhat smaller”

So, this information in mind, let’s look at table 2 in PMC3970823. Naltrexone has an abstinence aggregate of .116—which is not very different than a placebo. However, the numbers for “heavy drinking” (.189), craving (.144), and both heavy drinking and craving (.180) are not much better—they are “small but significant.”

It also looks like, when looking at multiple studies, the effect diminishes when doing follow-ups: “naltrexone studies tended to have effect sizes for heavy drinking outcomes at the last follow-up point (g = .135, k = 6) that were slightly smaller compared to end-of-treatment (g = .189, k = 39). Although only two studies provided data, the naltrexone-placebo effect sizes for craving at the last follow-up point were close to zero (g = .053).” In other words, this meta-study didn’t see naltrexone helping people not drink heavily over longer periods.

In light of Pendery 1982, and in light of what happened with the founder of Moderation Management (while she claimed to drink moderately without problem for years, she killed two people while driving drunk, and eventually killed herself), when people suggest moderation as an outcome for alcoholics, I will remain very very skeptical until when and if we have 10-year-followups with very high “g” values and very low “p” values.

Saturday, April 14, 2018

The Sinclair method revisited

An advocate for the Sinclair method pointed out, in an online debate, the study “Targeted Naltrexone for Problem Drinkers.”  Here is my response:

That 2011 study (“Targeted Naltrexone for Problem Drinkers”) is a 12-week trial, and the study considers “moderate drinking” a successful outcome. This is a very short trial period, and short-term studies since the 1960s showing alcoholics “successfully” drinking moderately have, if followed up, shown the same drinkers having serious drinking problems five or 10 years later (the most famous example of this is Sobell 1973, showing “moderate” drinking among alcoholics, but Pendery 1982 showed that the same “moderate” drinkers were either dead, abstinent, or drinking heavily a decade later).

The meta-study (a study looking at other studies) “Meta-analysis of naltrexone and acamprosate for treating alcohol use disorders: When are these medications most helpful?” (PMC3970823) points out that “Several reviews and meta-analyses have reported small or mixed effects for each medication (both naltrexone and another medication called acamprosate), and treatment providers cite concerns over efficacy as one barrier to greater medication use.”

This isn’t the first study which showed good results for Naltrexone, but considering the number of other studies which show only a modest effect, and considering the need for a long term follow up to see if an alcoholic is in fact successfully moderately drinking, I will need to see studies with five-year or 10-year follow ups showing continued “moderate” drinking before I will believe this drug is a functional way to let alcoholics control their drinking.