Friday, July 20, 2018

Kelly 2017: The kids assigned to AA did better, but we can't prove it

Let's look at the paper "Incorporating 12-step program elements improves youth substance-use disorder treatment." Here, the conclusion in the abstract is this " in terms of abstinence, a novel integrated 12‐Step facilitation treatment for adolescent substance use disorder (iTSF) showed no greater benefits." However, the conclusion is inaccurate.

The kids in the study who were assigned to TSF (Twelve Step Facilitation; the consoler tells the kid to go to more 12-step meetings) treatment did better than the kids who just had Cognitive Behavior Therapy (CBT), the control condition for this experiment.

In particular, nine months after the study ended (in the final follow-up), the group who randomly underwent TSF had a higher percentage of days abstinent (58% vs. 49% at the nine-month follow-up, overall p=0.33) and a larger number of subjects either completely or "mostly" abstinent (33% vs. 21% at the nine-month followup, overall p=0.30).  

The reason why these improvements were not considered significant is because there is a pretty high p value; in plain English, a p value of 0.33 means that there is a one-in-three chance the kids who were assigned to undergo TSF treatment (i.e. they went to more AA meetings) only did better because of random chance (the kids assigned to TSF might have been, by chance, more likely to do better).

The main problem with this experiment is that there were only 59 subjects; this is not enough people for them to conclude that, while the kids who used AA treatment did better, and it is more likely than not this was because of the TSF (i.e. "get your butt to an AA meeting") treatment they received, we can not be sure this was because the randomly picked kids who got TSF treatment were ones who would do better anyway.

Tuesday, July 17, 2018

12-step class vs. 12-step fellowship

There are a few randomized controlled trials out there, such as Brandsma 1980 [1] and Stahlbrandt 2007 [2] where the "Alcoholics Anonymous" experimental condition was questionable. With both of these experiments, there was no difference between subjects who had the 12-step-like condition and the control group.

What happened?

The 12-step condition was not an actual AA meeting.

With Brandsma 1980, as Kaskutas 2009 [3] describes it, the subjects undergoing "AA" treatment weren't actually going to AA meetings:
The description of the AA condition states that the steps were used for discussion content, the group focused on newcomers, and they told patients about sponsors, but it is not clear whether the meetings were led by AA members, whether crosstalk was allowed, whether the meeting leader shared their story as part of the meeting, or whether the meeting format was what one would encounter at an actual AA meeting. The meetings may not have been open to other AA members in the community, and not been listed in the AA meeting directory, which would mean that a potentially important therapeutic ingredient of AA--the experience of longer-term members--would not have been present in the AA condition.
With Stahlbrandt 2007, here is how the supposed "AA" meetings were run:
The TSI intervention was a 3-hour formal lecture, given by
therapists trained in the 12-step method.
Like Brandsma 1980, they did not see a difference between the supposed AA condition and the control group.

Point being, if an experimental condition is supposed to be an AA meeting, the numbers will not be reliable unless the AA condition is, in fact, an actual meeting. Which it wasn't with these two studies.

More recent studies, such as Litt et al. 2009, where we give the experimental group treatment which gets them going to more real AA meetings, show that the more people go to meetings, the more days they will be sober. It is unfortunate that these poorly done surveys have incorrectly given the wrong impression about 12 step efficacy.

[1] Brandsma, Jeffery M; Maultsby, Maxie C; Welsh, Richard J (1980). Outpatient Treatment of Alcoholism: a review and comparative study. Baltimore, MD: University Park Press. ISBN 0-8391-1393-5

[2] Stahlbrandt, Henriettæ; Johnsson, Kent O.; Berglund, Mats (2007). "Cluster Randomized Trial". Alcoholism: Clinical and Experimental Research. 31 (3): 458–66. doi:10.1111/j.1530-0277.2006.00327.x. PMID 17295731

[3] Kaskutas, Lee Ann (2009). "Alcoholics Anonymous Effectiveness: Faith Meets Science". Journal of Addictive Diseases. 28 (2): 145–157. doi:10.1080/10550880902772464. PMC 2746426. PMID 19340677.

Wednesday, July 4, 2018

It helps to actually understand the studies

The Fix has no, for all intents and purposes, editorial control over what gets posted to their magazine. It's a pleasant surprise to see someone post there use peer reviewed research; it's an unpleasant surprise to see how poorly they actually quote the research.

Case in point: This 2017 article from the members of the group "Handshake Media." Two points for actually quoting peer reviewed research.  -10000 points for ignoring key points made by recent addiction research. 

Let’s see here: Discounting studies showing showing TSF/12-step effectiveness without a control group (even though the surgeon general didn't), but accepting the results of Zenmore 2017/2018 which also didn't have a control group. Ignoring experimental studies which do show TSF effectiveness (e.g. Walitzer 2009, which she should have been aware of because it was mentioned in Humphreys 2014).

Discounting Humphreys without presenting a compelling argument: This silly article claims that "Their methods were inadequate for determining whether increased participation is causally linked to increased abstinence, or whether increases in abstinence occurred without self-selection bias", but can not show any actual problem with this paper. Humphreys 2014 shows that, when looking at the increase in AA caused by a randomized experimental condition, that, among other things "The 3 month model showed that an increase in attendance of 1% (i.e., one absolute percentage point, not 1% of prior attendance) predicted an increase in PDA [Percent days abstinent from alcohol] of about a third of a percentage point (B = 0.38, p = .001)." It would seem that they did not fully read or did not understand Humphreys 2014.

Also: There's no need to link to the paywall version of Humphreys 2014; It's a PMC open access article.

The author’s bias is obvious.

That said, happy 4th of July everyone. As per request, this page is now open to moderated comments and please be patient; it may take days or weeks to moderate your comment.

Edit: To be fair, this author later conceded that Humphreys 2014 shows evidence that 12-step approaches can be helpful.

Wednesday, June 27, 2018

People in AA or NA should not play doctor

People in AA and NA should not play doctor. Full stop. End of discussion. The Big Book is clear on this. From page 133:
we should never belittle a good doctor or psychiatrist. Their services are often indispensable in treating a newcomer and in following his case afterward.
Point being, it's not the place of people in the rooms of AA (or NA, or any other 12-step program) to play doctor. There's a reason why it takes years of study to become a doctor: Medical science is hard. An average alcoholic or addict seeking recovery is not qualified to give medical advice.

This brings us to this article. The problem: People in NA think they are in a position to tell recovering addicts to not engage in Opioid replacement therapy. No, they are not. Full stop. Never let someone in the 12 step rooms play doctor! If a recovering addict is recommended by a doctor to take, say, Suboxone, it's not the duty of someone in NA to question that doctor. 

Sunday, May 27, 2018

Dodes 2014: Bad science

Lance Dodes' book The Sober Truth is, bottom line, bad science. I have read it and compared what it says to the papers Dodes cites; in multiple cases, Dodes makes fundamental errors in his research. For example, Dodes claims that Fiorentine 1999 showed a 40% sobriety rate among active AA members; however, reading this study shows that it actually demonstrates a 75% sobriety rate. As another example, Dodes claims that Moos and Moos 2006 didn't look at how many active AA members stayed sober; actually Moos and Moos 2006 reports that 67% (or 77 people) were sober 16 years later.

I am not the only one who finds Dodes' reasoning flawed. The New York Times calls it a "polemical and deeply flawed book." The esteemed doctors Chad Emrick and Thomas Beresford, who find Dodes entire reasoning process questionable, looking at the second way Dodes tries to claim AA has a 5% success rate and say that "Albeit harsh to say, it appears that Dodes and Dodes did not let facts get in the way of drawing their second conclusion about AA’s effectiveness." The psychiatrists Jeffrey D. Roth and Edward J. Khantzian, in their review of The Sober Truth called Dodes' reasoning "pseudostatistical polemic." Addiction experts John Kelly and Gene Beresin also found Dodes' reasoning flawed.

For the record, the scientists criticizing Dodes' work come from the best schools: John Kelly, Eugene Beresin, and Edward J. Khantzian are all from Harvard. In addition, Jeffrey Roth graduated from Yale, and Thomas Beresford graduated from Stanford. I see a lot of very prestigious doctors strongly criticizing Dodes' work.

The outrage press (NPR, The Atlantic, Salon) placed far too much weight on the opinion of one doctor and his son even though so many other experts have looked at his reasoning and found it deeply flawed.

Saturday, May 26, 2018

Dodes and Moos and Moos 2006

Since Moos and Moos 2006 invalidates a central thesis of Dodes' false claim that Alcoholics Anonymous has a 5% success rate, Dodes goes to some effort to try to discredit this key study.

After discussing how the people who were successful in Moos and Moos 2006 were compliant (which is just another way to word the "self selection bias" problem), Dodes writes this:
The Moos study also employs some objectionable statistical methods. In one critical omission, its conclusions ignore all the people who died and the large number of people who dropped out of the study altogether, despite conceding that these were the people who statistically consumed the most alcohol. As early as year eight, the number of subjects who were left in AA had already shrunk by nearly 40 percent (from 269 to 166), yet these people are erased from the conclusions as if they had never existed at all. Add up all the people who died and the dropouts, and the results for AA become far grimmer than the authors suggest.
Dodes is flat out inaccurate here. The multiple Moos studies which use the same subjects, including Moos and Moos 2006, had a total of 628 people who said they would participate in the studies. Moos and Moos 2006 looks at 461 of them. We have a drop out rate (in other words, people who elected to be in the study but didn't follow up) of under 27%, yet Dodes is stating Moos/Moos 2006 had a drop out rate of 40%. It would appear Dodes is confusing the number of people who stopped going to meetings with the number of people who didn't do two or more follow-ups; while these are both "drop outs", they are very different figures.

What Moos and Moos measured was this: Of the 115 people who were heavily involved with AA in the first year, and who did two or more follow ups, 77 of them (67%) were sober 16 years later.

What about those 167 subjects in Moos and Moos 2006 who didn't do two or more follow ups? We can go back to Timko and Moos 1993, the one-year follow-up study which used the same subjects. Here, they look at 515 of those 628 people. Even when looking at a larger subset of subjects, we get the same level of people who tried AA: Moos and Moos 2006 has 269 of the 461 subjects studied who tried AA in their first year (that's 58%).  Timko and Moos 2003 has 294 of the 515 subjects trying AA in their first year (57%). It would appear that the drop outs who did not finish two or more surveys did not significantly affect the results.

Dodes continues:
The stated size of this survey is also misleading. Although the researchers began with 628 people, the total number of people who remained through the sixteen-year follow-up and also stayed in AA for longer than six months -- that is, the group on which the authors’ major findings are based -- was just 107, or 17 percent of the original sample. And of the remaining 107, the researchers never revealed the actual number of people who improved, or even stayed sober. They told us only which group "had better outcomes."
Dodes, again, is not correctly reading Moos and Moos 2006. First of all, there were 115 (not 107) people heavily involved with AA in the first year. Moos and Moos 2006, in addition, looked very closely at the number of people who were sober 16 years later: "Individuals who received 27 weeks or more of [Alcoholics Anonymous] treatment in the first year were more likely to be abstinent and less likely to have drinking problems at 16 years than were individuals who remained untreated in the first year." In addition, Moos and Moos 2006 tells us exactly how many people in that group were sober 16 years later in Table 3: 67% of 115. I will save Dr. Dodes the bother of bringing out a calculator: That's 77 people. I can not see how Dodes can look at Moos and Moos 2006 and claim it did not state how many people who took AA seriously were sober 16 years later.

In terms of other factors, table 2 of Moos and Moos 2006 shows a strong correlation between AA treatment and improvement in both self-efficacy and social functioning. To quote Moos/Moos 2006: "a longer duration of AA was independently related to a higher likelihood of 16-year abstinence, enhanced self-efficacy, and good social functioning, and less likelihood of 16-year drinking problems."

As far as I can tell, Dodes did not even fully read Moos and Moos 2006 before writing up a polemic trying to discredit its numbers. I expected better.

Let's get some real figures for AA success:  Moos and Moos 2006 showed that 67% of the people who took AA seriously were sober 16 years later. If the self section issue is problematic and you want experimental studies, both Litt et al. 2009 and Walitzer 2009 show that correctly getting subjects in the rooms of 12-step meetings work better than other addiction treatments.  Bottom line, AA works.

Wednesday, May 23, 2018

Cochrane 2006: Out of date

The Cochrane 2006 review of Alcoholics Anonymous (Ferri 2006), which concluded that no experimental studies show the effectiveness of Alcoholics Anonymous compared to other treatments, is out of date. There have been multiple experimental studies showing twelve step facilitation (which is when a doctor on counselor helps an alcoholic go to AA meetings and become a part of AA's culture) effectiveness which are more recent than this old review.  I just added two of those studies to the Wikipedia:

Litt et al. 2009
Litt et al. 2009 randomly assigned to patients to one of three treatments: Network support, network support and contingency management, or a control condition (case management). The network support condition, which was "designed to use AA as an efficient means to engage patients in a supportive abstinence-oriented social network", had significantly better abstinence rates compared to the control and the combined treatments. This study is PMC2661035.

Walitzer 2009
Walitzer 2009 is an experimental study which randomly assigned patients to one of three treatments. Two of the treatments were Twelve Step Facilitation (TSF) treatments: Treatments which helped patients become involved with Alcoholics Anonymous and other twelve step programs.

One of the two TSF treatments, the 12-Step-based directive approach, resulted in increased number of days abstinent and other positive outcomes compared to the other two treatments. This study is PMC2802221.

Both of these studies are free to read; I encourage anyone who believes that the old Cochrane study still shows that Alcoholics Anonymous does not work to read these post-Cochrane studies.