Kelly 2017: The kids assigned to AA did better, but we can't prove it

Let's look at the paper "Incorporating 12-step program elements improves youth substance-use disorder treatment." Here, the conclusion in the abstract is this " in terms of abstinence, a novel integrated 12‐Step facilitation treatment for adolescent substance use disorder (iTSF) showed no greater benefits." However, the conclusion is inaccurate.

The kids in the study who were assigned to TSF (Twelve Step Facilitation; the consoler tells the kid to go to more 12-step meetings) treatment did better than the kids who just had Cognitive Behavior Therapy (CBT), the control condition for this experiment.

In particular, nine months after the study ended (in the final follow-up), the group who randomly underwent TSF had a higher percentage of days abstinent (58% vs. 49% at the nine-month follow-up, overall p=0.33) and a larger number of subjects either completely or "mostly" abstinent (33% vs. 21% at the nine-month followup, overall p=0.30).  

The reason why these improvements were not considered significant is because there is a pretty high p value; in plain English, a p value of 0.33 means that there is a one-in-three chance the kids who were assigned to undergo TSF treatment (i.e. they went to more AA meetings) only did better because of random chance (the kids assigned to TSF might have been, by chance, more likely to do better).

The main problem with this experiment is that there were only 59 subjects; this is not enough people for them to conclude that, while the kids who used AA treatment did better, and it is more likely than not this was because of the TSF (i.e. "get your butt to an AA meeting") treatment they received, we can not be sure this was because the randomly picked kids who got TSF treatment were ones who would do better anyway.

12-step class vs. 12-step fellowship

There are a few randomized controlled trials out there, such as Brandsma 1980 [1] and Stahlbrandt 2007 [2] where the "Alcoholics Anonymous" experimental condition was questionable. With both of these experiments, there was no difference between subjects who had the 12-step-like condition and the control group.

What happened?

The 12-step condition was not an actual AA meeting.

With Brandsma 1980, as Kaskutas 2009 [3] describes it, the subjects undergoing "AA" treatment weren't actually going to AA meetings:

The description of the AA condition states that the steps were used for discussion content, the group focused on newcomers, and they told patients about sponsors, but it is not clear whether the meetings were led by AA members, whether crosstalk was allowed, whether the meeting leader shared their story as part of the meeting, or whether the meeting format was what one would encounter at an actual AA meeting. The meetings may not have been open to other AA members in the community, and not been listed in the AA meeting directory, which would mean that a potentially important therapeutic ingredient of AA--the experience of longer-term members--would not have been present in the AA condition.

With Stahlbrandt 2007, here is how the supposed "AA" meetings were run:

The TSI intervention was a 3-hour formal lecture, given by
therapists trained in the 12-step method.

Like Brandsma 1980, they did not see a difference between the supposed AA condition and the control group.

Point being, if an experimental condition is supposed to be an AA meeting, the numbers will not be reliable unless the AA condition is, in fact, an actual meeting. Which it wasn't with these two studies.

More recent studies, such as Litt et al. 2009, where we give the experimental group treatment which gets them going to more real AA meetings, show that the more people go to meetings, the more days they will be sober. It is unfortunate that these poorly done surveys have incorrectly given the wrong impression about 12 step efficacy.

[1] Brandsma, Jeffery M; Maultsby, Maxie C; Welsh, Richard J (1980). Outpatient Treatment of Alcoholism: a review and comparative study. Baltimore, MD: University Park Press. ISBN 0-8391-1393-5

[2] Stahlbrandt, Henriettæ; Johnsson, Kent O.; Berglund, Mats (2007). "Cluster Randomized Trial". Alcoholism: Clinical and Experimental Research. 31 (3): 458–66. doi:10.1111/j.1530-0277.2006.00327.x. PMID 17295731

[3] Kaskutas, Lee Ann (2009). "Alcoholics Anonymous Effectiveness: Faith Meets Science". Journal of Addictive Diseases. 28 (2): 145–157. doi:10.1080/10550880902772464. PMC 2746426. PMID 19340677.

It helps to actually understand the studies

The Fix has no, for all intents and purposes, editorial control over what gets posted to their magazine. It's a pleasant surprise to see someone post there use peer reviewed research; it's an unpleasant surprise to see how poorly they actually quote the research.

Case in point: This 2017 article from the members of the group "Handshake Media." Two points for actually quoting peer reviewed research.  -10000 points for ignoring key points made by recent addiction research. 

Let’s see here: Discounting studies showing showing TSF/12-step effectiveness without a control group (even though the surgeon general didn't), but accepting the results of Zenmore 2017/2018 which also didn't have a control group. Ignoring experimental studies which do show TSF effectiveness (e.g. Walitzer 2009, which she should have been aware of because it was mentioned in Humphreys 2014).

Discounting Humphreys without presenting a compelling argument: This silly article claims that "Their methods were inadequate for determining whether increased participation is causally linked to increased abstinence, or whether increases in abstinence occurred without self-selection bias", but can not show any actual problem with this paper. Humphreys 2014 shows that, when looking at the increase in AA caused by a randomized experimental condition, that, among other things "The 3 month model showed that an increase in attendance of 1% (i.e., one absolute percentage point, not 1% of prior attendance) predicted an increase in PDA [Percent days abstinent from alcohol] of about a third of a percentage point (B = 0.38, p = .001)." It would seem that they did not fully read or did not understand Humphreys 2014.

Also: There's no need to link to the paywall version of Humphreys 2014; It's a PMC open access article.

The author’s bias is obvious.

That said, happy 4th of July everyone. As per request, this page is now open to moderated comments and please be patient; it may take days or weeks to moderate your comment.

Edit: To be fair, this author later conceded that Humphreys 2014 shows evidence that 12-step approaches can be helpful.