Let's look at the paper "Incorporating 12-step program elements improves youth substance-use disorder treatment." Here, the conclusion in the abstract is this " in terms of abstinence, a novel integrated 12‐Step facilitation treatment for adolescent substance use disorder (iTSF) showed no greater benefits." However, the conclusion is inaccurate.
The kids in the study who were assigned to TSF (Twelve Step Facilitation; the consoler tells the kid to go to more 12-step meetings) treatment did better than the kids who just had Cognitive Behavior Therapy (CBT), the control condition for this experiment.
In particular, nine months after the study ended (in the final follow-up), the group who randomly underwent TSF had a higher percentage of days abstinent (58% vs. 49% at the nine-month follow-up, overall p=0.33) and a larger number of subjects either completely or "mostly" abstinent (33% vs. 21% at the nine-month followup, overall p=0.30).
The reason why these improvements were not considered significant is because there is a pretty high p value; in plain English, a p value of 0.33 means that there is a one-in-three chance the kids who were assigned to undergo TSF treatment (i.e. they went to more AA meetings) only did better because of random chance (the kids assigned to TSF might have been, by chance, more likely to do better).
The main problem with this experiment is that there were only 59 subjects; this is not enough people for them to conclude that, while the kids who used AA treatment did better, and it is more likely than not this was because of the TSF (i.e. "get your butt to an AA meeting") treatment they received, we can not be sure this was because the randomly picked kids who got TSF treatment were ones who would do better anyway.
