Tuesday, December 4, 2018

Brandsma 1980 redux

Here’s a paragraph I wrote criticizing Bradsma 1980 when I restored that study to the Effectiveness of Alcoholics Anonymous article (using footnotes):
This study had poor methodology. There was no effort to stop the people in the control group from attending Alcoholics Anonymous meetings, and the "Alcoholics Anonymous" treatment patients underwent in the Brandsma study did not use actual Alcoholics Anonymous meetings. "The control condition allowed for participation in actual AA meetings, while those in the AA condition attended a weekly AA-like meeting administered by the study (that was not an actual AA meeting)"
That might not have been neutral, so another editor tried to make that paragraph more balanced:
Notably, there was no effort to stop the people in the control group from attending Alcoholics Anonymous meetings, and the "Alcoholics Anonymous" treatment patients underwent in the Brandsma study did not use community Alcoholics Anonymous meetings. A later critical analysis noted: "the control condition allowed for participation in actual AA meetings, while those in the AA condition attended a weekly AA-like meeting administered by the study (that was not an actual AA meeting)"
I feel this wording undermines the significant criticism Kaskutas 2009 has about Brandsma 1980, so I reworded it again, hopefully more neutral:
There was no effort to stop the people in the control group from attending Alcoholics Anonymous meetings. The "Alcoholics Anonymous" treatment patients underwent in the Brandsma study did not use community Alcoholics Anonymous meetings; a later analysis says that there are "concerns with the Brandsma trial which call its experimental results into question" because "the control condition allowed for participation in actual AA meetings, while those in the AA condition attended a weekly AA-like meeting administered by the study (that was not an actual AA meeting)"
This study has been heavily abused

I think this is pretty neutral, especially in light of just how problematic Brandsma 1980 is, especially when quoted out of context in anti-AA polemics. As just one example of this, here is how Dodes’s The Sober Truth describes Brandsma 1980:
The investigators found “significantly more binge drinking at the 3-month follow-up” among the people assigned to the AA-oriented meetings. As the year mark approached, the researchers noted, “All of the lay-RBT clients reported drinking less during the last 3 months. This was significantly better than the AA or the control groups at the 0.005 level [meaning the finding was highly statistically significant].” The final data led the researchers to conclude: “In this analysis the AA group was five times more likely to binge than the control group and nine times more likely than the lay-RBT group. The AA group average was 2.4 binges in the last 3 months.”
Brandsma, who passed away in 2008, is probably rolling in his grave that his legacy is this particular study that he did in 1979-1980 being misrepresented. Scott Alexander, in his blog, does not view how this study has been abused very kindly:
Brandsma (1980) is the study beloved of the AA hate groups, since it purports to show that people in Alcoholics Anonymous not only don’t get better, but are nine times more likely to binge drink than people who don’t go into AA at all.
There are a number of problems with this conclusion. First of all, if you actually look at the study, this is one of about fifty different findings. [...]
Second of all, the increased binge drinking was significant at the 6 [this is an error; it was at three months we saw the increased binge drinking] month followup period. It was not significant at the end of treatment, the 3 month [Scott probably means 6-month] followup period, the 9 month followup period, or the 12 month followup period. Remember, taking a single followup result out of the context of the other followup results is a classic piece of Dark Side Statistics and will send you to Science Hell.
Scott also has the same issues with this study Kaskutas 2009 had:
Brandsma didn’t use a real AA group, because the real AA groups make people be anonymous which makes it inconvenient to research stuff. He just sort of started his own non-anonymous group, let’s call it A, with no help from the rest of the fellowship, and had it do Alcoholics Anonymous-like stuff. On the other hand, many members of his control group went out into the community and…attended a real Alcoholics Anonymous, because Brandsma can’t exactly ethically tell them not to. So technically, there were more people in AA in the no-AA group than in the AA group.
Point being, this study has been criticized for its methodological problems, abused by people more interested in attacking AA than in objective facts, and has possibly done more harm than good in finding out the truth of Alcoholics Anonymous’s efficacy.

Wednesday, October 10, 2018

Someone's planning their next drink.

Someone's planning their next drink.

When someone takes 2,000 words to say something they could say in 300 words, it would seem they are being dishonest with themselves.

Let me look at some relevant quotes:
I could see the abysmal rate of success (and watched as members touted the 75% success rate lie over and over to newcomers to get them to join and become lifetime members).
Anyone who reads through this blog can see that I have repeatedly pointed to the scientific evidence that meeting makers make it, along the lines of 75% in the short term and 67% or so over the long term (i.e. alcoholics who go to a lot of meetings their first year sober will be sober two thirds of the time 16 years later).
Anyone, even those who have a serious drug or alcohol problem, can choose to use moderately, and contrary to popular belief, they can do so successfully.
This is plain simply not true. The scientific fact is this: When alcoholics try to moderately drink again, only a very small percentage succeed. As per Pendery 1982, only 5% of chronic alcoholics can return to moderate drinking again at the 10-year followup point.

This “freedom model” book is very recent, from 2017. It’s not the first book claiming successful moderate drinking for alcoholics; one Audrey Kishline wrote a book about how alcoholics could moderately drink again back in 1995. What happened? Ms. Kishline killed two people while driving drunk. 

Friday, August 24, 2018

The 1990 triennial survey: 26%, not 5% one year retention rate

There is an old myth that a 1990 AA Triennial survey shows a 5% one-year retention rate for Alcoholics Anonymous. This is a false belief: That graph, frequently brought up by anti-steppers with little interest in objective truth, actually shows a 26% retention rate, as pointed out in the article "Alcoholics Anonymous (AA) Recovery Outcome Rates".

The graph is not a membership retention graph; it is a simple count of, among the members in their first year, how many were around a given number of months.  If AA had a 100% retention rate, the graph would have shown 8.3% (i.e. 100% divided by 12) of members (instead of the 5% it shows) in their 12th month of sobriety.

The wording in the survey itself says it is a graph of "the number of members that have been around a given number of months"; it was not a graph of per-month membership retention.

This graph does not show an 80% first month drop out rate and it does not show a 5% one-year retention rate.  It shows a 70% first-month retention rate and a 26% one-year retention rate.

Here are a number of books and papers which claimed the 1990 Triennial survey showed a 5% retention rate before "Alcoholic Anonymous Recovery Outcome Rates" was published:
  • McIntire, Don (2000). "How Well Does A.A. Work?". Alcoholism Treatment Quarterly. 18 (4): 1. doi:10.1300/J020v18n04_01. "at 12 months the figures are 5% remaining and 95% departed."
  • Carl G. Lukefield, "Behavioral Therapy for Rural Substance Abusers", 2000 has this quote: "By 12 months, 90 percent have dropped out (McCrady & Miller, 1993)"
  • Charles Bufe, AA: Cult or Cure 1997 has this quote Quote: "AA produced a large monograph, “Comments on A.A.’s Triennial Surveys,” that analyzed the results of all five surveys done to that point. [...] AA has a 95% new-member dropout rate during the first year of attendance."
However, after "Alcoholic Anonymous Recovery Outcome Rates" was published, a number of sources all of a sudden decided that the 1990 Membership retention survey showed a 26% one-year retention rate, which just happens to be the number in that paper. Even anti-AA polemics like Dodes' (poorly-argued) The Sober Truth now use the 26% figure. For example:
  • Lance Dodes, M.D.; Zachary Dodes (2014). The Sober Truth: Debunking the Bad Science Behind 12-Step Programs and the Rehab Industry. ISBN 978-0807033159. "AA itself has published a comparable figure in a set of comments on its own thirteen-year internal survey, stating that only 26 percent of people who attend AA stay for longer than one year"
  • Adrian, Manuella (2012). "Can Failure Carefully Observed Become a Springboard to Success?". Substance Use & Misuse. 47 (13–14): 1384. doi:10.3109/10826084.2012.706178. "the Alcoholics Anonymous triennial surveys from 1977 through 1989 found that one quarter (26%) of those who first attend an AA meeting are still attending after 1 year"
Point being, the conclusion in "Alcoholic Anonymous Recovery Outcome Rates" that those old surveys showed a 26% one-year retention rate are considered reliable among treatment experts.

As Slate Star Codex puts it: "Almost everyone’s belief about AA’s retention rate is off by a factor of five because one person long ago misread a really confusing graph and everyone else copied them without double-checking."

Monday, August 13, 2018

Moderation for alcoholics: Maybe not

Let’s take a look at this article (original link); this article is an opinion from 2016 on what the science says about Alcoholics Anonymous. First of all, it accurately shows that the 2006 Cochrare review on AA effectiveness is out of date:
Ferri, Amato, and Davoli’s conclusion in a 2006 meta-analysis published in the Cochrane Review [13] has been widely quoted (see e.g., [14]): “No experimental studies unequivocally demonstrated the effectiveness of AA or [Twelve-Step Facilitation] TSF approaches for reducing alcohol dependence or problems” [13].
[...]
Several studies do support some efficacy of TS programs of recovery [15-19]. AA participation is associated with fewer drinks and more abstinent days [15-17], and recent studies show that AA attendance improves sobriety even while controlling for self-selection bias [18]. While these studies do not show unequivocal evidence of success—and are not evidence of sufficient effectiveness to recommend AA/TS programs for everyone—they do support inclusion of TS in the set of appropriate interventions.
[...]
13. Ferri M, Amato L, Davoli M. Alcoholics Anonymous and other 12-step programmes for alcohol dependence. Cochrane Database Syst Rev. 2006;(3):CD005032.
14. Frakt A. What is known about the effectiveness of AA? Incidental Economist. December 20
15. Walitzer KS, Dermen KH, Barrick C. Facilitating involvement in Alcoholics Anonymous during out-patient treatment: a randomized clinical trial. Addiction. 2009;104(3):391-401.
16. Witbrodt J, Ye Y, Bond J, Chi F, Weisner C, Mertens J. Alcohol and drug treatment involvement, 12-step attendance and abstinence: 9-year cross-lagged analysis of adults in an integrated health plan. J Subst Abuse Treat. 2014;46(4):412-419.
17. Moos RH, Moos BS. Participation in treatment and Alcoholics Anonymous: a 16-year follow-up of initially untreated individuals. J Clin Psychol. 2006;62(6):735-750.
18. Humphreys K, Blodgett JC, Wagner TH. Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials. Alcohol Clin Exp Res. 2014;38(11):2688-2694.
19. Kaskutas LA. Alcoholics Anonymous effectiveness: faith meets science. J Addict Dis. 2009;28(2):145-157.
So far so good; this shows that AA effectiveness for a subset of alcoholics is pretty much undeniable at this point, and it even links to an experimentatal study (Walitzer 2009) showing this. At this point, the article jumps the shark by giving us this inaccurate whopper without footnotes:
Additionally, TS [Twelve Step] programs promote the goal of abstinence, but moderation is a better goal for some people.
This is a really bold claim, so we would expect about seven footnotes showing research supporting this claim, like we had with the claim that AA helps some alcoholics. But, no, this claim is made without a single footnote.

I will link to two articles which make a strong case this claim is downright false:
After the disaster that was Sobell 1973, where the moderate drinkers who supposedly did well were either drunk, dead, or abstinent 10 years later, I will not believe any study showing alcoholics moderately drinking again without a 10-year followup. The reason why we need a 10-year followup is because alcoholics are really good at being dishonest about their alcoholic drinking being “moderate”, and it takes up to a decade for the drinking to get so bad the alcoholic can no longer deny their drinking is out of control.

Friday, July 20, 2018

Kelly 2017: The kids assigned to AA did better, but we can't prove it

Let's look at the paper "Incorporating 12-step program elements improves youth substance-use disorder treatment." Here, the conclusion in the abstract is this " in terms of abstinence, a novel integrated 12‐Step facilitation treatment for adolescent substance use disorder (iTSF) showed no greater benefits." However, the conclusion is inaccurate.

The kids in the study who were assigned to TSF (Twelve Step Facilitation; the consoler tells the kid to go to more 12-step meetings) treatment did better than the kids who just had Cognitive Behavior Therapy (CBT), the control condition for this experiment.

In particular, nine months after the study ended (in the final follow-up), the group who randomly underwent TSF had a higher percentage of days abstinent (58% vs. 49% at the nine-month follow-up, overall p=0.33) and a larger number of subjects either completely or "mostly" abstinent (33% vs. 21% at the nine-month followup, overall p=0.30).  

The reason why these improvements were not considered significant is because there is a pretty high p value; in plain English, a p value of 0.33 means that there is a one-in-three chance the kids who were assigned to undergo TSF treatment (i.e. they went to more AA meetings) only did better because of random chance (the kids assigned to TSF might have been, by chance, more likely to do better).

The main problem with this experiment is that there were only 59 subjects; this is not enough people for them to conclude that, while the kids who used AA treatment did better, and it is more likely than not this was because of the TSF (i.e. "get your butt to an AA meeting") treatment they received, we can not be sure this was because the randomly picked kids who got TSF treatment were ones who would do better anyway.

Tuesday, July 17, 2018

12-step class vs. 12-step fellowship

There are a few randomized controlled trials out there, such as Brandsma 1980 [1] and Stahlbrandt 2007 [2] where the "Alcoholics Anonymous" experimental condition was questionable. With both of these experiments, there was no difference between subjects who had the 12-step-like condition and the control group.

What happened?

The 12-step condition was not an actual AA meeting.

With Brandsma 1980, as Kaskutas 2009 [3] describes it, the subjects undergoing "AA" treatment weren't actually going to AA meetings:
The description of the AA condition states that the steps were used for discussion content, the group focused on newcomers, and they told patients about sponsors, but it is not clear whether the meetings were led by AA members, whether crosstalk was allowed, whether the meeting leader shared their story as part of the meeting, or whether the meeting format was what one would encounter at an actual AA meeting. The meetings may not have been open to other AA members in the community, and not been listed in the AA meeting directory, which would mean that a potentially important therapeutic ingredient of AA--the experience of longer-term members--would not have been present in the AA condition.
With Stahlbrandt 2007, here is how the supposed "AA" meetings were run:
The TSI intervention was a 3-hour formal lecture, given by
therapists trained in the 12-step method.
Like Brandsma 1980, they did not see a difference between the supposed AA condition and the control group.

Point being, if an experimental condition is supposed to be an AA meeting, the numbers will not be reliable unless the AA condition is, in fact, an actual meeting. Which it wasn't with these two studies.

More recent studies, such as Litt et al. 2009, where we give the experimental group treatment which gets them going to more real AA meetings, show that the more people go to meetings, the more days they will be sober. It is unfortunate that these poorly done surveys have incorrectly given the wrong impression about 12 step efficacy.

[1] Brandsma, Jeffery M; Maultsby, Maxie C; Welsh, Richard J (1980). Outpatient Treatment of Alcoholism: a review and comparative study. Baltimore, MD: University Park Press. ISBN 0-8391-1393-5

[2] Stahlbrandt, Henriettæ; Johnsson, Kent O.; Berglund, Mats (2007). "Cluster Randomized Trial". Alcoholism: Clinical and Experimental Research. 31 (3): 458–66. doi:10.1111/j.1530-0277.2006.00327.x. PMID 17295731

[3] Kaskutas, Lee Ann (2009). "Alcoholics Anonymous Effectiveness: Faith Meets Science". Journal of Addictive Diseases. 28 (2): 145–157. doi:10.1080/10550880902772464. PMC 2746426. PMID 19340677.

Wednesday, July 4, 2018

It helps to actually understand the studies

The Fix has no, for all intents and purposes, editorial control over what gets posted to their magazine. It's a pleasant surprise to see someone post there use peer reviewed research; it's an unpleasant surprise to see how poorly they actually quote the research.

Case in point: This 2017 article from the members of the group "Handshake Media." Two points for actually quoting peer reviewed research.  -10000 points for ignoring key points made by recent addiction research. 

Let’s see here: Discounting studies showing showing TSF/12-step effectiveness without a control group (even though the surgeon general didn't), but accepting the results of Zenmore 2017/2018 which also didn't have a control group. Ignoring experimental studies which do show TSF effectiveness (e.g. Walitzer 2009, which she should have been aware of because it was mentioned in Humphreys 2014).

Discounting Humphreys without presenting a compelling argument: This silly article claims that "Their methods were inadequate for determining whether increased participation is causally linked to increased abstinence, or whether increases in abstinence occurred without self-selection bias", but can not show any actual problem with this paper. Humphreys 2014 shows that, when looking at the increase in AA caused by a randomized experimental condition, that, among other things "The 3 month model showed that an increase in attendance of 1% (i.e., one absolute percentage point, not 1% of prior attendance) predicted an increase in PDA [Percent days abstinent from alcohol] of about a third of a percentage point (B = 0.38, p = .001)." It would seem that they did not fully read or did not understand Humphreys 2014.

Also: There's no need to link to the paywall version of Humphreys 2014; It's a PMC open access article.

The author’s bias is obvious.

That said, happy 4th of July everyone. As per request, this page is now open to moderated comments and please be patient; it may take days or weeks to moderate your comment.

Edit: To be fair, this author later conceded that Humphreys 2014 shows evidence that 12-step approaches can be helpful.