Saturday, February 4, 2017

Another bogus 5% success rate claim found in the wild

I have found another bogus 5% success rate claim in the wild, in the comments for this post:

http://www.patheos.com/blogs/secularoutpost/2016/11/04/pseudo-random-thoughts-on-aa-and-free-will/

The claim comes from a misreading of this passage in "Alcoholics Anonymous Effectiveness: Faith Meets Science":
As stated at the outset, the experimental evidence for AA effectiveness(addressing specificity) is the weakest among the six criteria considered crucial for establishing causation. Only two studies provided strong proof of a specific AA or TSF effect: the outpatient arm of Project MATCH (with effects at 1 and 3 years), and the intensive referral condition in Timko’s trial (with effects for abstinence at 6 months and 1 year).
The effect sizes were similar, with the TSF/Intensive referral
conditions having a 5-10% advantage in abstinence rates.
(Emphasis mine)

This passage uses a lot of specialized terminology and I can see why someone would misread it to mean that AA only has a 5-10% success rate. I would render the above paragraphs like this:
As we mentioned before, the evidence for the AA program helping alcoholics (and not other factors, such as the fact that people who take AA's program seriously simply being the people who want to get sober the most) is weak. Only two studies (Note: This was written before Humphreys 2014) provided strong proof that the AA program itself, or doctors trying to get alcoholics engaged in the AA program via TSF, is helping alcoholics.
The amount of effect we saw the program helping people was similar; when doctors tried really hard to get their patients to work the AA program (TSF), the patients were 5-10% more likely to stay sober.
In other words, whether or not an alcoholic works the AA program is a decision only the alcoholic can make. If a doctor tries really hard to get someone to work the program, the alcoholic is 5-10% more likely to get sober. This figure is not an overall success rate for AA. We know that, if the alcoholic himself decides to work the program, they have, depending on the study, between a 75% (Vaillant 1995; Fiorentine 1999) and 67% (Moos and Moos 2006) chance of staying sober.

Monday, December 12, 2016

Comparison of 12-step groups to mutual help alternatives

Kaskutas,whose 2009 paper "Alcoholics Anonymous Effectiveness: Faith Meets Science" I frequently cite when pointing out that AA works for people who work it, or why early randomized controlled trials did not show an accurate picture of 12-step effectiveness, recently co-published a paper comparing 12-step groups to other mutual help groups entitled "Comparison of 12-Step Groups to Mutual Help Alternatives for AUD in a Large, National Study: Differences in Membership Characteristics and Group Participation, Cohesion, and Satisfaction."

One of the groups they looked at is a no-drinking, no-God fellowship called "LifeRing Secular Recovery." Like AA, they are very adamant about complete abstinence on their webpage: in one article they talk about the "low success rate of Moderation Management, and of individuals trying to moderate on their own," and one of their fundamental philosophies is sobriety, which they make clear means complete abstinence.

Despite this, Kaskutas' paper mentions that "LifeRing [...] members were less likely to endorse the most stringent abstinence goal," which is interesting because stringent abstinence is a fundamental part of the LifeRing program. My take on it, having not read the paper (I am not about to spend 36 dollars getting a paper which will be free in six months), is that LifeRing, and other non-AA mutual aid fellowships, mainly attract people who are not satisfied with AA. My experience working with newcomers is that the biggest stumbling block is the fact that AA strongly suggests complete abstinence for alcoholics; we frequently read a part of the Big Book describing how an alcoholic can never drink moderately again.

Indeed, the majority of people online who are opposed to AA online are people who claim alcoholics can drink moderately; it is not surprising that even an AA alternative which also advocates for abstinence attracts the kinds of people who still think they can control their drinking again.

In terms of the study itself, while showing promising results, it has the same issue that studies which claim alcoholics can drink moderately again have: It is a short-term study using self reported data.

Tuesday, May 17, 2016

A FAQ

A brief FAQ
Effectiveness of Alcoholics Anonymous

For the 20%[1][2][3] or so of alcoholics who become highly engaged with the program, Alcoholics Anonymous has about a 75%[4][5] effectiveness rate.[6]
For people who do not take AA as seriously, the number of alcoholics who achieve long-term sobriety is about 20%[7], with the amount of people achieving sobriety directly correlated to AA participation[8].
While some of the positive effects of the AA program keeping people sober can be attributed to self-selection bias, one 2014 study makes the case that AA's effectiveness shown in these studies does not come from self-selection[9]; a 2016 study demonstrates that some of the specific effect is the prayers AA members say.[10]

Other data that is inaccurate

Randomized controlled trials

Early randomized controlled trials were ineffective because the control group was not really a control group (people who wanted to stay sober went to AA anyway, even if they were not assigned to AA, and people assigned to AA did not necessarily take the program seriously), so their numbers can not be trusted.[11]

Brandsma 1980

Anti-AA advocates claim that Brandsma 1980[12] shows increased binge drinking among AA members. However 1) The study looked at about three dozen different findings at multiple follow-up periods, but this result was only seen in this one finding at one follow-up period 2) This result has not been replicated in recent studies[13] even though 3) The study is 36 years old.[14]

AA is actually not rated #37 (sometimes #38) compared to other treatments

Another common talking point popular with the anti-AA crowd is a table from Handbook of Alcoholism Treatment Approaches which ranks AA fairly lowly compared to other treatments. That table is, shall we say, weird: It rates acupuncture as one of the top 20 treatments of alcoholism, and gives motivational enhancement (#2 in the table), cognitive therapy (#13), and AA (#37) very different ratings, when the 2006 Cochrane study rate those as being about equal[15]; other studies rate AA and/or TSF (Twelve Step Facilitation: teaching patients how to go to meetings and be a part of the AA culture) as being superior.[16] No other reliably published meta-analysis comparing AA to other treatments has rated AA as being inferior.[17]

5% success rate myth #1: The Sober Truth

Lance Dodes' The Sober Truth is a recent attempt to cook numbers to claim AA has a 5% success rate (for some reason, anti-AAs have a real obsession with coming up with a 5% success rate figure, facts be ignored). While this book was widely reported in the press in 2014, its claimed 5-8% success rate for AA does not come from any peer-reviewed studies, but is instead derived by multiplying numbers from different studies together. Beresford looks at the math Dodes performed to come up with these bogus figures, and explains why the math is invalid at https://www.ncadd.org/blogs/research-update/alcoholics-anonymous-call-for-better-science.

5% success rate myth #2: AA triennial survey

There is a chart in the 1990 AA triennial survey showing that 5% of the members in their first year happened to be in their 12th month, instead of the 8.3% we would expect if AA had a 100% success rate. This has frequently been misread to indicate that only 5% of members of AA stay sober one year. An actual analysis of the graph shows a 26% retention rate.[3]

5% success rate myth #3: Reading Vaillant poorly

Some claim that Vaillant's book The Natural History of Alcoholism showed a 5% success rate, but that is a myth. The old green-papers site refuted it quite nicely: http://web.archive.org/web/20150329052802/http://green-papers.org/ "95% of patients had relapsed at some time during the study, even though many of these eventually attained sobriety. [...] It's well known that most severe alcoholics only get sober after many relapses, to the extent that relapses can be considered part of the recovery process. So don't pretend that's a failure. [...] And anyway, this was a study of a health network, not AA."

 

Alcoholics can not moderate their drinking

While not directly about AA's effectiveness, there is no reliable scientific evidence that alcoholics can drink moderately again. The studies that claim alcoholics can do so are based on self-reported information from alcoholics over period of a couple years or less. What is happening is that alcoholics have a pattern of being dishonest about their drinking, to the point that surveys using self-reported data from alcoholics are inaccurate. This has been seen time and time again: The Sobell Study from the early 1970s was shown to be a sham in 1982; the Rand report from 1976 was shown to be wrong in the 1980 follow-up; the NESARC study from 2001-2002 was a two-year study which never had a follow-up; and while Moderation Management's effectiveness has never been scientifically analyzed, its founder realized she could not control her drinking, but it was too late: Her runaway drinking resulted in her killing two people while driving drunk; she eventually killed herself.[18]

How the moderation myth tainted studies about AA

A couple of surveys from the 1970s and early 1980s erroneously reported that AA makes people who drink again have worse relapses. The reason why these surveys were inaccurate is probably because they were based on self-reported data. Since people in AA are less likely to be dishonest about their drinking, they will self-report any drinking they engage in as being more chronic, while people not in AA are more likely to dishonestly report their drinking as being "moderate" or "under control" (e.g. lying about the number of drinks they have when filling out the survey).

References
  1. The Natural History of Alcoholism Revisited by George E. Vaillant has a table showing that 19 out of 100 alcoholics attended 300 or more meetings over a 10-year period. The table is on page 197, and can be downloaded from http://web.archive.org/web/20150329052802/http://www.green-papers.org/Vaillant-Nat_hist_of_alcoholism_revisited-excerpts.pdf
  2. This article shows a roughly 80% drop out rate in Brazil: Terra, Mauro Barbosa; Barros, Helena Maria Tannhauser; Stein, Airton Tetelbom; Figueira, Ivan; Palermo, Luiz Henrique; Athayde, Luciana Dias; Gonçalves, Marcelo de Souza; Da Silveira, Dartiu Xavier (2008). "Do Alcoholics Anonymous Groups Really Work? Factors of Adherence in a Brazilian Sample of Hospitalized Alcohol Dependents". American Journal on Addictions 17 (1): 48–53. doi:10.1080/10550490701756393. PMID 18214722.  "AA adherence was below 20%"
  3. http://hindsfoot.org/recout01.pdf, while independently published and not peer-reviewed, performs statistical analysis of the unpublished 1990 triennial survey which anti-AA advocates frequently quote (since unsophisticated people reading it have incorrectly concluded it shows a 5% retention rate among AA members), showing that 26% of members surveyed make it from their first month to their twelfth month. If you insist on these figures in a more mainstream source, Dodes parrots that 26% figure in The Sober Truth.
  4. That table in The Natural History of Alcoholism Revisited has, of those 19 alcoholics who took AA seriously, 14 who achieved long-term sobriety. While the actual 14 people figure is not on this table, 29 people * 48% is 14.
  5. Beresford, Thomas (2016), Alcoholics Anonymous and The Atlantic: A Call For Better Science, National Council on Alcoholism and Drug Dependence, retrieved 2016-04-15  This page refers to Fiorentine 1999, pointing out that "the data Fiorentine presents is as follows: 77.7% of individuals who attended AA 12-step meetings at least weekly reported being free of drug use for 6 months prior to a 24-month follow-up, a finding corroborated by urinalysis at the time of the interview, and 74.8 % reported being free of alcohol use during the same time period"
  6. Another study, which measures something slightly different, has similar high success figures for people who were highly engaged with Alcoholics Anonymous: Moos, Rudolf H.; Moos, BS (June 2006). "Participation in Treatment and Alcoholics Anonymous: A 16-Year Follow-Up of Initially Untreated Individuals". Journal of Clinical Psychology 62 (6): 735–750. doi:10.1002/jclp.20259. PMC 2220012. PMID 16538654. 
  7. Of the 81 alcoholic subjects in The Natural History of Alcoholism Revisited who did not attend 300 AA meetings over a 10-year period, 15 achieved long-term sobriety
  8. See the graphs in Kaskutas, Lee Ann (2009). "Alcoholics Anonymous Effectiveness: Faith Meets Science". Journal of Addictive Diseases 28 (2): 145–157. doi:10.1080/10550880902772464. PMC 2746426. PMID 19340677. 
  9. Humphreys, Blodgett, Wagner (2014). "Estimating the efficacy of Alcoholics Anonymous without self-selection bias: an instrumental variables re-analysis of randomized clinical trials". Alcoholism: Clinical and Experimental Research 38 (11): 2688–94. doi:10.1111/acer.12557. PMC 4285560. PMID 25421504. 
  10. Marc Galanter, Zoran Josipovic, Helen Dermatis, Jochen Weber, and Mary Alice Millard (2016-03-25). "An initial fMRI study on neural correlates of prayer in members of Alcoholics Anonymous". The American Journal of Drug and Alcohol Abuse. doi:10.3109/00952990.2016.1141912. Lay summary. 
  11. There is some discussion of this in Kaskutas, Lee Ann (2009). "Alcoholics Anonymous Effectiveness: Faith Meets Science". Journal of Addictive Diseases 28 (2): 145–157. doi:10.1080/10550880902772464. PMC 2746426. PMID 19340677. 
  12. Here's the proper citation for Brandsma 1980 Brandsma, Jeffery M; Maultsby, Maxie C; Welsh, Richard J (1980). Outpatient treatment of alcoholism: a review and comparative study. Baltimore, MD: University Park Press. ISBN 0-8391-1393-5. OCLC 5219646. 
  13. While I haven't seen this cited by an anti-stepper in the wild, I should note this: Ogborne AC, Bornet A (1982). "Abstinence and abusive drinking among affiliates of Alcoholics Anonymous: are these the only alternatives?". Addict Behav. PMID 7102452.  This study, like Brandsma 1980, is a museum piece; the study uses data from the "Rand report", which is available for download and a table showing AA effectiveness there is on top of page 98 (page 114 in the PDF), showing fewer AA members who engage in "normal drinking." The notion that the Rand report showed alcoholics drinking normally again has since been discredited, see the blog entry at https://elplatt.com/return-moderate-drinking-still-lie for discussion; what that 1982 paper probably saw was that AA members are less likely to report their drinking as "normal drinking" and more likely to be honest about the negative consequences of their drinking
  14. While not a reliable source, the best discussion I have found about Brandsma 1980 is at http://slatestarcodex.com/2014/10/26/alcoholics-anonymous-much-more-than-you-wanted-to-know/, in the beginning of section III. Note that this blog dismisses longitudinal studies too easily, but, then again, it was published before Humphreys, Blodgett, Wagner 2014 demonstrated a specific effect for AA, and before Galanter 2016 showed that prayer is part of that specific effect.
  15. Ferri, Marcia; Amato, Laura; Davoli, Marina (19 July 2006). "Alcoholics Anonymous and other 12-step programmes for alcohol dependence". Cochrane Database of Systematic Reviews (3): CD005032. doi:10.1002/14651858.CD005032.pub2. PMID 16856072. 
  16. Humphreys, Keith; Moos, Rudolf (May 2001). "Can encouraging substance abuse patients to participate in self-help groups reduce demand for health care? A quasi-experimental study". Alcoholism: Clinical and Experimental Research 1 (5): 711–716. doi:10.1111/j.1530-0277.2001.tb02271.x. PMID 11371720. 12-step patients had higher rates of abstinence at follow-up (45.7% versus 36.2% for patients from CB [cognitive-behavioral] programs, p < 0.001 
  17. While not published by a mainstream press nor peer-reviewed, the blog entry at http://slatestarcodex.com/2014/10/26/alcoholics-anonymous-much-more-than-you-wanted-to-know/ devotes an entire section to this table and why it is inaccurate. Scroll down to section VI.
  18. While, yes, a blog, a good discussion with references of how alcoholics can not moderate their drinking and why any study claiming otherwise is wrong is at the blog entry https://elplatt.com/return-moderate-drinking-still-lie

Tuesday, May 10, 2016

Prayer helps keep alcoholics sober

For a long time we have known that people who choose to go to meetings are much more likely to stay sober. Recently, PMC4285560 gave evidence that AA has a “specific effect”; in other words, there is something about the AA program itself that is keeping people sober. And now, we have a new study (mirror) which tells us one aspect of AA’s specific effect: Prayer.

The actual study can be read online (mirror). To summarize, they got some people with long term sobriety, put them in MRI machines, then saw if prayers from the Big Book would affect their reactions to seeing people drinking alcohol. It did. Their level of craving of alcohol was diminished by saying 12-step prayers.

What does this tell us? It tells us that the AA program and the prayers in the Big Book can really help keep people sober.

Friday, April 29, 2016

The table in Handbook of Alcoholism Treatment Approaches

There is a table in the book Handbook of Alcoholism Treatment Approaches comparing various treatments of Alcoholism; it ranks AA as being one more the most ineffective treatments.

This table is, shall we say, weird. It rates acupuncture as one of the top 20 treatments of alcoholism, and gives motivational enhancement (#2 in the table), cognitive therapy (#13), and AA (#37) very different ratings, when the 2006 Cochrane study rate those as being about equal; other studies rate AA and/or TSF (Twelve Step Facilitation: teaching patients how to go to meetings and be a part of the AA culture) as being superior.

I also note that they only looked at seven different studies when rating AA, which leads me to believe they have only looked at old randomized controlled trials which do a terrible job of measuring AA's effectiveness.

I mean, if there was something besides this book out there arguing that acupuncture was a good deal more effective than AA, than maybe this table would convince me. But the results seem to be randomly placed on the table and I do not know of any other meta-study ranking AA as being inferior to other treatment methods. On the other hand, Cochrane 2006 said that AA is about as effective as other treatments, and Project MATCH shows that TSF is somewhat more effective than other therapies (See PMC2746426 for discussion). Fiorentine 1999, Vaillant 1995, and, yes AA's own big book give a success rate for AA around 75% for people who choose to attend one or more meeting a week; Moos & Moos 2006, which measures something slightly different, has similar figures.


Slate Star Codex discusses this table in depth in section VI (scroll down).

Saturday, April 23, 2016

The most common anti-AA talking points

There are three key studies anti-steppers like to bring out when claiming the steps do not work:
  • Ditman 1967. This is one of the earliest studies on AA, done before we knew how to properly study AA. It was a RCT -- a "randomized controlled trial." A RCT doesn't work with AA because 1) The people assigned to AA do not necessarily work the steps as suggested by oldtimers in AA meetings 2) The people not assigned to AA who are serious about getting sober end up going to the program anyway.
  • Brandsma 1980. Like Ditman 1967, this was a RCT, with the same problems. In addition, there is an opt-repeated claim that this study shows AA results in increased binge drinking. However 1) The study looked at about three dozen different findings at multiple follow-up periods, but this result was only seen in this one finding at one follow-up period 2) This result has not been replicated in recent studies [1] even though 3) The study is 36 years old.
  • The 1990 AA Triennial survey. This was not a scientific survey; this was an informal survey AA sent to members. Of the people in their first year of recovery, about 5% were in their 12th month of recovery, which is lower than the 8.3% rate we expect to see if AA had a 100% retention rate. Statistical extrapolations show that this survey shows a 29% retention rate; the question I ask is this: Is it AA's fault that 71% of newcomers decide to not come back to an AA meeting?
Note that the most recent information anti-steppers tend to bring out is over a quarter century old. What they don't like to talk about is the evidence that AA works: The more someone works AA, (That's from 2006 -- this century) the more likely they are to stay sober (That's from 2009). They dismiss this evidence with some hand-waving about "self selection", arguing that the people who stay sober by going to AA would have stayed sober anyway. While this is one possible explanation why people who go to AA stay sober, there is no evidence that this is the case. Indeed, there is a study from 2014 (under two years ago) showing that it's not just self-selection that keeps alcoholics in AA sober

Now, this study showing that it's not just self selection is just one study, and we need to have further studies to confirm that AA itself (and not just self-selection) helps keep alcoholics sober, but that's the direction the evidence points to right now. I would worry about these results being in just one study if the study was 36 years old, but this study is not even two years old.

Bottom line: If you have a drinking problem, and want to get and stay sober, AA is an effective way to do just that. 

[1] Edit: I need to point out Ogborne 1982, but this study is most likely just observing the fact that AA members are more likely to be honest about the negative consequences of their drinking.

Saturday, April 16, 2016

A reasonable overview of the AA research out there

Here is a reasonable, if not great overview of the various studies about Alcoholics Anonymous' effectiveness:

http://slatestarcodex.com/2014/10/26/alcoholics-anonymous-much-more-than-you-wanted-to-know/

The biggest defect this essay has is that it too easily dismisses longitudinal studies, claiming they are completely ineffective because of selection bias. Unfortunately, because of the peculiar nature of AA, a proper randomized controlled trial is not feasible (the people who get sober in the control group and/or non-AA-treatment group will, as likely as not, end up going to AA meetings), so longitudinal studies are a reasonable research mechanism, albeit with the thorny self selection bias issue.

PMC4285560, which was published around the same time that blog entry was made, uses sophisticated statistical methods to minimize self selection bias; once that is done, we see that there is a specific effect from the AA program that keeps people sober.

This blog entry's strength is that it has good summaries of studies which are only available in books or behind paywalls. For example, reading its summary of Brandsma 1980, a long-time chestnut anti-steppers love to bring out, shows that the study does not actually reveal a pattern of increased binge drinking among AA members (a claim which is frequently parroted by anti-steppers, such as in the "Rational" wiki).