Monday, May 19, 2014

Sell Side Bias and Scientific Stockholm Syndrome: A Report from the Annual Meeting of the American Thoracic Society

What secrets lie inside?
Analysts working on Wall Street are sometimes categorized as working on either the "buy side" or the "sell side" depending on whether their firm is placing orders for stocks (buy side, such as institutional investors for mutual funds) or filling orders for stocks (sell side, which makes commissions on stock trades).  Sell side bias refers to any tendency for the sell side to "push" stocks via overly optimistic ratings and analyses.

Well, I'm at the American Thoracic Society (ATS) meeting in San Diego right now, and it certainly does feel like people - everyone - is trying to sell me something.  From the giant industry sponsored banners, to the emblazoned tote bags, to the bags of propaganda left at my hotel room door every morning, to the exhibitor hall filled with every manner of new and fancy gadgets (but closed to cameras), to the investigators themselves, everybody is trying to convince me to buy (or prescribe) something.  Especially ideas.  Investigators have a promotional interest in their ideas.  And they want you and me to buy into their ideas.  I have become convinced that investigators without industry ties (that dying breed) are just about as susceptible to sell side bias as those with industry ties.  Indeed, I have also noted that the potential consumer of many of the ideas himself seems biased - he wants things to work, too, and he has a ready explanation for why some ideas didn't pan out in the data (see below).  It's like an epidemic of scientific Stockholm Syndrome.

The first session I attended was a synopsis of the SAILS trial by the ARDSnet investigators, testing whether use of a statin, rosuvastatin, in patients with sepsis-incited lung injury would influence 60 day mortality.  The basis of this trial was formed by observational associations that patients on statins had better outcomes in this, that, and the other thing, including sepsis.  If you are not already aware of the results, guess whether rosuvastatin was beneficial in this study.

You are correct, it was not, and there was not even a signal indicating an underlying trend towards benefit. The scientific Stockholm Syndrome was apparent in many of the questions from the audience:  "What if you had used a different statin?"  "What if you had used a higher dose?"  "What if you had selected a different population/subset, etc?"  I have to apologize that I've grown impatient with this post-hoc nitpicking.  The fact is that if you design a very solid trial like the ARDSnet did, and it fails without signal, the simplest explanation prevails:  statins are not beneficial in ARDS.  Period.

Alan Morris made some comments to a question I posed asking whether the failure of all of the ARDSnet trials save for ARMA to show improvement in the primary outcome can be attributed to feebleness of the hypotheses or intractability of the outcome (mortality), a question that I don't think was really answered.  But Alan did say, in relation to protocol driven care, that a social scientist (whose name I forget) said that the decision landscape is characterized by mostly bad decisions.  I can extend that basic principle to answer my own question.  I'm going to posit that most associations that are found via observational data are not causal associations.  That is, the biological landscape is most heavily populated by associations, and tractable causal pathways are much more rare.

So you can also guess whether statins affected outcomes in COPD (no) and whether vitamin D improved outcomes in asthmatics with low vitamin D levels (no).  Most associations are not causal.

Perhaps the most interesting thing so far to come out of ATS was last night's online first NEJM publications relating to treatments for idiopathic pulmonary fibrosis (IPF).  I will start by saying that I have a strong negative prior probability estimate that anything is going to work in this disease which is characterized by runaway inflammation and fibrosis.  But I sat back and read the three trials to see if my mind could be changed.  And I have to admit that there's something there.  In the ASCEND trial of pirfenidone in IPF the data showed a reduction in decline of FVC of about 200 cc.  In the IMPULSIS trials of nintedanib, a reduction in FVC decline of about 100 cc was found in each trial.  And in the IPF Clinical Research Network trial of n-acetylcysteine (NAC), absolutely no difference in the rate of FVC decline was found.  Without this last trial, I would have been tempted to suggest that unmasking of study assignment by side effects may have contributed to the results via effort dependence of the FVC measurements.  The NAC trial is reassuring that that was not the case.

While I was pleasantly surprised by the findings, I forced myself to step away from the punch bowl and reflect disinterestedly on their interpretation and implications.  First, there seems to be agreement among the investigators that FVC decline is the best outcome for these studies and I'm pleased to see this outcome used consistently - it allows the studies to be analyzed in aggregate much more easily.  The investigators do measure other variables, but the results of those secondary outcomes are inconsistent and I'm inclined to dismiss them as preliminary, particularly because of obvious cheery-picking in the ASCEND trial which had a hired gun medical ghost writer on the writing team.  (The reasons for that deserve some explaining.)  Second, while it is strongly correlated with clinical outcomes, the rate of change of FVC remains, after all, a surrogate endpoint, like cholesterol, blood sugar, PaO2, etc.  Third, the studies lasted just 52 weeks - I think it's imperative that we do longer term studies with clinical endpoints.  But if we (the FDA as our representative) approve these medications before such studies are conducted, what patients will allow us to randomized them to placebo in subsequent trials?  The cautionary tale was told by several decades of hurting patients with steroids and immunosuppresives.  The feeling of compulsion to "do something" for this desperate illness should not serve as a substitute for measured evaluation of the things we propose doing.

But it may be too late.  The article reporting the trials' findings in the NYT today is already among the 15 most emailed articles.  The sell side has already erected their banners, and the buy side can't get their orders in fast enough.  


  1. "I have to apologize that I've grown impatient with this post-hoc nitpicking." How do you advance the science if you don't tear apart studies, favorable or against your views?

    I can think of many reasons the vitamin D in asthma study failed without negating a potential therapeutic role in the disease. A number of recent studies are finding that there are problems with vitamin D metabolism (CYP27B1 and CYP24A1) which render vitamin D supplementation ineffective, but which might be addressed by providing calcitriol, for example.

  2. As my trusted former colleague texted me the other day, I wish there were a futures market for these therapies, yet another extension of my stock market analogies. I would retire in short order after betting against therapies such as statins and Vitamin D. Only time will tell.

  3. Awesome post!!!
    Thanks for sharing extremely nice information i am also sharing more about: Pharmaceutical Formulations Company