ECMO and logic: Absence of Evidence is not Evidence of Absence

I have been interested in ECMO for adults with cardiorespiratory failure since the late 1990s during the Hantavirus cardiopulmonary syndrome endemic in New Mexico, when I was a house officer at the University of New Mexico. Nobody knows for sure if our use of AV ECMO there saved any lives, but we all certainly suspected that it did. There were simply too many patients too close to death who survived. It made an impression.

I have since practiced in other centers where ECMO was occasionally used, and I had the privilege of writing a book chapter on ECMO for adult respiratory failure in the interim.

But alas, I now live in the Salt Lake Valley where, for reasons as cultural as they are scientific, ECMO is taboo. The main reason for this is, I think, an over-reliance on outdated data, along with too much confidence in and loyalty to, locally generated data.

And this is sad, because this valley was hit with another epidemic two years ago - the H1N1 epidemic, which caused the most severe ARDS I have seen since the Hanta days in New Mexico. To my knowledge, no patients in the Salt Lake Valley received ECMO for refractory hypoxemia in H1N1 disease.


Thus I read with interest the Pro Con debate in Chest a few months back, and revisited in the correspondence of the current issue of Chest, which was led by some of the local thought leaders (and those who believe that, short of incontrovertible evidence, ECMO should remain taboo and outright disparaged) - See: http://chestjournal.chestpubs.org/content/139/4/965.1.citation and associated content.

It was an entertaining and incisive exchange between a gentleman in Singapore with recent ECMO experience in H1N1 disease, and our local thought leaders, themselves led by Dr. Alan Morris. I leave it to interested readers to read the actual exchange, as it is too short to merit a summary here. My only comment is that I am particularly fond of the Popper quote, taken from The Logic of Scientific Discovery: "If you insist on strict proof (or disproof) in the empirical sciences, you will never benefit from experience and never learn from it how wrong you are." Poignant.

I will add my own perhaps Petty insight into the illogical and dare I say hypocritical local taboo on ECMO. ECMO detractors would be well-advised to peruse the first Chapter in Martin Tobin's Principles and Practice of Mechanical Ventilation called "HISTORICAL PERSPECTIVE ON THE DEVELOPMENT OF MECHANICAL VENTILATION". As it turns out, mechanical ventilation for most diseases, and particularly for ARDS, was developed empirically and iteratively during the better part of the last century, and none of that process was guided, until the last 20 years or so, by the kind of evidence that Morris considers both sacrosanct and compulsory. Indeed, Morris, each time he uses mechanical ventilation for ARDS, is using a therapy which is unproved to the standard that he himself requires. And indeed, the decision to initiate mechanical ventilation for a patient with respiratory failure remains one of the most opaque areas in our specialty. There is no standard. Nobody knows who should be intubated and ventilated, and exactly when - it is totally based on gestalt, is difficult to learn or to teach, and is not even addressed in studies of ARDS. Patients must be intubated and mechanically ventilated for entry to an ARDS trial, but there are no criteria which must be met on how, when, and why they were intubated. It's just as big a quagmire as the one Morris describes for ECMO.

And much as he, and all of us, will not stand by idly and allow a spontaneously breathing patient with ARDS to remain hypoxemic with unacceptable gas exchange, those of us with experience with ECMO, an open mind, equipoise, and freedom from rigid dogma will not stand by idly and watch a ventilated patient remain hypoxemic with unacceptable gas exchange for lack of ECMO.

It is the same thing. Exactly the same thing.

The EOLIA ECMO Bayesian Reanalysis in JAMA

A Hantavirus patient on ECMO, circa 2000

Spoiler alert:  I'm a Bayesian decision maker (although maybe not a Bayesian trialist) and I "believe" in ECMO as documented here.

My letter to the editor of JAMA was published today (and yeah I know, I write too many letters, but hey, I read a lot and regular peer review often doesn't cut it) and even when you come at them like a spider monkey, the authors of the original article still get the last word (and they deserve it - they have done far more work than the post-publication peer review hecklers with their quibbles and their niggling letters.)

But to set some thing clear, I will need some more words to elucidate some points about the study's interpretation.  The authors' response to my letter has five points.

1. I (not they) committed confirmation bias, because I postulated harm from ECMO.  First, I do not have a personal prior for harm from ECMO, I actually think it is probably beneficial in properly selected patients, as is well documented in the blog post from 2011 describing my history of experience with it in hantavirus, and as well in a book chapter I wrote in Cardiopulmonary Bypass Principles and Practice circa 2006.  There is irony here - I "believe in" ECMO, I just don't think their Bayesian reanalysis supports my (or anybody's) beliefs in a rational way!  The point is that it was a post hoc unregistered Bayesian analysis after a pre-registered frequentist study which was "negative" (for all that's worth and not worth), and the authors clearly believe in the efficacy of ECMO as do I.  In finding shortcomings in their analysis, I seek to disconfirm or at least challenge no only their but my own beliefs.  And I think that if the EOLIA trial had been positive, that we would not be publishing Bayesian reanalyses showing how the frequentist trial may be a type I error.  We know from long experience that if EOLIA had been "positive" that success would have been declared for ECMO as it has been with prone positioning for ARDS.  (I prone patients too.)  The trend is to confirm rather than to disconfirm, but good science relies more on the latter.
2. That a RR of 1.0 for ECMO is a "strongly skeptical" prior.  It may seem strong from a true believer standpoint, but not from a true nonbeliever standpoint.  Those are the true skeptics (I know some, but I'll not mention names - I'm not one of them) who think that ECMO is really harmful on the net, like intensive insulin therapy (IIT) probably is.  Regardless of all the preceding trials, if you ask the NICE-SUGAR investigators, they are likely to maintain that IIT is harmful.  Importantly, the authors skirt the issue of the emphasis they place on the only longstanding and widely regarded as positive ARDS trial (of low tidal volume).  There are three decades of trials in ARDS patients, scores of them, enrolling tens of thousands of patients, that show no effect of the various therapies.  Why would we give primacy to the the one trial which was positive, and equate ECMO to low tidal volume?  Why not equate it to high PEEP, or corticosteroids for ARDS?  A truly skeptical prior would have been centered on an aggregate point estimate and associated distribution of 30 years of all trials in ARDS of all therapies (the vast majority of them "negative").  The sheer magnitude of their numbers would narrow the width of the prior distribution with RR centered on 1.0 (the "severely skeptical" one), and it would pull the posterior more towards zero benefit, a null result.  Indeed, such a narrow prior distribution may have shown that low tidal volume is an outlier and likely to be a false positive (I won't go any farther down that perilous path).  The point is, even if you think a RR of 1.0 is severely skeptical, the width of the distribution counts for a lot too, and the uninitiated are likely to miss that important point.
3. Priors are not used to "boost" the effect of ECMO.  (My original letter called it a Bayesian boost, borrowing from Mayo, but the adjective was edited out.) Maybe not always, but that was the effect in this case, and the respondents did not cite any examples of a positive frequentist result that was reanalyzed with Bayesian methods to "dampen" the observed effect.  It seems to only go one way, and that's why I alluded to confirmation bias.  The "data-driven priors" they published were tilted towards a positive result, as described above.
4. Evidence and beliefs.  But as Russell said "The degree to which beliefs are based on evidence is very much less than believers suppose."  I support Russell's quip with the aforementioned.
5. Judgment is subjective, etc.  I would welcome a poll, in the spirit of crowdsourcing, as we did here to better understand what the community thinks about ECMO (my guess is it's split ratherly evenly, with a trend, perhaps strong, for the efficacy of ECMO).  The authors' analysis is laudable, but it is not based on information not already available to the crowd; rather it transforms it in ways may not be transparent to the crowd and may magnify it in a biased fashion if people unfamiliar with Bayesian methods do not scrutinize the chosen prior distributions.

ECMO and H1N1 - more fodder for debate

There is perhaps no better way to revive the dormant blog than to highlight an article published in JAMA yesterday about the role and effect of ECMO in the H1N1 epidemic in England: http://jama.ama-assn.org/content/early/2011/09/28/jama.2011.1471.full . Other than to recognize its limitations which are similar if not identical to those of the CESAR trial, there is little to say about this study beyond that it further bolsters the arguments of my last post about ECMO and the ongoing debate about it.

In light of the recent failures of albuterol and omega-3 fatty acids in ARDS treatment, I echo the editorialist in calling for funding for a randomized controlled trial of ECMO in severe ARDS (see: http://jama.ama-assn.org/content/early/2011/09/28/jama.2011.1504.full ).

HFOV Fails as a Routine Therapy for moderate-to-severe ARDS. Musings on the Use and Study of “Rescue Therapies”.

Ferguson et al report the results of the OSCILLATE randomized controlled trial of HFOV for moderate to severe ARDS in this week’s NEJM.  (A similar RCT of HFOV, the OSCAR trial, is reported in the same issue but I limit most of my commentary to OSCILLATE because I think it’s a better and more interesting trial and more data are presented in its report.)  A major question is answered by this trial, but an important question remains open:  is HFOV an acceptable and rational option as “rescue therapy” in certain patients with “refractory” ARDS?  I remain undecided about this question, and its implications are the subject of this post.

Before I segue to the issue of the study and efficacy of rescue therapies, let’s consider some nuances of this trial:

·         Patients in both groups received high doses of sedatives (average midazolam dose for the first week: 8.3 mg/hour in the HFOV group versus 5.9 mg/hour in the control group – a 41% increase in HFOV).  Was this “too much” sedation?  What if propofol had been used instead?

·         Patients in the HFOV group received significantly more paralytics.  If you believe the Papazian data (I don’t) paralytics should confer a mortality benefit in early ARDS and this should contribute to LOWER mortality in the HFOV group.  What if paralytics had been used less frequently?

·         Does HFOV confer a nocebo effect by virtue of its “unnatural” pattern of ventilation, its “requirement” for more sedation and paralysis, or the noise associated with its provision, or its influence on the perceptions of caregivers and patient’s families (recognizing that deaths after withdrawal of life support were similar in HFOV versus conventional ventilation (55 versus 49%, P=0.12)?

·         The respiratory frequency in the HFOV group (5.5 Hz) was at the low end of the usual range (3-15 Hz).  If a higher frequency (and a lower tidal volume) had been delivered, would the result have changed?  (Probably not.)

·         What about the high plateau pressure in the control group (32 cm H2O) despite the low tidal volume of 6.1 ml/kg PBW?  Why was not tidal volume reduced such that plateau pressure was lower than the commonly recommended target of 30 cm H2O?  Did this make a difference?  (Probably not.)

·         Why was mortality higher in the minority (12%) of control patients who were changed to HFOV (71% mortality)?  Is this related to confounding by indication or reflective of the general harmful effects of HFOV?

·         Why was there a difference between the OSCILLATE study and the OSCAR study, reported in the same issue, in terms of mortality?  Because OSCILLATE patients were sicker?  Because OSCAR control patients received higher tidal volumes, thereby curtailing the advantage of conventional ventilation?  I find this last explanation somewhat compelling.

No Sham Needed in Sham Trials: Polymyxin B Hemoperfusion in Abdominal Septic Shock (Alternative Title: How Meddling Ethicists Ruin Everything)

This a superlative article to jab at to demonstrate some interesting points about randomized controlled trials that have more basis in hope than reason and whose very design threatens to invalidate their findings: http://jama.ama-assn.org/cgi/content/abstract/301/23/2445?maxtoshow=&HITS=10&hits=10&RESULTFORMAT=&fulltext=polymyxin&searchid=1&FIRSTINDEX=0&resourcetype=HWCIT . Because endotoxin has an important role in the pathogenesis of gram-negative sepsis, there has been interest in interfering with it or removing it in the hopes of abating the untoward effects of the sepsis inflammatory cascade. Learning from previous experiences/studies (e.g., http://content.nejm.org/cgi/content/abstract/324/7/429 ) that taking a poorly defined and heterogenous illness (namely sepsis) and using therapy that is expected to work in only a subset of patients with the illness (gram-negative source), the authors chose to study abdominal sepsis because they expected that the majority of patients will have gram-negatives as a causative or contributory source of infection. They randomized such patients to receive standard care (not well defined) or the insertion of a dialysis catheter with subsequent hemoperfusion over a Polymyxin B impregnated surface because this agent is known to adsorb endotoxin. The basic biological hypothesis is that removing the endotoxin in this fashion will cause amelioration of the untoward effects of the sepsis inflammatory cascade in such a way as to improve blood pressure, other phyisological parameters, and hopefully, mortality as well. There is reason to begin one's reading of this report with robust skepticism. The history of modern molecular medicine, for well over 25 years, has been polluted with the vast detritus of innumerable failed sepsis trials founded on hypotheses related to modulation of the sepsis cascade. During this period, only one agent has been shown to be efficacious, and even its efficacy remains highly doubtful to perhaps the majority of intensivists (myself excluded; see: http://content.nejm.org/cgi/content/abstract/344/10/699 ).


Mortality was not the primary endpoint in this trial, but rather was used for the early stopping rule. Even though I am currently writing an article suggesting that mortality may not be a good endpoint for trials of critical illness, this trial reminds me why the critical care community has selected this endpoint as the bona fide gold standard. Who cares if this invasive therapy increases your MAP from the already acceptable level of ~77mmHg to the supertarget level of 86? Who cares if it reduces your pressor requirements? Why would a patient, upon awakening from critical illness, thank his doctors for inserting a large dialysis catheter in him to keep his BP a little higher than it otherwise would have been? Why would he rather have a giant hole in his neck (or worse - GROIN!) than a little more levophed? If it doesn't save your life or make your life better when you recover, why do you care? We desperately need to begin to study concepts such as "return to full functionality at three (or six) months" or "recovery without persistent organ failures at x,y,z months". (This latter term I would define as not needing ongoing therapy for the support of any lingering organ failure after critical illness [that did not exist in the premorbid state], such as oxygen therapy, tracheostomy, dialysis, etc.). Should I be counted as a "save" if my existence after the interventions of the "saviors" is constituted by residence in a nursing home dependent on others for my care with waxing and waning lucidity? What does society think about these questions? We should begin to ask.

And we segue to the stopping issue which I find especially intriguing. Basing the stopping rule on a mortality difference seems to validate my points above, namely that the primary endpoint (MAP) is basically a worthless one - if it were not, or if it were not trumped by mortality, why would we not base stopping of the trial on MAP? (And if this is a Phase II or pilot trial, it should be named accordingly, methinks.) This small trial was stopped on the basis of a mortality difference significant at P=0.026 with the stopping boundary at P<0.029. I will point out again on this blog for those not familiar with it this pivotal article warning of the hazards of early stopping rules (http://jama.ama-assn.org/cgi/content/abstract/294/17/2203 ). But here's the real rub. When they got these results at the first and only planned interim analysis, (deep breath), they consulted with an ethicist. The ethicist said that it is unethical to continue the trial because to do so would be to deny this presumably effective therapy to the control group. But does ANYONE in his or her right state of mind agree that this therapy is effective on the basis of these data? And if these data are not conclusive, does not that condemn future participants in a future trial to the same unfair treatment, namely randomization to placebo? Does not stopping the trial early just shift the burden to other people? It does worse. It invalidates to large degree the altruistic motives of the participants (or their surrogates) in the current trial because stopping it early invalidated it scientifically (per the above referenced article) and because stopping it early necessitates the performance of yet another larger trial where participants will be randomized to placebo, and which, it is fair to suspect, will demonstrate this therapy to be useless, which is tantamount to harmful in the net because of the risk of catheters and wasted resources in performing yet another trial. Likewise, if we assume that this therapy IS beneficial, stopping it has reduced NET utility to current participants, because now NOBODY is receiving the therapy. So, from a consequentialist or utilitarian standpoint, overall utility is reduced and net harm has resulted from stopping the trial. What if the investigators of this trial had made it more scientifically valid from the outset by using a sham hemoperfusion device (an approach that itself would have caused an ethical maelstrom)? And what if the sham group proved superior in terms of mortality - would the ethicists have argued for stopping the trial because continuing it would mean depriving patients of sham therapy? Would there have been a call for providing sham therapy to all patients with surgically intervened abdominal sepsis? I write this with my tongue in my cheek, but the ludicrousness of it does seem to drive home the point that the premature stopping of this trial is neither ethically clear-cut nor obligatory, and that from a utilitarian standpoint, net negative utility (for society and for participants - for everyone!) has resulted from this move. And that segues me to the issue of sham procedures. It is abundantly obvious that patients with a dialysis catheter inserted for this trial (probably put in by an investigator, but not stated in the manuscript) will be likely to receive more vigilant care. This is the whole reason that protocols were developed in critical care research, as a result of the early ECMO trials (Morris et al 1994) where it was recognized that you would have all sorts of confounding by the inability to blind treating physicians in such a study. While it is not feasible to blind an ECMO study, the investigators of this study do little to convince us that blinding was not possible and feasible, and they make light of the differences in care that may have resulted from lack of blinding. Moreover, they do not report on the use of protocols for patient care that may/could have minimized the impact of lack of blinding, and in a GLARING omission, they do not describe fluid balance in these patients, a highly discretionary aspect of care that clearly could have influenced the primary outcome and which could have been differential between groups because of the lack of blinding and sham procedures. Unbelievable! (As an afterthought, even the mere increased stimulation [tactile, auditory, or visual] of patients in the intervention group, by more nursing presence or physician presence in the room may have led to increases in blood pressure.) There are also some smaller points, such as the fact that by my count 10 patients (not accounting for multiple organisms) in the intervention group had gram positive or fungal infections making it difficult to imagine how the therapy could have influenced these patients. What if patients without gram-negative organisms isolated are excluded from the analysis? Does the effect persist? What is the p-value for mortality then? And that point segues me to a final point - if our biologically plausible hypothesis is that reducing endotoxin levels with this therapy leads to improvements in parameters of interest, why, for the love of God, did we not measure and report endotoxin levels and perform secondary analyses of the effect of the therapy as a function of endotoxin levels and also report data on whether these levels were reduced by the therapy, thus supporting the most fundamental assumption of the biological hypothesis upon which the entire study is predicated?