CMS never events: Evidence of smoke in mirrors?

Let me tell you a fascinating story. In 1999, I was still fresh out of my Pulmonary and Critical Care Fellowship, struggling for breath in the vortex of private practice, when a cute little paper appeared in the Lancet from a great group of researchers in Spain, describing a study performed in one large academic urban medical center's two ICUs: one respiratory and one medical. Its modest aim was to see if semi-recumbent (partly sitting up) compared to supine (lying flat on the back) positioning could reduce the incidence of that bane of the ICU, ventilator-associated pneumonia (VAP). The study was a well done randomized controlled trial, and the investigators even went so far as to calculate the power (the number needed to enroll in order to detect a pre-determined magnitude of effect [in this case an ambitious 50% reduction in clinically suspected VAP]), and this number was 182 based on the assumption of a 40% VAP prevalence in the control (supine) group. The primary endpoint was the prevalence (percentage of all mechanically ventilated [MV] patients developing) and the secondary the incidence density (number of cases among all MV patients spread over all the cumulative days of MV [patient-days of MV]) of clinically suspected VAP, based on the CDC criteria, while microbiologically confirmed VAP (also rigorously defined) served as the secondary endpoint.

Here is what they found. The study was stopped early due to efficacy (this means that the intervention was so superior to the control in reaching the endpoint that it was deemed unethical after the interim look to continue the study), enrolling only 86 patients, 39 in the intervention and 47 in the control groups. And here are the results for the primary and secondary outcomes:

So, this is great! No matter how you slice it, VAP is reduced substantially; there is a microbiologically confirmed prevalence reduction of nearly 6-fold (this is unadjusted for potential differences between groups; and there were differences!). Well, you know what's coming next. That's right, the "not so fast" warning. Let's examine the numbers in context.

First of all, if we look at the evidence-based guideline on HCAP, HAP and VAP from the ATS and IDSA, the prevalence of VAP is generally between 5 and 15%; in the current study the control group exceeds 20%. Now, for the incidence density, for years now the CDC has been keeping and reporting these numbers in the US, and the rate in patients comparable to the ones in the study should be around 2-4 cases per 1,000 MV days. In this study, no matter how you slice it, clinically or microbiologically, the incidence density is exceedingly high, more in line with some of the ex-US numbers reported in other studies. So, they started high and ended high, albeit with a substantial reduction.

Second of all, there is a wonderful flow chart in the paper that shows the enrollment algorithm. One small detail has always been somewhat obscure to me: the 4 patients in the semi-recumbent group that were excluded from analysis due to reintubation (this means that they were taken off MV, but had to go back on it within a day or two), which was deemed a protocol violation. Now, you might think that 4 patients is a pretty small number to worry about. But look at the total number of patients in the group: 39. If the excluded 4 all had microbiologically confirmed VAP, that would bring our prevalence from 5% to 14% (6 out of 43). This would certainly be a less than 6-fold reduction in VAP.

Thirdly, and this I think is critical, the study was not blinded. In other words, the people who took care of the patients knew the group assignment. So what, you ask. Well remember that VAP is a pretty difficult, elusive and unclear diagnosis. So, let us pretend that I am a doc who is also an investigator on the study, and I am really invested in showing how marvelous semi-recumbent positioning is for VAP prevention. I am likely to have a much lower threshold for suspecting and then diagnosing VAP in the comparator group than in my pet intervention group. And this is not an indictment of anyone's judgment or integrity; it is just how our brains are wired.

Next, there were indeed important differences between groups in their baseline risk factors for VAP. For example, more patients in the control (38%) than in the intervention (26%) group were on MV for a week or longer, the single most important risk factor for developing VAP. Likewise, the baseline severity of illness was higher in the control than the intervention group. To be sure, the authors did statistical analyses to adjust these differences away, and still found an adjusted odds ratio of VAP among the supine group to be 6.8, with the 95% confidence interval between 1.7 and 26.7. This is generally taken to mean that, on average, the risk of VAP increases nearly 7-fold for supine position as opposed to semi-recumbent, and if the trial was repeated 100 times, 95 of those times this estimate would fall between a 1.7 and a 26.7-fold increase. OK, so we can accept this as a possible viable strategy, right?

But wait, there is more. Remember what we said about the odds ratio? When the event happens in more than 10% of the sample, the odds ratio vastly overestimates the risk of this event. 28.4% anyone?

Now, let's put it all together. A single center study from a Spanish academic hospital, among respiratory and medical ICU patients, with a minuscule sample size, yet halted early for efficacy, an exceedingly high baseline rate of VAP, a substantial number of patients excluded for a nebulous reason, unblinded and therefore prone to biased diagnosis, reporting an inflated reduction in VAP development in the intervention group. It would be very easy to write this off as a flawed study (like all studies tend to be in one way or another) in need of confirmatory evidence, if it were not so critical in the current punitive environment of quality improvement. (By the way, to the best of my knowledge, there is no study that replicates these results). The ATS/IDSA guideline includes semi-recumbent positioning as a level I (highest possible level of evidence) recommendation for VAP prevention, and it is one of the elements of the MV bundle, as promoted by the Institute for Healthcare Improvement, which demands 95% compliance with all 5 elements of the bundle in order to get the "compliant" designation. And even this is not the crux of the matter. The diabolical detail here is that CMS is creeping up on making VAP into one of their magical "never" events, and the efforts by hospitals will most assuredly be including this intervention. So, ICU nurses are already expected to fall in step with this deceptively simple yet not-so-easily executable practice.

And this is what is under the hood of just one simple level I recommendation by two reputable professional organizations in their evidence-based guidelines. One shudders to think...              

CMS "never events": Are we rewarding the right thing?

A while ago I promised to write more about my thoughts inspired by Dan Ariely's Predictably Irrational. I still plan to do this, but in the meantime, this blog post by him made me think of medicine, and more specifically our incentives structure to prevent hospital-acquired complications (HACs). I am sure that by now everyone has heard of the "never events", so named because in the view of the Centers for Medicare and Medicaid Services (CMS) they should never happen. CMS is so convinced that they can be eradicated completely that they have eliminated payments for these complications in their effort to get hospitals to get serious about prevention. I do not have to tell you that hospitals these days are in dire financial straights, and every penny counts; so CMS is really getting them where it hurts. Here is the current list of never events, and it is likely to grow over time:

In fact, these very events are now being publicly reported as well, to help healthcare consumers navigate the quality labyrinth.

While waging an all out assault on these HACs is the absolute right thing to do, the question remains: Is it reasonable to expect zero rates of these complications? Well, for some definitely, while others I am not so sure. Take for example an object left inside the body during a surgery. Or how about amputation of the wrong limb? Or what about getting the wrong unit of blood intended for another patient? All of these examples are egregious consequences of human and systems error, and it is completely reasonable and even desirable to expect them never to happen. On the other hand, complications such as catheter-associated UTI or catheter-associated blood stream infection may not be as easily eliminated, and in fact may be impossible to eliminate completely. Why? Here is where Dan Ariely's analysis comes in.

He starts off by presenting a scenario of a soccer player kicking a ball with his eyes closed. It is clear that in the absence of intervening meddling from the wind or a playful dog (Fluffy, if you must know), the player can quickly perfect his aim despite not seeing the goal. However, it is these unpredictable events that interfere with the direction of the ball that are precisely the point, and can be applied to a business situation:

The problem is that there’s plenty of random noise in competitive strategic decisions. Predicting where the ball will go is equivalent to deciding whether to open a chain of seafood restaurants on the Gulf Coast. The dog running off with the ball is the BP oil spill. When the board reviews the manager’s performance, they’ll focus on the failed restaurants. The stock is down. The chain lost money. Since the manager’s compensation is tied to results, he’ll incur financial penalties. To save face and appear to be taking action, the board may even fire him—thus giving up on someone who may be a good manager but had bad luck.

So, in business what is rewarded or punished is not a systematic approach to decision making, but that random noise responsible for either getting the ball into the goal or alternatively diverting its trajectory. And this of course says nothing about either the soccer player or the business manager.

This situation is certainly analogous to the incentive structure for never events, where the hospital becomes the soccer player with its eyes closed aiming for the goal, but gets derailed by Fluffy, in this case in the persona of the patient's susceptibility to infection, say, and gets dinged for something that is completely random and not in the control of the clinician. In fact, Ariely unequivocally states something rather startling:

The oil spill example is an extreme case. In the real world, the random noise is often more subtle and various—a hundred little things rather than one big thing. But the effect is the same. Rewarding and penalizing leaders based on outcomes overestimates how much variance people actually control. (This works both ways: Just as good managers can suffer from bad outcomes not of their own making, bad managers can be rewarded for good outcomes that occur in spite of their ineptitude.) In fact, the more unpredictable an environment becomes, the more an outcomes-based approach ends up rewarding or penalizing noise. [emphasis mine]

Unpredictable you say? What can be more unpredictable and filled with uncertainty than the interaction of human biology with disease and external interventions? This repositioning of the argument really confirms for me that demanding never events is really just window dressing. For a long time there has been a joke among health services researchers that, over the next few years, these events will disappear, but not because these events actually disappear. Sophisticated hospital coders will simply stop putting these outcomes in their documentation to the payers. And will this achieve better care or just better looking documentation? I think you know what I am driving at.

What is the answer? Well, here is what Ariely suggests:

We can’t entirely avoid outcome-based decisions. Still, we can reduce our reliance on stochastic outcomes. Here are four ways companies can create more-sound reward systems.

1. Change the mind-set. Publicly recognize that rewarding outcomes is a bad idea, particularly for companies that deal in complex and unpredictable environments.

2. Document crucial assumptions. Analyze a manager’s assumptions at the time when the decision takes place. If they are valid but circumstances change, don’t punish her, but don’t reward her, either.

3. Create a standard for good decision making. Making sound assumptions and being explicit about them should be the basic condition for getting a reward. Good decisions are forward-looking, take available information into account, consider all available options, and do not create conflicts of interests.

4. Reward good decisions at the time they’re made. Reinforce smart habits by breaking the link between rewards and outcomes.

And even though this advice is aimed at business leaders, it can be easily adapted to medicine. If we heed this warning, we may avoid having to wring our hands in another 10 years about the "unintended consequence" of losing critical data needed to understand how we are really doing with improving healthcare outcomes.

I will end with Ariely's own words, since I can do no better to summarize (you can substitute appropriate healthcare related terms for the business ones):

Our focus on outcomes is understandable. When a company loses money, people demand that heads roll, even if the changes are more about assuaging shareholders than sound management. Moreover, measuring outcomes is relatively easy to do; decision-making–based reward systems will be more complex. But as I’ve said before, “It’s hard” is a terrible reason not to do something. Especially when that something can help reward and retain the people best able to help you grow your business.     

VAP: A case of mistaken identity?

This week in my class we are talking about systematic reviews and meta-analyses. As in the past, I assigned this excellent example published a couple of years ago by Canadian friends:

BMJ. 2007 Apr 28;334(7599):889. Epub 2007 Mar 26.

Oral decontamination for prevention of pneumonia in mechanically ventilated adults: systematic review and meta-analysis.

Chan EY, Ruest A, Meade MO, Cook DJ.

Department of Nursing Services, Tan Tock Seng Hospital, Singapore. ee_yuee_chan@ttsh.com.sg

Comment in:

• BMJ. 2007 Apr 28;334(7599):861-2.

Abstract

OBJECTIVE: To evaluate the effect of oral decontamination on the incidence of ventilator associated pneumonia and mortality in mechanically ventilated adults.

DESIGN: Systematic review and meta-analysis.

DATA SOURCES: Medline, Embase, CINAHL, the Cochrane Library, trials registers, reference lists, conference proceedings, and investigators in the specialty.

REVIEW METHODS: Two independent reviewers screened studies for inclusion, assessed trial quality, and extracted data. Eligible trials were randomised controlled trials enrolling mechanically ventilated adults that compared the effects of daily oral application of antibiotics or antiseptics with no prophylaxis.

RESULTS: 11 trials totalling 3242 patients met the inclusion criteria. Among four trials with 1098 patients, oral application of antibiotics did not significantly reduce the incidence of ventilator associated pneumonia (relative risk 0.69, 95% confidence interval 0.41 to 1.18). In seven trials with 2144 patients, however, oral application of antiseptics significantly reduced the incidence of ventilator associated pneumonia (0.56, 0.39 to 0.81). When the results of the 11 trials were pooled, rates of ventilator associated pneumonia were lower among patients receiving either method of oral decontamination (0.61, 0.45 to 0.82). Mortality was not influenced by prophylaxis with either antibiotics (0.94, 0.73 to 1.21) or antiseptics (0.96, 0.69 to 1.33) nor was duration of mechanical ventilation or stay in the intensive care unit.

CONCLUSIONS: Oral decontamination of mechanically ventilated adults using antiseptics is associated with a lower risk of ventilator associated pneumonia. Neither antiseptic nor antibiotic oral decontamination reduced mortality or duration of mechanical ventilation or stay in the intensive care unit.

The meta-analysis looks at effectiveness of oral care in preventing VAP. Of interest was the overall finding that VAP could indeed be prevented, but preventing it altered neither mortality nor such hospital utilization parameters as duration of mechanical ventilation (MV) or ICU length of stay (LOS).

The study has precipitated a vigorous discussion in class. I will excerpt below some of my responses to the students' questions (all right, so I feel a little tacky quoting myself, but perish the thought I should be accused of plagiarizing anyone, even myself).

One of the students brought up CMS never events (you know, those hospital-acquired conditions that CMS will no longer pay for because they should never happen), and presented me with an opportunity to talk about the subtleties of VAP diagnosis within that context:  

I could not agree more that prevention is critical. The question of whether we can prevent VAP 100% of the time is a little more complicated, however. For one, we are not even sure how to diagnose VAP. Applying the CDC's surveillance definition results in rates of VAP that are quite different from invasive diagnostic testing data. Applying the same definitions to different populations results in rates that are vastly different. Furthermore, diagnostics are driven by somewhat arbitrary thresholds for bacterial counts that may not have the greatest sensitivity or specificity. So, when you are dealing firstly with the wild west of the patient and disease interaction, then add the muddy diagnostic issues to the stew, and season everything with variable processes of care, the issue to me, at least, becomes a little less straight forward. 

Then, when I asked them what is the use of preventing VAP when it does not impact such important outcomes as mortality or LOS, I got some great answers, appropriately ranging from "you are full of s**t" to "OK, my intuition tells me VAP is good to prevent, but here we have no reason for it". Some even referred to these outcome as patient-oriented, so this was a great teachable moment. So, I responded, reiterating:

I am personally a great believer in prevention, but a). it has to be sensible prevention and not just a convenient conglomeration of poorly tested modalities, and b). not everything can be prevented. There are a couple of points to make here.

CMS has not curtailed payment for VAP for the reasons that I outlined above -- what exactly constitutes VAP, what are the best preventive strategies, and exactly how good are they. CMS, on the other hand, HAS stopped paying for completely preventable errors (yes, there are such things), such as leaving instruments inside patients during surgery or administering the wrong unit of blood. These are process errors for which zero tolerance is reasonable. 

Now, on to VAP. From the Chan MA we are under the impression that there is no reason to prevent VAP, since there is no difference in patient-oriented outcomes. First, let me challenge your notion that LOS and MV duration are patient-oriented outcomes. I would argue that the patient cares less about this than about comfort, quality of life, post-critical illness functional status and the development of PTSD after the ICU stay, to name a few. These are rarely measured in RCTs. So, even if the LOS and mortality are not altered by VAP prevention, there may be other perfectly valid reasons to prevent it. Not to mention curbing the use of antibiotics to curtail the spread of resistance. 

One final point to make. Each of the studies was not powered for mortality difference. Having said that, combining the data into a very respectable total number of >3,000 patients analyzable for mortality, this should have been enough to at least show an important trend, if there was one. And in fact the VAP literature is fraught with controversy on whether VAP imparts attributable mortality or not. The LOS issue is even more complex, however. Because LOS is an infinitely variable outcome, an RCT powered to capture this difference would have to be enormously large (a measly 2000 patients would not do). However, the epidemiologic and outcomes literature abounds with data on attributable LOS and $$ due to VAP.

So, here is a valuable MA that shows that there are sound strategies to prevent at least some cases of VAP, but by implication does not justify the effort for its prevention. Here is a situation, where policy requires expert analysis. 

Bottom line, MAs are useful and dangerous at the same time. Their results, if not examined carefully and in context can worsen rather than improve care. And the MA that I assigned is actually of great quality!

I posted this to underline how tricky applying evidence can be. Particularly in an area where there is so much diagnostic confusion. This of course does not mean that we should not strive to understand things better. On the contrary, this calls for more integration of knowledge in a multidisciplinary fashion.

Fools rush in where wise people fear to tread...