Eminence or evidence, or how not to look like a fool when reporting your own data

A study presented a the ICAAC meeting was reported by the Family Practice News that piqued my interest. Firstly, it is a study on C. difficile infection treatment, and secondly it is counter to the evidence that has accumulated to date. So, I read the story very carefully, as, alas, the actual study presentation does not appear to be available.

Before I launch into the deconstruction of the data, I need to state that I do have a potential conflict of interest here. I am very involved in the CDI research from the health services and epidemiology perspective. But equally importantly, I have received research and consulting funding from ViroPharma, the manufacturer of oral Vancocin that is used to treat severe CDI.

And here is an important piece of background information: the reason the study was done. The recent evidence-based guideline on CDI developed jointly by SHEA and IDSA recommends initial treatment with metronidazole in the case of an infection that does not meet severe criteria, while advocating the use of vancomycin for severe disease. We will get into the reasons for this recommendation below.  

OK, with that out of the way, let us consider the information at hand.

My first contention is that this is a great example of how NOT to conduct a study (or how not to report it , or both). The study was a retrospective chart review at a single VA hospital in Chicago. All patients admitted between 1/09 and 3/10 who had tested positive for C. difficile toxin were identified and their hospitalizations records reviewed. A total of 147 patients were thus studied, of whom 25 (17%) received vancomycin and 122 (83%) metronidazole. It is worth mentioning that of the 122 initially treated with vancomycin, 28 (23%) were switched over to metronidazole treatment. The reasons for the switch as well as their outcomes remain obscure.

The treatment groups were stratified based on disease severity. Though the abstract states that severity was judged based on "temperature, white blood cell count, serum creatinine , serum albumin, acute mental status changes, systolic blood pressure<90, requirement for pressors," the thresholds for most of these variables are not stated. One can only assume that this stratification was done consistently and comported with the guideline.

Here is how the severity played out:

Nowhere can I find where those patients who were switched from metronidazole to vancomycin fell in these categories. And this is obviously important.

Now, for the outcomes. Those assessed were "need for colonoscopy, presence of pseudomembranes, adynamic ileus, recurrence within 30 days , reinfection > 30 days post therapy, number of recurrences >1, shock, megacolon, colon perforation, emergent colectomy, death." But what was reported? The only outcome to be reported in detail is recurrence in 30 days. And here is how it looks:

The other outcomes are reported merely as "M was equivalent to V irrespective of severity of illness (p=0.14). There was no difference in rate of recurrence (p= 0.41) nor in rate of complications between the groups (p=0.77)."
What the heck does this mean? Is the implication that the p-value tells the whole story? This is absurd! In addition, it does not appear to me from the abstract or the FPC report as if the authors bothered to do any adjusting for potential confounders. Granted, their minuscule sample size did not leave much room for that, but a lack of attempt alone invalidates the conclusion.

Oh, but if this were only the biggest of the problems! I'll start with what I think is the least of the threats to validity and work my way to the top of that heap, skipping much in the middle, as I do not have the time and the information available is full of holes. First, in any observational study of treatment there is a very strong possibility of confounding by indication. I have talked about this phenomenon previously here. I think of it as a clinician noticing something about the patient's severity of illness that does not manifest as a clear physiologic or laboratory sign, yet is very much present. A patient with this characteristic, although looking to us on paper much like one without a disease that is that severe, will be treated as someone at a higher threat level. In this case it may translate into treatment with vancomycin of patients who do not meet our criteria for severe disease, but nevertheless are severely ill. If present, this type of confounding blunts the observed differences between groups.

The lack of adjustment for potential confounding of any sort is a huge issue that negates any possibility of drawing a valid conclusion. Simply comparing groups based on severity of CDI does not eliminate the need to compare based on other factors that may be related to both the exposure and the outcome. This is pretty elementary. But again, this is minor compared to the fatal flaw.

And here it is, the final nail in the coffin of this study for me: sample size and superiority design. Firstly, the abstract and the write-up say nothing of what the study was powered to show. At least if this information had been available, we could make slightly more sense out of the p-values presented. But, no, this is nowhere to be found. As we all know, finding statistical significance is dependent on the effect size and variation within the population: the smaller the effect size and the greater the variation, the more subjects are needed to show a meaningful difference. Note, I said meaningful, NOT significant, and this they likewise neglect. What would be a clinically meaningful difference in the outcome(s)? Could 11% difference in recurrence rates be clinically important? I think so. But it is not statistically significant, you say! Bah-humbug, I say, go back and read all about the bunk that p-values represent!

One final issue, and this is that a superiority study is the wrong design here, in the absence of a placebo arm. In fact, the appropriate design is a non-inferiority study, with a very explicit development of valid non-inferiority margins that have to be met. It is true that a non-inferiority study may signal a superior result, but only if it is properly designed and executed, which this is not.

So, am I surprised that the study found "no differences" as supported by the p-values between the two treatments? Absolutely not. The sample size, the design and other issues touched on above preclude any meaningful conclusions being made. Yet this does not seem to stop the authors from doing exactly that, and the press from parroting them. Here is what the lead author states with aplomb:

              "There is a need for a prospective, head-to-head trial of these two medications, but I’m not sure who’s going to fund that study," Dr. Saleheen said in an interview at the meeting, which was sponsored by the American Society for Microbiology. "There is a paucity of data on this topic so it’s hard to say which antibiotic is better. We’re not jumping to any conclusions. There is no fixed management. We have to individualize each patient and treat accordingly."

OK, so I cannot disagree with the individualized treatment recommendation. But do we really need a "prospective head-to-head trial of these two medications"? I would say "yes," if there were not already not 1 but 2 randomized controlled trials addressing this very question. One by Zar and colleagues and another done as a regulatory study of the failed Genzyme drug tolevamer. Both of the trials contained separate arms for metronidazole and vancomycin (the Genzyme trial also had a tolevamer arm), and both stratified by disease severity. Zar and colleagues reported that in the severe CDI group the rate of clinical response was 76% in the metronidazole-treated patients versus 97% in the vancomycin group, with the p=0.02. In the tolevamer trial, presented as a poster at the 2007 ICAAC, there was an 85% clinical response rate to vancomycin and 65% to metronidazole (p=0.04).

We can always desire a better trial with better designs and different outcomes, but at some point practical considerations have to enter the equation. These are painstakingly performed studies that show a fairly convincing and consistent result. So, to put the current deeply flawed study against these findings is foolish, which is why I suspect the investigators failed to mention anything about these RCTs.

Why do I seem so incensed by this report? I am really getting impatient with both scientists and reporters for willfully misrepresenting the strength and validity of data. This makes everyone look like idiots, but more importantly such detritus clogs the gears of real science and clinical decision-making.

The value of a test

Reading this vintage paper on C diff from the Archives of Pediatric and Adolescent Medicine, I came upon this irresistible conclusion:

Priceless!

What is the harm in repeat CDI testing?

This is cross-posted on my new False Positive blog where I plan to curate some of my writing specific to research methods. 

I recently peer-reviewed a manuscript on Clostridium difficile surveillance for a journal and was prompted to reread this paper by Lance Peterson and Ari Robicsek from the Annals of Internal Medicine August 2009. The paper is a great reminder of the high risk of false positive results from repeating a moderately sensitive yet highly specific test for a disease of low prevalence.
The case in point is C. difficile testing, of specific interest to me with the enzyme immunoassays (EIA) for toxins A and/or B. A frequent clinical dilemma is what to do when a patient has symptoms and signs suggestive of C. diff infection (CDI), yet the initial test comes back negative. Frequently the clinician will repeat testing, and possibly more than once. And on occasion, the repeat testing will yield a positive result. But how accurate is this positive result?
To answer this question, Peterson and Robicsek constructed a simple model, depicted in Table 2 of the paper (I have reproduced a part of it below). The authors derived EIA characteristics from the literature to be: sensitivity = 73.3%, specificity = 97.6%. They also assumed a true prevalence of disease of 10% among patients with suspected CDI. So, starting out with a hypothetical 1,000 patient cohort with suspected CDI, the results look as follows:
What comes out loud and clear is that with every round of repeat testing, the positive predictive value degrades dramatically, until the result becomes completely useless. But to go even further, the result may be harmful since we have now gone from having 1/4 of all detected positives being false to a whopping > 1/2! And now we need to think long and hard what to do with these results. Now, one other point: Merely going from a single test to a second round, while adding a substantial number of true positives (18), ends up identifying even a greater number of people without the disease as having CDI (22)!
This illustrates the great potential for misclassification of CDI that we run into in surveillance studies, leading to an inflated disease estimates. The moral of the story is that we always need to know what proportion of positive tests are in fact repeat tests following a negative assay. And I am even beginning to think that we need to limit ourselves at most to two consecutive tests when talking about incidence and prevalence of CDI.

Our PLoS H1N1 predictions: right so far (UPDATED)

In August we published a knol in PLoS Currents: Influenza, where, based on the data to date, we modeled hospitalization, respiratory failure and death rates from H1N1. Because "preditions are hard, especially about the future", we could not be sure that we got the numbers right, especially since they were quite staggering.

Well, yesterday, the CDC and ANZIC published the US experience with the first 272 hospitalized patients and the winter flu season Australian-New Zealand ICU experience with H1N1. Here is a table that compares our predictions to what the CDC they found:

	Zilberberg knol (base case)	CDC NEJM study	ANZIC study
ICU admission		25%
ARF (relative to ICU)			65%
Acute Respiratory Failure rate [ARF] (relative to hospitalization)	12%	25% x 63% = 16%
Mortality rate (relative to ARF)	58%	Not reported	17%/65% =26%*
Mortality rate (relative to ICU)			17%
Mortality rate (relative to hospitalization)	12% x 58% = 7%	7%

*Based on assumption that all deaths occurred in ICU patients
Clearly, we got it right at least partly right -- totally consistent with the CDC numbers, and slightly higher on mortality than ANZIC. Also of interest in the CDC study is the data on antibiotics use:

Of 260 patients for whom data were available regarding antibiotic therapy, 206 (79%) received antibiotics. Of 198 patients for whom the date of initiation of antibiotics was available, such therapy was started before admission in 30 patients (15%), on admission in 117 patients (59%), within 48 hours after admission in 44 patients (22%), and more than 48 hours after admission in 7 patients (4%). Patients received a median of two antibiotics (range, one to seven); 70% of the patients received more than one antibiotic. Commonly used antibiotics included ceftriaxone (in 94 patients), azithromycin (in 84 patients), vancomycin (in 56 patients), and levofloxacin (in 47 patients). Seventy-threepercent of patients who had radiographic findings that were consistent with pneumonia were treated with antiviral drugs, and 97% were treated with antibiotics.

It is not surprising that there was such a high rate if antibiotics use, as these patients represent a group with very high severity of illness, where you throw everything at them up front and ask questions later. The rate is also similar to what was reported in the Spanish study, though those patients might have been even sicker since they were all in the ICU.

What concerns me more, however, is the 15% rate of use of outpatient antibiotics in the setting of this viral illness. I am not pointing any fingers here, as it is a decision that has to be made individually in the office by the doctor and patient. But, as I have done in the past, I do want to caution about what indiscriminate use of antibiotics could do to the incidence of C. diff and other resistant infections. We need to be very judicious in our use of these therapies as the onslaught of the H1N1 pandemic continues.

H1N1 pandemic and gut superbug

Yesterday afternoon I was on a Society of Critical Care Medicine's webcast on the ICU and H1N1 preparedness. Despite my very busy day, I had paid my $50 bucks for the privilege, and, damn it, I was going to get my money's worth. So, I made it, albeit a few minutes late.

Breathless at my monitor, I was promptly reminded by Randy Wax that in the Mexican experience over 70% of the patients with respiratory failure received antibiotics prior to hospitalization. And these are just the ones that required breathing support, representing only 18% of all the hospitalized H1N1 cases. In the recent Spanish study of H1N1 victims requiring ICU care, on the other hand, all patients received initial empiric antibiotic therapy, with 63% being treated with a fluoroquinolone.

These numbers are concerning for several reasons. Granted, these were the sickest of the sick H1N1 patients, where the ICU ethos is rightly to throw the kitchen sink and ask questions later. And it is important to note that a number of patients in the Spanish group did have a superinfection with a bacterial pathogen. But what happens if in an onslaught of cases in our hospitals later in the fall and winter engenders prodigious overuse of antibiotics in these patients?

Before there was an H1N1 pandemic, there had already existed and established pandemic of antibiotic resistance in our communities and healthcare institutions. One of these superbugs is Clostridium difficile, affectionately called C diff, which causes florid diarrhea and potentially life-threatening inflammation of the colon. This bug gets very happy when its neighbors in the gut succumb to death from antibiotics. Unopposed by its foes, it proliferates out of control and elaborates a toxin which squeezes the large intestine of all its contents, causing cramping, distention of the colon and frequent watery bowel movements. What used to be thought of as a nuisance infection primarily in chronically ill hospitalized patients, in recent years,  has mutated into a much more virulent form and is now galloping in an epidemic fashion through nearly every state in the US. More ominous is the fact that, much like its brother-in-arms MRSA, C diff is now causing diarrhea in the community setting, among people without any chronic illnesses. Furthermore, this superbug is more likely to recur despite adequate treatment and to cause much more severe disease in its victims than the bug we used to know and love (well, not really love) a decade ago.

So, how is all this relevant to the H1N1 story? One word: antibiotics! And more specifically, fluoroquinolones. This new and improved C diff seems to love fluoroquinolones. The reason for this oversized affection is that is has developed resistance to them. Practically speaking, it now has a further survival advantage over its neighbors when fluoroquinolones hit the gut.

What does all of this amount to? Well, I go back to my previous assertion that an ounce of flu prevention is going to be worth far more than a pound of cure; we need to take prevention efforts at both the personal and public levels extremely seriously. Short of that, we, both healthcare professionals and its consumers, need to say an emphatic "no" to antibiotics unless we are sure that they will help -- after all this is no longer a "might help won't hurt scenario". Finally, physicians and hospitals need to put systems into place where a critically ill patient can still get the kitchen sink before any questions are asked. But this magnitude of therapeutic interventions, including antibiotics, must be assiduously de-escalated in the absence of evidence of a bacterial superinfection. These systems need to be in place before the predicted onslaught of flu season cases. Otherwise, we may be in a situation of "our treatment was successful, but the patient died". This kind of Pyrrhic victory we can all do without!