"The illusion of certainty"

This post was in part inspired by my recent exchange with e-Patient Dave. But really I have been thinking about this for quite some time now, and Dave provided the necessary nudge for me to write about it. Hence, his words, "the illusion of certainty".

As I like to tell my students, in science we are always on the way. We never really reach the end of the road because science is advanced through curiosity and constant questioning. For this reason, even when we think that we know something, think again. This reality is anathema not only to our model of medicine, but to our current model of society as well. What do I mean by that?

Well, let us reflect on our system of education. My son has a science workbook developed by what s considered to be a fairly progressive curriculum group. One of the early sections deals with the Linnaean classification of the animal kingdom (yes, the King Philip Came Over From Germany... well, not exactly sober). The worksheet asks the student to fill in blanks in a paragraph with the appropriate gradation of the classification. And believe it or not, the top of the page sports the full list of these strata. So, what is the message? The message, as I see it, is that this classification is so important and so solid that it requires rote memorization, as it is the only correct answer. Is this really true? Almost every day we hear about organisms requiring reclassification because we learn something new about their features that excludes them from their previous cubby. Bacteria have gained their own domain, which they did not have when I studies them in college. So, what is important here? What is important is why it is that we need these classifications, as well as how they are developed. My son is learning more from our explorations of what animals do not fit neatly into this system than from regurgitating the 7 vocabulary words. He is also learning that science is messy and the illusion of certainty that our educational systems foster are a house of cards. We could extend this argument to the way we test our children's knowledge in school as well, reducing what they learn to only one neatly packaged correct answer with all of the rest being wrong.

Now, let us wander into other parts of this American life. Take for instance the advertising industry. Cleverly they have devised market segmentation schemes that allow them to target products with very specific characteristics to a very specific population of consumers. (I wish we could be as good at doing this kind of a subgroup analysis in clinical sciences!) We are told with certainty that Charmin is better than Scotties, that Gif is more natural than Peter Pan, that buying Ford is more patriotic than buying a Toyota. And even though we are constantly faced with an absurd volume of choices, we are somehow under the impression that there is only one that is right for us.

My final example is in politics, which these days cannot be separated from religion in the US. Our politicians have realized that reducing every important issue to two ways of addressing it, and from those two ways choosing the one that can get them the most votes is indeed the way to get elected and re-elected, so that they can have a constant wedding banquet without ever getting bogged down in the hard work of marriage. This falsely reductionist and dichotomous approach has led us to a deeply divided nation, where, although we are more same than different, it is indeed these differences that are amplified to a deafening hysterical pitch. Yet, we are all secure in our certainty of being right.

I used these examples to illustrate our dependence on certainty in some major aspects of our every-day lives. This paternalism of certainty in education, politics and trusted brands likely extends to the sense of entitlement to certainty in healthcare as well. When I was in practice and on call for my group one Saturday morning, with a line out the door and the ancillary personnel about to go home for the day, I walked into one of the examining rooms to greet a woman in her 40s, well dressed and groomed, and not appearing in the least distressed. Immediately she let me know that she was being seen for a cough that was going on for a couple of days (no, she was not a smoker and did not have any chronic conditions), and wanted to nip it in the bud with a prescription for an antibiotic. After listening to her thoroughly and ascultating her lungs, I came to the conclusion that her ailment was overwhelmingly more likely to be viral than bacterial, and an antibiotic was not indicated. I shared my thinking and probability analysis with her, and following my thorough explanation of why she was going to leave my office empty-handed, she responded, "But my doctor always prescribes me an antibiotic!" I looked at her with incredulity, glanced at the growing line outside my door, and then at the clock, and, though I am not proud of it to this day, I pulled out my prescription pad. Her expectation of certainty did not leave any room for probability-based thinking, and her physician, and now I, were colluding in continuing this deception.

Today I look at medicine from a completely different perspective. As a clinical researcher, I am keenly aware of all the uncertainties inherent in what we do. The richness of research is in asking the question, and getting the answer, though it gets us published, is but a fleeting pleasure on the way to the next question. Science is the classic example of "there is no there there"; it is the road that is the gift. In the clinician's office, this attitude becomes difficult to hold together with the need for answers and certainty. Yet hold it we must. Yes, we have to give patients the best care possible based on what we know. But we cannot for a moment forget that all knowledge is evolving. This epiphany is an invitation to physicians actually to work with individual patients and their values and wishes to the extent possible. Dogmatic certainty, the alternative, is tantamount to worshipping at the altar of a false god.        

Paternalism of "science-based" medicine

This is a post I did originally for BMJ's doc2doc site. I am cross-posting it here for your convenience.

Lately I find myself getting more and more incensed with colleagues who seem to preach science-based medicine as the be-all and end-all in the truth about everything. I have blogged about this many times, and some of the references I make below are expanded upon in some of my recent posts. 

The argument goes like this: We who practice science-based medicine are the only ones that are correct in our approach because everything we do comes from a scientifically derived pool of evidence. Anything else is quackery, also referred to as "woo", which includes all modalities that do not fall in line with this EBM paradigm. They rail equally against healing crystal charlatanism, acupuncture, Reiki, and "anti-vaxers", lumping them all into the ignorant masses that dare go against what we "know" at their own and societal peril. They maintain that there is not room for anything in healthcare other than the "proven" scientific interventions.

So, if they advocate for science as the prevailing force in medicine, why should I, a firm believer in evidence as the backbone of healthcare, object? Well, here are some of my reasons, and let's see where you stand.

First of all, it is my belief that all interventions should be approached with equanimity, if not equipoise. Although I am quite dubious that either healing crystals or Reiki can produce actual results, I do not want to confuse the absence of any evidence to this effect with the evidence of absence of the effect. Although I am not that interested in allocating resources to studying these fields, it would be paternalistic of me to bar their further investigation. So the society can decide what it wants to do with them, and in the meantime every individual can make her/his own choice whether to spend their money on them. Interventions in the broad area of traditional Chinese medicine (TCM), including herbs and acupuncture, elicit less visceral aversion in me, and may be shown down the road to have desired health effects, though their study may not be amenable to our current methodologies. Again, at this time, if an individual wants to try them out, they should be keenly aware of how little we explicitly understand about them, including both their risks and potential benefits. 

The vaccine debate is yet another completely different issue altogether. In our privileged society the specter of infectious epidemics is for the most part but a distant memory, yet our concerns for the safety of our children combined with a rampant, albeit not altogether unjustified, distrust of the pharmaceutical industry and the government, have colluded to promote a biased yet durable suite of misinformation about the risk-benefit profile of vaccinations. Some of this prevailing sentiment may be because, in my opinion, we have focused on the wrong outcomes (e.g., the economics rather than morbidity associated with chicken pox), or that we have overstated what we really know and understand, at the risk of sounding too confident and therefore not trustworthy partners in this important area of public health. But these are just my own speculations. What is more important is what we do know about "science-based" medicine.

The uncomfortable truth about clinical science is that by its nature it is inexact. If it were precise, we would be able to describe it through simple mathematics without resorting to statistics, where the wild card is probability of an event. Relying on statistics is wise to understand what happens in large groups of subjects; it is altogether a mistake to take this information for God's own truth for any individual. As I have explained before, statistics are akin to that joke about assuming a spherical cow in order to help a farmer with his milk production quandary. The assumptions of normality, the dark art of log transformations, the cramming of a binary individual into a continuous distribution of population probabilities are all part and parcel of how we develop our understanding in the clinical sciences. No wonder when we bring this knowledge to the bedside a lot of the time it does not seem to apply to the specimen before us, and we are faced with having to use our judgment much more than the science-based medicine mavens are willing to admit. I often wonder how a 2-week prolongation in median survival in a cancer population, even if statistically distinguishable from the much-maligned placebo effect, is in practical terms all that useful. Yet these are effect sizes that we often see in science-based medicine.

My intent is not to belittle what clinical science has and continues to bring to the society. After all, thanks to it, we have a much better understanding of many illnesses now, and we can even take credit for effecting better outcomes for many a sick individual. At the same time we really need to sow humbleness in place of arrogance in all the places where the latter runs rampant. Philosophers of science have understood for decades that there really is no universal truth, and everything we think we know we understand through the prism of our collective human and individual experience. So, instead of wasting time on throwing boulders at our competition, why not take a closer look at the material that our house is made of. Dropping our inveterate paternalism in favor of being truthful about the degree of our uncertainty may make us look more human and fallible, yes. At the same time, it may show our patients and the public at large that we are on the same side. Though this is a risk, it is a necessary one to repair our rapidly crumbling therapeutic relationship. Admitting to not knowing is not only not a weakness, it is the only way to scientific progress.              

The reality of "science-based medicine"

Dr. Novella continues with his egregious oversimplification of the concept of science-based medicine here. I again felt compelled to respond. And given my previous difficulties with getting my responses accepted by the web site, I thought I'd post it here too. Tell me what you think.

Dear Dr. Novella,

Once again I have to agree with some of your premises, but disagree with your misguided leaps of illogic. I agree that if a modality has not been proven effective, the only way it should be left alone to be used by the public is if it has been shown to be safe. Alas, the risk-benefit equation is an individual choice, and we cannot impose our quantitative bottom line on it. Your assertion that scientific medicine is being eschewed because of acceptance of alternative modalities is as flawed as maintaining that a rain dance brings on rain. I know you said the relationship was complicated, but let's be honest: you think that CAM acceptance is killing allopathic medicine.

Now, let's get on to "proof" in science-based medicine. As you well know, while we do have evidence for efficacy and safety of some modalities, many are grandfathered without any science. Even those that are shown to have acceptable efficacy and safety profiles as mandated by the FDA, are arguably (and many do argue) not all that. There is an important concept in clinical science of heterogeneous response to treatment, HTE, which I have addressed extensively on my blog. I did not make it up, it is very real, and it is this phenomenon that makes it difficult to predict how an individual will respond to a particular intervention. This confounds much of what we think is God's own word on what is supposed to work in allopathic medicine.

Finally, do you really think that agents that are approved based on a 2-week prolongation of median survival in a desperately ill population of patients are used because of their supposed scientific merit? I would have to argue that there is a lot of subcortical emotional thinking that goes into these decisions. Can you really prove to me that a 2-week increase in median survival is not tantamount to placebo effect, aka type I error? Yet this is science-based. I think if I had a horrible disease, I might opt for acupuncture to make me feel better in the weeks I have left rather than rely on this kind of "science" to prolong my misery by 2 weeks.

Bottom line, we need to appreciate that none of the science is all that straightforward. Let us not dumb down the arguments and create false dichotomies. If we do, no one wins.

Astroturfing diseases

One of the top hits that comes up on the Oracle of Googlius for "astroturf" is for the original company, now renamed SYNLawn, but still proudly stating "We invented synthetic grass", which is what astroturf is. More recently, this already synthetic term has become a verb, as in "astroturfing". My favorite of all credible information sources on the web, Wikipedia, has this to say about it:

Astroturfing denotes political, advertising, or public relations campaigns that are formally planned by an organization, but are disguised as spontaneous, popular "grassroots" behavior. The term refers to AstroTurf, a brand of synthetic carpeting designed to look like natural grass.

I think the readers will recognize a lot of the fabricated causes taking front and center place in today's political landscape as qualifying for rovian astroturfing. In fact, you would have to be living under a dense rock to have missed this recent New Yorker article by Jane Mayer about the manufacturing of the Tea Party by the billionaire Koch brothers. A perfect example of astroturfing, if you ask me.

But what about diseases? Can there be astroturfed diseases? Is it possible that some of the 40% of Americans who have at least one chronic disease are suffering from an astroturfed disease? And is it possible that there are more astroturfed diseases cropping up every day? How can this be? That would be some futuristic dystopia, nothing to do with our reality. Well, bear with me, and see what you think.

A friend recently blogged about something called "hypoactive sexual desire disorder", affectionally abbreviated as HSDD, lending it that much more medical gravitas, as abbreviations are apt to do. Turning once again to my favorite source of information, here is how it is defined:

Hypoactive sexual desire disorder (HSDD), is considered as a sexual dysfunction and is listed under the Sexual and Gender Identity Disorders of the DSM-IV.^[1] It was first included in the DSM-III under the name Inhibited Sexual Desire Disorder^[2], but the name was changed in the DSM-III-R.

Aha, so it is a DSM-IV defined disorder. Let's see who qualifies (thanks to my friend who is a psychiatrist for this):

A. Persistently or recurrently deficient (or absent) sexual fantasies and desire for sexual activity. The judgment of deficiency or absence is made by the clinician, taking into account factors that affect sexual functioning, such as age and the context of the person's life. 

B. The disturbance causes marked distress or interpersonal difficulty.

C. Sexual dysfunction is not better accounted for by another Axis I disorder and is not due exclusively to the direct physiological effect of a substance (drug of abuse or medication) or a general medical condition (like age).

I especially love the substituted judgment here: "The judgment of deficiency or absence is made by the clinician..." So, ladies, do not try this at home -- these people are professionals.

Being that research methodologies are in my blood, I cannot help wanting more precision in order to define the condition. When precision is lacking, as in this "diagnosis", misclassification becomes a big problem. This, of course, can go either way (that is in favor of diagnosing or away from diagnosing) when the clinician has no preconceived notions about the population, the specific patient, the condition or its treatment. But how often does that happen? All of us walk around with many a preconceived notion, most concealed at the subconscious level, and we make our decisions based on those. And the insidious part is that these preconceptions in medicine can be hijacked by unethical marketing practices. Remember this story in the New York Times last week, which brilliantly described the arc of manufacturing the market for antipsychotics? Do you think that it is possible for a sales rep to increase the misclassification of HSDD in favor of more non-cases being called cases, so that they can "benefit" from treatment? You bet. My point is that, even if the syndrome is real in a small fraction of the population, the less precisely defined it is, the easier it is to grow the market by classifying even marginal or non-existent cases as disease.

But of course we should not single out psychiatry here. Even in areas with "hard" data, like cardiovascular disease, the thresholds for such quantifiable risk factors as cholesterol and blood pressure keep drifting downward with every new iteration of "evidence-based" guidelines, to the point where we now are coming up with pre-disease nosologies, such as pre-hypertension, pre-diabetes, etc. Whether these pre-diseases result in actual clinical syndromes is quite open to debate, as we are seeing even in the mammography and osteopenia debates. So, even when we complacently refer to precise and objective measures of disease definition vigilance and skepticism are required. However, from the perspective of generating markets, the earlier the disease is defined, the larger the market potential. Thus, misclassification becomes a market strategy. And harms to public's health increase.

So, is the phenomenon of astroturfing diseases real? You decide. For someone who spends her life worrying about critical illness, it is an absurd, albeit too real, idea. For someone walking in the street who yesterday was healthy but today carries a pseudodiagnosis, astroturfing is all too real. For some unethical manufacturers and clinicians, the proof is in their all too real income. For the rest of the society, we reap what they sow by paying an enormous, and very real, price in human lives, happiness and dollars.

Just as in politics, in healthcare astroturfing represents Orwellian dystopia. We as clinicians, researchers, manufacturers and citizens need to guard against it at all costs, even at the detriment to some of the economic dogmas that make up the fabric of our consumerist society. We can do this through greater emphasis on personal responsibility and communication, as well as better understanding of the nature of scientific evidence. I guarantee that we will not only seem healthier, but we will also be healthier. And perhaps happier too.      

Will healthcare be a byproduct of our math illiteracy?

OK, so when I was in practice I pretty much worked every day, including weekends, even when not on call, and can attest to the burn-out potential of this life-style. My "day off" meant that I only spent eight hours in the hospital, instead of the customary twelve to fourteen. My infant son cried when I picked him up because he did not know who I was, there was little time for such mundane pleasures as doing laundry or going grocery shopping, and bedtime reading consisted of two-to-three lines of printed text swimming on a page until the book collapsed on my chest to the accompaniment of my snoring. But let's assume that I am just a wimp, and that most of my peers can take such schedule in stride.

A propos this, I came upon this column by Peter Orszag in the Sunday Times regarding the need to increase hospital efficiencies by keeping hospitals operating 100% 24/7, including the weekends. He cites not only the financial upsides of this arrangement -- basically taking advantage of spreading many of a center's fixed costs over more days of operation -- but also the potential for improving patient safety and quality of care. What I am curious about is what it means for clinicians. Doing the mental math, when the pie size increases without a commensurate rise in the pie consumers, each pie eater has to then eat a bigger piece of the pie. This is certainly a welcome quandary  when it comes to my mother's-in-law apple pie, but for the healthcare system it is a horse of an entirely different color. The growth of the healthcare delivery pie must mean an increase in physician and staff work hours, correct? By about 40%, right? Unless of course we can increase the ranks by the equivalent 40% to share the burden.

So up until now I am pretty clear on the math. Here is where confusion begins to creep in. Being a healthcare researcher, I am keenly aware of the projections of shortages for many medical specialties. But most contradictory for the Orszag plan is the monumental shorfall of hospital nursing staff, estimated to reach over 1 million by year 2020! And similar sitution holds for pharmacists, respiratory therapists, etc., just look at some of the annual reports from the American Hospital Association. Given these dire prediction, how do we implement the Orzsag plan? Do we just ask the present personnel to give up all of their free time entirely? What do we think will happen then? Especially as they watch their professions get increasingly more commoditized and their incomes shrink?

And if these conflicting ideas are not enough, there is this headline in today's amednews: "2010 could end up as worst year for hospital layoffs". OK, I will pause for a moment to let it sink in. The story goes on

As of August, 8,233 employees had been affected by mass layoffs at hospitals. Continuing at this rate, 12,349 hospital employees would be cut in mass layoffs by the end of 2010. That would be the second-highest loss recorded by the BLS since 2000, second only to 13,282 in 2005 -- with 8,687 of those jobs lost because of hospitals shut down because of damage from Hurricane Katrina.

In 2009, 11,757 hospital employees lost their jobs through mass layoffs, according to the BLS.

OK, so what am I not seeing? Orzsag is a smart economist, but if even he cannot do this math, how can our embattled public with poor math literacy and even less understanding of the intricacies of our Byzantine healthcare bureaucracy get it?

Disruptive innovation in healthcare: Overcoming HTE

I have been working on a talk for the American College of Chest Physicians (Chest) annual meeting. The session is on alternative research study designs, and I chose to talk about the N of 1 trials. I think that you can get an idea of why I chose this topic from reading my previous posts here, here and here. Doing my research, I came upon a term "heterogeneous treatment effect", or HTE, that is well worth exploring. I think that every clinician who has ever seen patients is familiar with this effect, but let us trace its explanation.

This excellent article in the UK Independent summarizes the premiss with some scathing comments from none other than GSK's chief geneticist, Dr. Allen Roses:

"The vast majority of drugs - more than 90 per cent - only work in 30 or 50 per cent of the people," Dr Roses said. "I wouldn't say that most drugs don't work. I would say that most drugs work in 30 to 50 per cent of people. Drugs out there on the market work, but they don't work in everybody."

There is even a table presented with response rates by therapeutic area, though the reference(s) is(are) not cited, so, please, take with a grain of salt:

Response rates

Therapeutic area: drug efficacy rate in per cent

• Alzheimer's: 30
• Analgesics (Cox-2): 80
• Asthma: 60
• Cardiac Arrythmias: 60
• Depression (SSRI): 62
• Diabetes: 57
• Hepatits C (HCV): 47
• Incontinence: 40
• Migraine (acute): 52
• Migraine (prophylaxis): 50
• Oncology: 25
• Rheumatoid arthritis: 50
• Schizophrenia: 60

In essence, what Dr. Roses was referring to is the phenomenon of HTE, described aptly by Kent et al. as the fact that "the effectiveness and safety of a treatment varies across the patient population". The authors preface it by saying that 

Although “evidence-based medicine” has become the dominant paradigm for shaping clinical recommendations and guidelines, recent work demonstrates that many clinicians’ initial concerns about “evidence-based medicine” come from the very real incongruence between the overall effects of a treatment in a study population (the summary result of a clinical trial) and deciding what treatment is best for an individual patient given their specific condition, needs and desires (the task of the good clinician). The answer, however, is not to accept clinician or expert opinion as a replacement for scientific evidence for estimating a treatment’s efficacy and safety, but to better understand how the effectiveness and safety of a treatment varies across the patient population (referred to as heterogeneity of treatment effect [HTE]) so as to make optimal decisions for each patient.

Ah, so it is not your imagination: when someone brings an evidence-based guideline to you, and insists that unless you comply 95% of the time, you are providing less than great quality of care, and you say "this does not represent my patients", you are actually not crazy. To be sure, a good EBPG will apply to most patients encountered with the particular condition. But the devil, of course is in the details. As I have already pointed out, we impose statistical principles onto data to whip it into submission. When we do a good job, we acknowledge the limitations of what measures of central tendency provide us with. But so much of the time I see physicians relying on the p value alone to compare the effects, that I am convinced that the variation around the center is mostly lost on us. And further, how does this variance help a clinician faced with an individual patient who has at best a probability of response on some continuum of a population of probabilities? And more importantly, what will this individual patient's risk-benefit balance be for a particular therapy?

I think what I am walking away with after thinking about this issue is that it is of utmost importance to understand what kind of data have gone into a recommendation. What is the degree of HTE in the known research, and specifically, what is known about the population that your patient represents. The less HTE and the more knowledge about the specific subgroups, the more confident you can be that the therapy will work. Ultimately, however, each patient is a universe onto herself, since no two people will share same genetics, environmental exposures, chronic condition profile or other treatments, to name just a few potential characteristics that may impact response to therapy.

This is the reason that we need better trials, where people are represented more broadly, leading to an increase in external validity. To make this information useful at the bedside, we need a priori plans to analyze many different subgroups, as that will give clinicians at least some granularity so desperately needed in the office. And while pharmacogenomics may be helpful, I am sure that it will not be the panacea for reducing all of this complexity to zero.

Until technology gives us a better way (assuming that it will), where possible, a systematic approach to treatment trials should be undertaken. Later I will blog about N of 1 trials, which, though not appropriate in every situation, may be quite helpful in optimizing treatment in some chronic conditions. With the advent of health IT, these trials may become less daunting and, in aggregate, provide some very useful generalizable information on what happens in the real world. Each clinician will need to take some ownership in advancing our collective understanding of the diseases s/he treats. This may truly be the disruptive innovation we are all looking for to improve the quality of care not just to please the bureaucrats, but to promote better health and quality of life.      

 

Does disproving a statistical null automatically render the clinical null disproved?

A good friend of mine lost her mother to pancreatic cancer recently. The whole process from diagnosis to her death took 6 weeks. And despite wonderful care from a palliative medicine team, the process proved grueling to her family. And no wonder: how do you assimilate a loved one's going from healthy to dead in six weeks? Of course, my friend's family made all the right choices, forgoing aggressive treatment in favor of maximizing their mother's comfort and quality of life. Their experience made me think of the new generation of cancer treatments, experienced by my father in his dying days, and how it all fits in the healthcare debate. 

Much of the debate around the healthcare reform has centered around the unsustainable cost trajectory, as well as the value of evidence in improving the quality and effectiveness of care. The Congress has appropriated a substantial amount of dollars for comparative effectiveness research (CER) intended to provide data comparing one treatment to another, rather than just a treatment to a placebo. This in turn has sparked the debate as to whether to include cost-effectiveness in this comparison. I am definitely in the camp that believes it should be included for the reasons so eloquently outlined by Weinstein and Skinner. At the same time this MSNBC article made me re-evaluate its need, at least in some special cases. Let's peel back the cabbage, specifically looking at Tarceva in advanced pancreatic cancer, exactly what my friend's mother was diagnosed with.

Tarceva, or erlotinib, is a kinase inhibitor manufactured by Genentech, indicated for the treatment of some cases of non-small cell lung carcinoma, and most recently approved by the FDA for advanced pancreatic cancer. Reading the package insert, it becomes clear that the FDA-approved 100 mg dose of this drug, if given in combination with gemcitabine, prolongs median survival by <2 weeks, from 6 months in gemcitabine+placebo arm to 6.4 months in the gemcitabine+Tarceva arm, for a p=0.028. This difference imparts statistical significance at the conventionally set p<0.05, and therefore renders Tarceva better than placebo. Period. 

Delving a tad more deeply into the peer-reviewed publication of the phase 3 trial, one gleans a few other facts. I quote:

Survival and Response

The final analysis was conducted after 486 deaths (239 on erlotinib and gemcitabine and 247 on placebo and gemcitabine). Overall survival was significantly longer in the erlotinib and gemcitabine arm with an estimated HR of 0.82 (95% CI, 0.69 to 0.99; P = .038; log-rank test stratified for performance status, extent of disease, and pain score at baseline; Fig 1A). Median survival times were 6.24 months versus 5.91 months for the erlotinib and gemcitabine versus placebo and gemcitabine groups with 1-year survival rates of 23% (95% CI, 18% to 28%) and 17% (95% CI, 12% to 21%), respectively (P = .023). A multivariate Cox regression analysis showed that erlotinib treatment (HR, 0.82; 95% CI, 0.69 to 0.99; P = .04) and female sex (P = .03) were significantly associated with longer overall survival. While there was an imbalance in male:female ratio between the arms, the treatment effect remains significant when adjusted for sex.

 Results of subgroup analyses of survival by baseline stratification factors and other factors such as sex, race, pain intensity score, and age are displayed in Figure 2.

Progression-free survival was significantly longer in the erlotinib and gemcitabine arm than the placebo and gemcitabine arm with an estimated HR of 0.77 (95% CI, 0.64 to 0.92; P = .004; log-rank test stratified for performance status, extent of disease, and pain score at baseline; median, 3.75 months v 3.55 months; Fig 1B).

So, what do we have overall? We have a hazard ratio of dying that very nearly crosses 1.0, thus coming perilously close to not disproving the null; we have a prolongation of median survival by 1/3 of a month, and a progression-free median survival prolongation by 1/5 of a month.

But given my fondness for Gould's "The Median is not the Message" essay, let's practice full disclosure and look at the tail of the Kaplan-Meyer curves above. As the text points out,

...1-year survival rates of 23% (95% CI, 18% to 28%) and 17% (95% CI, 12% to 21%), respectively (P = .023).

Indeed, these are significant differences, both statistically and clinically. Within the trial this represents the difference in favor of survival for 18 additional patients, thus rendering the cost of roughly $1.5 million for 1 year of life saved by my back-of-the envelope calculation. Of course, this difference is not adjusted for confounders, so it is difficult to say of the number is real of under- or over-estimated. Because the adjusted analysis is given as a hazard ratio of death, I cannot calculate the corresponding adjusted cost.

So, for me this begs the following question (and I would love to hear the thoughts from my colleagues who proudly proclaim being science-based and thus eschewing placebo effect as a valid way to get a therapeutic response): Is what we are seeing here real or is this in fact equivalent to a placebo effect? Is the median survival prolongation of <2 weeks indeed a real effect that means something to the patient and the clinician, or is it just clinical noise, if you will? In other words, should "disproving" the statistical null by default disprove the clinical null? Or does the bar for disproving the clinical null need to be set just a tad higher than a statistically significant increase in life expectancy of 2 weeks?

Obviously, no one knows a priori who will do better and who will not. So, without a crystal ball, it is one's values and preferences that have to drive these decisions. But does the society have a say in any of this, as we struggle with equitable distribution of a limited resource? Is $1.5 million for 1 year of life a good societal investment? And what is the quality of this life? And given that at least 1/2 of all treated patients get far less than extra 2 weeks of life, how do we strike the balance between a reasonable expectation of a response and a false hope?

My friend's family based their choices on their mother's wishes and their values and utilities for her comfort. The choice they made was very different from that made by my parents with regard to my father's palliation. Both were right for the respective families. But one may have been far too costly, both financially and emotionally.    

Is VAP prevention woo science?

My students and I are continuing with the VAP theme, this week exploring the development and implementation of evidence-based practice guidelines (EBPG) through the ATS/IDSA HAP/VAP/HCAP guideline. To continue what we started last week, we are specifically talking about VAP prevention. In this EBPG, there are over 20 suggested maneuvers to prevent VAP, most of them based on level I or II evidence. One lively thread in the discussion deals with the logistics of implementing so many interventions. The complexity of codifying and introducing durably these many processes was duly acknowledged. And although I used the word "bundle" once so far, I have not alluded yet to the IHI's effort to simplify the process. As some of you know, I have been somewhat critical of our current CMS Administrator's approach to quality and safety improvements, and I have even engendered the wrath of some colleagues by publishing this evidence-driven criticism in a review paper (yes, quoting myself again):

    Given that the MV bundle simply represents a conglomeration of some of the recommended practices, it is still important to evaluate how it performs as a VAP preventive strategy for several reasons. First, for example, it is possible that one or two of the interventions chosen to be included into the bundle drive most of the VAP preventive benefit. If this is the case, then it may be inefficient to include the remaining elements, as they may divert the necessary implementation resources from the elements that truly matter. One example of seemingly simple, cheap, yet rarely attainable goal is compliance with head of the bed elevation. Some studies have indicated that a variety of reasons preclude this goal from being achieved 85% of the time, and even call its effectiveness into question (³¹). Understanding which elements of the bundle drive improvements in which populations may deprioritize head of the bed elevation as a goal to be achieved across the board. Alternatively, it may help to make a more forceful argument to improve compliance with this recommendation. Second, other evidence-based recommendations included in the EBPG, but not the bundle, may impart a greater magnitude of VAP prevention, thus once again making the current approach inefficient. For example, some educational strategies incorporating more broadly the EBPG recommendations have been demonstrated to effect a substantial reduction in the rates of VAP (^{27, 32}). Third, neither the expenditures associated with building the infrastructure for bundle implementation nor the potential return on such investment has been explicitly quantified. In general, the recent disappointing results of two meta-analyses of studies evaluating the impact of a rapid response team on hospital outcomes should serve as a cautionary note for adoption of any new process, even one with a great deal of face validity, that has not undergone rigorous testing as a whole (^{33, 34}). More importantly, in the absence of such rigorous validation scoring requiring nearly complete compliance with these processes (e.g., 95% compliance advocated in the case of the MV bundle) as a quality measure would be misguided.

The 95% compliance refers to the IHI's stipulation that an institution reach this level of implementation of all the components of the bundle in order to be considered compliant.

Although several studies out there have demonstrated that by applying a group of evidence-based preventive strategies at least some cases of VAP can be prevented, none has addressed the potential hierarchy of or interactions between the components. What is clear from the literature, however, is that a concomitant educational effort is necessary to make the guideline stick.

And how much of the effect is actually due to Hawthorne effect rather than the result of the specific intervention? I could of course argue that Hawthorne effect is not something to avoid if it helps reduce VAP rates, but then I might get accused of advocating "woo". Perish that thought!

     

VAP: A case of mistaken identity?

This week in my class we are talking about systematic reviews and meta-analyses. As in the past, I assigned this excellent example published a couple of years ago by Canadian friends:

BMJ. 2007 Apr 28;334(7599):889. Epub 2007 Mar 26.

Oral decontamination for prevention of pneumonia in mechanically ventilated adults: systematic review and meta-analysis.

Chan EY, Ruest A, Meade MO, Cook DJ.

Department of Nursing Services, Tan Tock Seng Hospital, Singapore. ee_yuee_chan@ttsh.com.sg

Comment in:

• BMJ. 2007 Apr 28;334(7599):861-2.

Abstract

OBJECTIVE: To evaluate the effect of oral decontamination on the incidence of ventilator associated pneumonia and mortality in mechanically ventilated adults.

DESIGN: Systematic review and meta-analysis.

DATA SOURCES: Medline, Embase, CINAHL, the Cochrane Library, trials registers, reference lists, conference proceedings, and investigators in the specialty.

REVIEW METHODS: Two independent reviewers screened studies for inclusion, assessed trial quality, and extracted data. Eligible trials were randomised controlled trials enrolling mechanically ventilated adults that compared the effects of daily oral application of antibiotics or antiseptics with no prophylaxis.

RESULTS: 11 trials totalling 3242 patients met the inclusion criteria. Among four trials with 1098 patients, oral application of antibiotics did not significantly reduce the incidence of ventilator associated pneumonia (relative risk 0.69, 95% confidence interval 0.41 to 1.18). In seven trials with 2144 patients, however, oral application of antiseptics significantly reduced the incidence of ventilator associated pneumonia (0.56, 0.39 to 0.81). When the results of the 11 trials were pooled, rates of ventilator associated pneumonia were lower among patients receiving either method of oral decontamination (0.61, 0.45 to 0.82). Mortality was not influenced by prophylaxis with either antibiotics (0.94, 0.73 to 1.21) or antiseptics (0.96, 0.69 to 1.33) nor was duration of mechanical ventilation or stay in the intensive care unit.

CONCLUSIONS: Oral decontamination of mechanically ventilated adults using antiseptics is associated with a lower risk of ventilator associated pneumonia. Neither antiseptic nor antibiotic oral decontamination reduced mortality or duration of mechanical ventilation or stay in the intensive care unit.

The meta-analysis looks at effectiveness of oral care in preventing VAP. Of interest was the overall finding that VAP could indeed be prevented, but preventing it altered neither mortality nor such hospital utilization parameters as duration of mechanical ventilation (MV) or ICU length of stay (LOS).

The study has precipitated a vigorous discussion in class. I will excerpt below some of my responses to the students' questions (all right, so I feel a little tacky quoting myself, but perish the thought I should be accused of plagiarizing anyone, even myself).

One of the students brought up CMS never events (you know, those hospital-acquired conditions that CMS will no longer pay for because they should never happen), and presented me with an opportunity to talk about the subtleties of VAP diagnosis within that context:  

I could not agree more that prevention is critical. The question of whether we can prevent VAP 100% of the time is a little more complicated, however. For one, we are not even sure how to diagnose VAP. Applying the CDC's surveillance definition results in rates of VAP that are quite different from invasive diagnostic testing data. Applying the same definitions to different populations results in rates that are vastly different. Furthermore, diagnostics are driven by somewhat arbitrary thresholds for bacterial counts that may not have the greatest sensitivity or specificity. So, when you are dealing firstly with the wild west of the patient and disease interaction, then add the muddy diagnostic issues to the stew, and season everything with variable processes of care, the issue to me, at least, becomes a little less straight forward. 

Then, when I asked them what is the use of preventing VAP when it does not impact such important outcomes as mortality or LOS, I got some great answers, appropriately ranging from "you are full of s**t" to "OK, my intuition tells me VAP is good to prevent, but here we have no reason for it". Some even referred to these outcome as patient-oriented, so this was a great teachable moment. So, I responded, reiterating:

I am personally a great believer in prevention, but a). it has to be sensible prevention and not just a convenient conglomeration of poorly tested modalities, and b). not everything can be prevented. There are a couple of points to make here.

CMS has not curtailed payment for VAP for the reasons that I outlined above -- what exactly constitutes VAP, what are the best preventive strategies, and exactly how good are they. CMS, on the other hand, HAS stopped paying for completely preventable errors (yes, there are such things), such as leaving instruments inside patients during surgery or administering the wrong unit of blood. These are process errors for which zero tolerance is reasonable. 

Now, on to VAP. From the Chan MA we are under the impression that there is no reason to prevent VAP, since there is no difference in patient-oriented outcomes. First, let me challenge your notion that LOS and MV duration are patient-oriented outcomes. I would argue that the patient cares less about this than about comfort, quality of life, post-critical illness functional status and the development of PTSD after the ICU stay, to name a few. These are rarely measured in RCTs. So, even if the LOS and mortality are not altered by VAP prevention, there may be other perfectly valid reasons to prevent it. Not to mention curbing the use of antibiotics to curtail the spread of resistance. 

One final point to make. Each of the studies was not powered for mortality difference. Having said that, combining the data into a very respectable total number of >3,000 patients analyzable for mortality, this should have been enough to at least show an important trend, if there was one. And in fact the VAP literature is fraught with controversy on whether VAP imparts attributable mortality or not. The LOS issue is even more complex, however. Because LOS is an infinitely variable outcome, an RCT powered to capture this difference would have to be enormously large (a measly 2000 patients would not do). However, the epidemiologic and outcomes literature abounds with data on attributable LOS and $$ due to VAP.

So, here is a valuable MA that shows that there are sound strategies to prevent at least some cases of VAP, but by implication does not justify the effort for its prevention. Here is a situation, where policy requires expert analysis. 

Bottom line, MAs are useful and dangerous at the same time. Their results, if not examined carefully and in context can worsen rather than improve care. And the MA that I assigned is actually of great quality!

I posted this to underline how tricky applying evidence can be. Particularly in an area where there is so much diagnostic confusion. This of course does not mean that we should not strive to understand things better. On the contrary, this calls for more integration of knowledge in a multidisciplinary fashion.

Fools rush in where wise people fear to tread...  

 

  

Does a flip of the switch cause the bulb to light?

"When men are most sure and arrogant they are commonly most mistaken, giving views to passion without that proper deliberation which alone can secure them from grossest absurdities"
                                                                                             --David Hume

The seemingly absurd question in the title of today's post becomes much less absurd when reading Rothman and Greenland's paper in the AJPH Supplement from 2005 found here. This paper, which is really an abridged version of a chapter in their textbook Modern Epidemiology, is creating a lot of angst among my MPH students this semester. As well it should: the authors make a compelling case for not trusting any assumptions whatsoever, and thus seem to imply that every time we think we know something, think again.

First, they talk about multicausality of any phenomenon in human biology. Pouring water on the dry ice of our cognitive abilities, they delight in vaporizing our most coveted ideas, that, for example, contributions of all causes to a condition cannot add up to more that 100%:

How can we attribute 75% of the cases to smoking and 67% to alcohol drinking among those who are exposed to both? We can because some cases are counted more than once. Smoking and alcohol interact in some cases of head and neck cancer, and these cases are attributable both to smoking and to alcohol drinking. One consequence of interaction is that we should not expect that the proportions of disease attributable to various component causes will sum to 100%.

Their discussion of causal inference is quotable for the context it provides for the various approaches of scientific inquiry. I feel that ideas in this entire section, with its apt name, are completely lost in our day-to-day scientific discourse:

Impossibility of Proof

Vigorous debate is a characteristic of modern scientific philosophy, no less in epidemiology than in other areas. Perhaps the most important common thread that emerges from the debated philosophies stems from 18th-century empiricist David Hume’s observation that proof is impossible in empirical science. This simple fact is especially important to epidemiologists, who often face the criticism that proof is impossible in epidemiology, with the implication that it is possible in other scientific disciplines. Such criticism may stem from a view that experiments are the definitive source of scientific knowledge. Such a view is mistaken on at least two counts. First, the nonexperimental nature of a science does not preclude impressive scientific discoveries; the myriad examples include plate tectonics, the evolution of species, planets orbiting other stars, and the effects of cigarette smoking on human health. Even when they are possible, experiments (including randomized trials) do not provide anything approaching proof, and in fact may be controversial, contradictory, or irreproducible. The cold-fusion debacle demonstrates well that neither physical nor experimental science is immune to such problems.

Some experimental scientists hold that epidemiologic relations are only suggestive, and believe that detailed laboratory study of mechanisms within single individuals can reveal cause–effect relations with certainty. This view overlooks the fact that all relations are suggestive in exactly the manner discussed by Hume: even the most careful and detailed mechanistic dissection of individual events cannot provide more than associations, albeit at a finer level. Laboratory studies often involve a degree of observer control that cannot be approached in epidemiology; it is only this control, not the level of observation, that can strengthen the inferences from laboratory studies. Furthermore, such control is no guarantee against error. All of the fruits of scientific work, in epidemiology or other disciplines, are at best only tentative formulations of a description of nature, even when the work itself is carried out without mistakes.

They further proceed to decimate Hill's criteria for confirming causality. In fact, in their usual scholarly fashion, having gone back to the original source, the authors convey Hill's own ambivalence:

As is evident, the standards of epidemiologic evidence offered by Hill are saddled with reservations and exceptions. Hill himself was ambivalent about the utility of these "viewpoints" (he did not use the word criteria in the paper). On the one hand, he asked, "In what circumstances can we pass from this observed association to a verdict of causation?" Yet despite speaking of verdicts on causation, he disagreed that any "hard-and-fast rules of evidence" existed by which to judge causation: This conclusion accords with the views of Hume, Popper, and others that causal inferences cannot attain the certainty of logical deductions.

Indeed, they even poke gaping holes in Bayesian thinking, which I myself am quite partial to. And incidentally, I am a big fan of Hill's "criteria", as well.

So, what is the point of assigning this paper to my students early in the course that addresses evidence generation and evaluation in policy? The paper serves as a reminder to address our assumptions systematically and regularly. By its nature, our science is imprecise. While most physical and biological processes are subject to mathematical rules, clinical sciences rely heavily on statistics, which is simply mathematics with uncertainty introduced into it. This uncertainty magnifies with every assumption that we make, and for this reason we need to be quite circumspect when insisting that we know something. In this respect, our science is more similar to civil prosecutions, where preponderance of evidence is enough for a conviction, rather that the standard of "beyond the shadow of a doubt" in criminal law.

Now, let's bring this back to the real world of clinical practice, where we do not have the luxury to ruminate on these uncertainties. This is the true intent of EBM: understand what the preponderance of evidence tells us about a phenomenon at the level of the pertinent population, but do not forget that these are mere approximations of what really happens, some closer to and some further from reality. So, while systematically observed measures of central tendency should give us comfort in the current gestalt at the bedside, the uncertainty around them is an invitation to tailor our approach to the individual before us.