Peeling the cabbage of "works" in treatment interventions

What exactly does it mean when we say that a treatment works? Do we mean the same thing for all treatments? Are there different ways of assessing whether and how well a treatment works? I am sure you've guessed that I wouldn't be asking this question if the answer were simple. And indeed, the answer is "it depends."

What I am talking about is examining outcomes. I did a post a couple of years ago here, where I use the following quote from a Pharma scientist:

"The vast majority of drugs - more than 90 per cent - only work in 30 or 50 per cent of the people," Dr Roses said. "I wouldn't say that most drugs don't work. I would say that most drugs work in 30 to 50 per cent of people. Drugs out there on the market work, but they don't work in everybody."

Here is that word "work" again. What does this mean? well, let's take such common condition as heart disease. What does heart disease do to a person? Well, it can do many things, including give him/her such symptoms as a chest pain, shortness of breath, dizziness and palpitations, to name a few. These symptoms may have at least two sets of implications: 1) they are bothersome to the individual, and in this way may impair his/her enjoyment of life, and 2) they may signal either a present or a future risk of a heart attack. Why are heart attacks important? Well, they are important because one may kill the person who is having it, or one (or several) may weaken the heart to the point of a substantial disability and thus a deterioration in the quality of life. So, there certainly seems to be a good rationale to prevent heart disease either from happening in the first place or from at least worsening when it's already established.


Now, what's available to us to prevent heart disease? Well, some think that lowering one's cholesterol is a good thing. OK, let's go with that. What is the sign that the statins (cholesterol-lowering drugs) "work"? What would it look like if it was about lowering the cholesterol? Say, your total cholesterol is 240. You go on a statin and in 6 months your total cholesterol is 238. Your cholesterol was lowered, it worked! Well, yes, but if you are asking what this 2-point drop really accomplishes, you are beginning to understand the meaning of "work." So, just intuitively we can say that there needs to be a certain, perhaps "clinically significant," drop in the total cholesterol in order for us to say that the drug "worked." 


Great! Now we are sidling up to the real issue: What constitutes a "clinically significant" drop in cholesterol? Is it some arbitrary number that looks high enough? Probably not. How about some drop that correlates to a drop in the risk of the actual condition we are trying to impact, heart disease? Say, a 40-point drop, or getting to below 200, may be the right threshold for the "works" judgment. Ah, but there is yet another question to ask: How often does this type of a drop lead to a reduction in heart disease? Is it always (not likely), or is it the majority of the time (rarely) or at least some of the time (most likely in clinical medicine)? And what portion of that time do we consider satisfactory -- 60%? 40%? 20%? 2%? 


Let me bring just one more layer into this discussion. Many people walk with heart disease and don't know that they have it. Some of these people are destined to have a heart attack and/or die from it. Many others are likely to die from something else before they ever experience any symptoms or signs of their heart disease. This raises the question of whether the statins' ability to reduce cholesterol and hence reduce the risk of heart disease is enough to say that the drugs "work." Perhaps "work" means that by lowering cholesterol (say in the majority of those who take it) they reduce the risk of hear disease in some proportion of those who are at risk for it, and among that proportion whose risk is reduced they also reduce the risk of a heart attack in a few, and of death in even fewer. 


So, to sum up, "works" is a loaded term. For the case we are discussing, there is what I call a "dwindle" effect, where the main outcome, cholesterol lowering, is likely to show a somewhat robust result. On the other hand, this (surrogate) outcome itself is not all that interesting when divorced from what we really care about -- symptoms, heart attacks and death. And I haven't even gone into the side of the equation where the patient gets to decide what "work" means for him/herself. The layers of the possible "works" are a cabbage that we all need to peel when discussing treatment plans with our clinicians and when reading news about new technologies.                     
 

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Unpacking the meat data

So, this story from Harvard on how red meat is bad for you deserves some unpacking. First, allow me to say that all meat is not created equal: the cows that graze on the farm around the corner from where I live make meat that is quite different from that reconstituted slime used by fast-"food" restaurants. Cows that are raised on CAFOs and fed corn-based diets are practically different species from those guernseys down the street.

But putting that aside, let's just look at what the paper reports and what the numbers add up to. The investigators examined two large observational cohort studies totaling over 100,000 subjects and tried to estimate the risk of death associated with red meat consumption. Now, first, it has been widely acknowledged that dietary habit surveys are a difficult beast, and that is how these two studies got at the food history. Next, let us look at some of the numerators and denominators. The paper reports 23,926 deaths among these >100,000 subjects over 22 to 28 years of observation. The denominator for this type of a study is person-years, where you simply multiply the number of persons observed by the corresponding number of years of observation. In this instance, this value is 2,960,000 person-years. So, the roughly 24,000 deaths occurred over 2.96 million years of observation, simplifying to 24,000/2,960,000 = 8 deaths per 1,000 years overall. If we were to translate this to an individual's risk for death over 1 year, it would be 0.008, or under 1%.

The study further reports that at its worst, meat increases this risk by 20% (95% confidence interval 15-24%, for processed meat). If we use this 0.8% risk per year as the baseline, and raise it by 20%, it brings us to 0.96% risk of death per year. Still, below 1%. Need a magnifying glass? Me too. Well, what if it's closer to the upper limit of the 95% confidence interval, or 24%? The risk still does not quite get up to 1%, but almost. And what if it is closer to the lower limit, 15%? Then we go from 0.8% to 0.92%.

Does this effect size matter, even if statistically significant? What if this were a randomized controlled trial for a statin? What would we say to this result? Even if this is a real signal, which is questionable given the observational design (yes, despite holding a special affection for observational studies, I don't think that this cause-effect is completely unconfounded; and this matters greatly in view of this minuscule magnitude), I am far more likely to die next time I get into my car than from eating burgers, even if I do indulge in one a couple of times per week. I am certainly not advocating eating red meat 7 days per week, though this view is driven more by practical concerns for sustainable beef farming than by the data presented in this paper.

There are a few political issues to disentangle. I despise CAFOs and their product; I despise their contempt for the animals and for the environment; and I despise their disregard for human health. I would love to see a study that shows that CAFO-raised beef kills people, as my cognitive biases tell me it must. I would love to see them all shut down, period. And this goes for the meat packing and distributing oligopoly as well. This venom notwithstanding, the current paper gives us no ammunition to this end: it failed to explore this pivotal question. Pity!

Furthermore, a study like this is likely to feed extremist marketing messages to suit someone's agenda that will likely drag us farther away from the moderation that is conducive to our health. But my local farmers should rest easy knowing that this is not by any means a game changer, that there is nothing in this paper to make us any less enthusiastic about their product, and that our New England pastures will not any time soon be devoid of the beautiful sight of these lovely ruminants. Moderation in everything including meat consumption, is probably still the best course of action. If we focus our energies on what is genuinely good for our health, we will do right by the environment.      

Three central questions about medical technologies

Every day my inbox gets filled with announcements for and invitations to attend all kinds of conferences. While many of them are of the traditional medical education sort, more and more I hear about meetings where new technologies and gadgetry in healthcare are the focus. And while the former feature healthcare professionals speaking medicalese from the stage to rapt audiences of other healthcare professionals in dimly lit halls, the latter capture their audience' imaginations with the promise of the future, in all its glitter and glory. And naturally, the latter are what attract techies and patients alike, steamrolling over our staid medieval medical conventions. I heard that the HIMSS conference in Las Vegas last month attracted 37,000 attendees! And the excitement was palpable even through Twitter feeds. This is clearly the preferred way to effect public engagement with healthcare.

But here is the thing: medical advances happen much more slowly than the speed of technology development. That is why, year after year, we go to our professional society meetings and have deja vus all over again. Year after year we hear the same people present the same studies, sometimes, if we are lucky, with a slightly different twist. The last group of breakthroughs I heard about at one of our critical care meetings was over a decade ago. And we have had to backpedal from that quite a bit with the removal of Xigris from the market, and with the realization that tight glucose control had to be used with extreme caution, so as not to kill more critically ill patients than it was meant to save.

This disconnect between the glacial pace of true progress in the clinical sciences and the lightening speed of technological progress raises some obvious questions about our assumptions. If we are not making dramatic breakthroughs in medicine every day, what is this breakneck pace of technology innovation delivering, save for the glitter and a seductive promise of health and wealth? Is there evidence supporting this promise? You might counter by saying that we have years, maybe even decades, of translational catching up to do, bringing all the advances from the bench to the bedside. I would have to say that the magnitude of such advances, as well as clinicians' resistance to them, may have been overstated. You might also point out that the vast advances in computing capabilities have not penetrated sufficiently into our healthcare system, and there I cannot disagree. But whether bringing these advances into clinic without careful planning will improve our health or our healthcare finances remains in question.

Moreover, I see numerous downsides to rushing ahead without thinking through what we are rushing toward. Ostensibly, the light at the end of this bright technological tunnel is better health. What concerns me is that the journey has become a sort of an end in itself: the sheer beauty of the tunnel has itself become the prize. To regain our compass, we need to ask three tough questions, each of them central to understanding the impending advent of too much out-of-context information:
1. Do we really want to walk around with sensors (pdf document, see page 16)? Personally, I find 24/7 monitoring of our vital functions to be a depressing prospect. Furthermore, I sincerely doubt that this is a better (or more cost-effective) way to achieve health than through focus on public health and socioeconomic equity.
2. What is the use of having your genome in your pocket?
How is that going to help us at the stage where all we can do is identify certain levels of risk (bracketed by broad intervals of uncertainty) in isolation from all the influences that modify that risk?
3. Do we want an epidemic of false positive findings and pseudo-disease?
We have enough trouble interpreting positive findings from medical screening tests. Do we really want the public falling prey to the anxiety and over-testing that false positives bring? As a healthcare system, can we sustain such an avalanche? As clinicians are we able to mange these screening snafus?
(I have done so many posts on this issue that you can barely navigate this site without stumbling over their debris).

All of the above is not sexy or shiny, and it brings in the ugly four-letter word "risk" to balance the discussion of the holy grail of benefit. So, how do we sell it to people so hypnotized by technology? I am asking this question quite seriously. I, and many of you out there, really would like people to become more cognizant of these nuances, but how do we accomplish this? Do we need to change our byzantine approach to medical meetings to start attracting a broader audience for our messages? If so, let's get started. Perhaps other forums that are already exceedingly successful can teach us how. I commend TEDMED (which I am attending as a Front-Line Scholar) for starting to bring this important viewpoint to their audiences. Now, how about having HIMSS, Health 2.0 and others join to clarify this other edge of the medical sword? Perhaps we can prevent the wild pendulum swings by thinking things through now. And think how much more credibility we all will have if, by thinking and debating now, we minimize the unintended consequences later?                            

An open letter to my past and future students

As most of my readers know, I teach Public Health graduate students and the University of Massachusetts, sometimes on campus, and sometimes online. This is an open letter to all of my past and future students. 

First, I want to thank everyone in my June course who took the time to complete the evaluation -- the feedback is very helpful to me. I also want to thank the whole class (and all my previous and future classes) for the privilege of learning together with you. Looking at this and previous rounds of evaluations made me realize that I need to make a public statement that all students contemplating taking my courses can read before they commit.

My evaluations tend to be bimodal -- a peak around "love", a (smaller, thankfully) peak around "hate", and mostly a trough in between. Oddly, the reasons for the love and the hate are the same: not too much structure, not too much interim formal performance evaluations, a lot of opportunities for discussions and questions. Everyone seems to appreciate my effort to make the material interesting and relevant, but a substantial number do not seem to like it.

I am actually fascinated by the convergence of the reasons for liking and disliking. By way of inference, I am going to suggest that where you fall out on the teaching should tell you more about your learning style than how I teach. To clarify, let me spell out my philosophy of teaching.

Whether I teach on campus or online, I limit my classes to graduate students. The reason for this is not that I do not think that undergrads cannot handle the material. Rather it is because I believe nothing replaces time with a topic to develop a depth of understanding and discussion about it. So, I view my classes as incubators of ideas. I do not see myself as the oracle delivering answers. My role is to get you excited about the questions. Furthermore, it is not my questions that should excite you, but the questions that you come to at the limits of your knowledge seen through the prism of the class material and discussions.

To be sure, I realize that this not a comfortable place for many. Most of us glide through an educational system that convinces us that there is a single correct answer, and, after teaching us to parrot it compliantly, punishes us if we stray. So, swimming in the sea of questions, seemingly answering them only to realize that the answers lead to further questions is disquieting. Yet it is at this edge that we gain access to the next level of understanding of our universe. Here, the feedback is not about an arbitrary letter, but about the exuberance of ideas, discussion generated and the richness of asking the questions.

I cannot tell you how much I love the learning environment that we create together. I gain something from each and every one of you, and I hope that each of you walks away with at least one idea that is new. What I suggest to you if you are a potential student considering taking a course with me in the future is to contemplate the boundaries of your own comfort zones in learning. If you like the feeling of vertigo that you get when old dogmatic answers are shattered and uncertainty reigns, take my classes. If you are worried about how it might feel, but curious to try, I will meet you where you are and help you weave a net to cradle your fall. But if you know that uncertainty cripples you, that you would rather have a map for every step of the way, my classes may be the wrong stop along your educational path at this time. But perhaps sometime in the future?

Again, thanks to everyone for enriching my learning. I miss you and look forward to future opportunities for exchanging ideas.                    

Eminence or evidence, or how not to look like a fool when reporting your own data

A study presented a the ICAAC meeting was reported by the Family Practice News that piqued my interest. Firstly, it is a study on C. difficile infection treatment, and secondly it is counter to the evidence that has accumulated to date. So, I read the story very carefully, as, alas, the actual study presentation does not appear to be available.

Before I launch into the deconstruction of the data, I need to state that I do have a potential conflict of interest here. I am very involved in the CDI research from the health services and epidemiology perspective. But equally importantly, I have received research and consulting funding from ViroPharma, the manufacturer of oral Vancocin that is used to treat severe CDI.

And here is an important piece of background information: the reason the study was done. The recent evidence-based guideline on CDI developed jointly by SHEA and IDSA recommends initial treatment with metronidazole in the case of an infection that does not meet severe criteria, while advocating the use of vancomycin for severe disease. We will get into the reasons for this recommendation below.  

OK, with that out of the way, let us consider the information at hand.

My first contention is that this is a great example of how NOT to conduct a study (or how not to report it , or both). The study was a retrospective chart review at a single VA hospital in Chicago. All patients admitted between 1/09 and 3/10 who had tested positive for C. difficile toxin were identified and their hospitalizations records reviewed. A total of 147 patients were thus studied, of whom 25 (17%) received vancomycin and 122 (83%) metronidazole. It is worth mentioning that of the 122 initially treated with vancomycin, 28 (23%) were switched over to metronidazole treatment. The reasons for the switch as well as their outcomes remain obscure.

The treatment groups were stratified based on disease severity. Though the abstract states that severity was judged based on "temperature, white blood cell count, serum creatinine , serum albumin, acute mental status changes, systolic blood pressure<90, requirement for pressors," the thresholds for most of these variables are not stated. One can only assume that this stratification was done consistently and comported with the guideline.

Here is how the severity played out:

Nowhere can I find where those patients who were switched from metronidazole to vancomycin fell in these categories. And this is obviously important.

Now, for the outcomes. Those assessed were "need for colonoscopy, presence of pseudomembranes, adynamic ileus, recurrence within 30 days , reinfection > 30 days post therapy, number of recurrences >1, shock, megacolon, colon perforation, emergent colectomy, death." But what was reported? The only outcome to be reported in detail is recurrence in 30 days. And here is how it looks:

The other outcomes are reported merely as "M was equivalent to V irrespective of severity of illness (p=0.14). There was no difference in rate of recurrence (p= 0.41) nor in rate of complications between the groups (p=0.77)."
What the heck does this mean? Is the implication that the p-value tells the whole story? This is absurd! In addition, it does not appear to me from the abstract or the FPC report as if the authors bothered to do any adjusting for potential confounders. Granted, their minuscule sample size did not leave much room for that, but a lack of attempt alone invalidates the conclusion.

Oh, but if this were only the biggest of the problems! I'll start with what I think is the least of the threats to validity and work my way to the top of that heap, skipping much in the middle, as I do not have the time and the information available is full of holes. First, in any observational study of treatment there is a very strong possibility of confounding by indication. I have talked about this phenomenon previously here. I think of it as a clinician noticing something about the patient's severity of illness that does not manifest as a clear physiologic or laboratory sign, yet is very much present. A patient with this characteristic, although looking to us on paper much like one without a disease that is that severe, will be treated as someone at a higher threat level. In this case it may translate into treatment with vancomycin of patients who do not meet our criteria for severe disease, but nevertheless are severely ill. If present, this type of confounding blunts the observed differences between groups.

The lack of adjustment for potential confounding of any sort is a huge issue that negates any possibility of drawing a valid conclusion. Simply comparing groups based on severity of CDI does not eliminate the need to compare based on other factors that may be related to both the exposure and the outcome. This is pretty elementary. But again, this is minor compared to the fatal flaw.

And here it is, the final nail in the coffin of this study for me: sample size and superiority design. Firstly, the abstract and the write-up say nothing of what the study was powered to show. At least if this information had been available, we could make slightly more sense out of the p-values presented. But, no, this is nowhere to be found. As we all know, finding statistical significance is dependent on the effect size and variation within the population: the smaller the effect size and the greater the variation, the more subjects are needed to show a meaningful difference. Note, I said meaningful, NOT significant, and this they likewise neglect. What would be a clinically meaningful difference in the outcome(s)? Could 11% difference in recurrence rates be clinically important? I think so. But it is not statistically significant, you say! Bah-humbug, I say, go back and read all about the bunk that p-values represent!

One final issue, and this is that a superiority study is the wrong design here, in the absence of a placebo arm. In fact, the appropriate design is a non-inferiority study, with a very explicit development of valid non-inferiority margins that have to be met. It is true that a non-inferiority study may signal a superior result, but only if it is properly designed and executed, which this is not.

So, am I surprised that the study found "no differences" as supported by the p-values between the two treatments? Absolutely not. The sample size, the design and other issues touched on above preclude any meaningful conclusions being made. Yet this does not seem to stop the authors from doing exactly that, and the press from parroting them. Here is what the lead author states with aplomb:

              "There is a need for a prospective, head-to-head trial of these two medications, but I’m not sure who’s going to fund that study," Dr. Saleheen said in an interview at the meeting, which was sponsored by the American Society for Microbiology. "There is a paucity of data on this topic so it’s hard to say which antibiotic is better. We’re not jumping to any conclusions. There is no fixed management. We have to individualize each patient and treat accordingly."

OK, so I cannot disagree with the individualized treatment recommendation. But do we really need a "prospective head-to-head trial of these two medications"? I would say "yes," if there were not already not 1 but 2 randomized controlled trials addressing this very question. One by Zar and colleagues and another done as a regulatory study of the failed Genzyme drug tolevamer. Both of the trials contained separate arms for metronidazole and vancomycin (the Genzyme trial also had a tolevamer arm), and both stratified by disease severity. Zar and colleagues reported that in the severe CDI group the rate of clinical response was 76% in the metronidazole-treated patients versus 97% in the vancomycin group, with the p=0.02. In the tolevamer trial, presented as a poster at the 2007 ICAAC, there was an 85% clinical response rate to vancomycin and 65% to metronidazole (p=0.04).

We can always desire a better trial with better designs and different outcomes, but at some point practical considerations have to enter the equation. These are painstakingly performed studies that show a fairly convincing and consistent result. So, to put the current deeply flawed study against these findings is foolish, which is why I suspect the investigators failed to mention anything about these RCTs.

Why do I seem so incensed by this report? I am really getting impatient with both scientists and reporters for willfully misrepresenting the strength and validity of data. This makes everyone look like idiots, but more importantly such detritus clogs the gears of real science and clinical decision-making.

Quality measures: Process, outcome, or both?

In the last week I wrote about our quality improvement, or QI, efforts in healthcare. And although there is a burgeoning field representing itself as the "science" of QI, I question much of its scientific validity. As always, VAP is my poster child for these discussions, where neither the definition of the condition itself nor its prevention efforts are subject to much scientific scrutiny. This makes VAP have a surreal, ghost-like quality: now you see it, now you don't. And this alone makes it difficult to assess prevention efforts. Much as in the heated mammography debate, where passionate anecdote prevails, the sanctity of the QI rubric blunts the usual critical approach to the data.

So, the central point that I made in this post was essentially to devalue the VAP eradication efforts as not grounded in solid scientific evidence. What has occurred to me, however, is that this position may be in fact at odds with a realization I blogged about here and here, wherein I agreed with Dan Arieli's suggestion that outcomes in the real world, where they are influenced by so much randomness, are not the thing to reward. It would be much more rational to reward best efforts at best results, thus the process rather than the outcome. So, here is the apparent contradiction: On the one hand I agree that outcomes may be too unpredictable, being that they are influenced by too many factors that are not in our control, yet I am also advocating that we start measuring such outcomes as antibiotic use associated with VAP and its reduction. What gives?

Well, on the one hand, I am OK with contradiction; life is full of instances where we have to hold conflicting information and feelings together. But as a scientist it is my predisposition to analyze (which literally means splitting into smaller, more manageable chunks), so I have given this ostensible paradox more thought. What I came up with is that measuring process is the right thing to do, but only under very specific conditions. Avedis Donabedian, who is considered the father of quality science, introduced the triad of structure-process-outcome as the backbone of quality science. This relationship certainly lends validity to the "process" metrics as surrogates for "outcome." But the condition that has to be met is that there be an actual correlation between the process and the said outcome. If there is no such solid correlation, then we are simply going through the motions, doing a rain dance to cause rain.

So, what I have said about VAP prevention in particular is that we are nowhere near being able to say that the recommended processes correlate with any changes in meaningful clinical outcomes. And because the data on these interventions are so weak, throwing massive resources behind implementing them is irrational and resembles religious fervor more than scientific pragmatism.

It is entirely understandable that we would jump on this bandwagon so rapidly, given the magnitude of harm in our healthcare system combined with the need to reign in the healthcare spending. But there is a more subtle point to be made here too. It relates to the fertile soil of our American psyche, where doing something is always perceived as better than thinking about our course of action, which is frequently referred to with contempt as "doing nothing." In the end, this crisis response mentality is good in a crisis, but potentially detrimental in the long term: we are unlikely to be altering meaningful outcomes, and we are spending billions of dollars on interventions lacking evidence.

So, I stand behind both of my assertions and maintain that they are not mutually exclusive. Yes, outcomes are subject to much randomness; yes, processes known to alter these outcomes are the sensible measures of our efforts to improve quality; and yes, these processes need first to be rigorously validated for their impact on the outcomes in question. Anything short of this pathway is not just a waste of our collective resources, but a manipulation of the public trust. And that is as far from the intent of science as it can get.

Health surveys: Run the other way!!!

Sometimes when I get an unsolicited call about answering survey questions, I feel a karmic obligation to participate; after all, if everyone said no to everything, I would not have any data to analyze. So, for this very reason, I just got off the phone with a poor young woman conducting a survey who called me randomly. The survey had to do with healthcare delivery, and she had no idea what she was getting herself into. First, I queried her whom the survey was for. She proceeded to tell me that she did not have that information specifically, but gave me a general idea of who the customers tend to be. Then she launched into the survey questions.

Now, I realize that they all have to ask the same questions the same way in order not to bias the data. But man, who writes these questions? "What would you say is the reputation of the cardiac surgery program at thus-and-such a hospital in your area: a). good locally, 2). good locally and state-wide, 3). good locally, statewide and regionally, 4). good locally, statewide, regionally and nationally, 5). good locally, statewide, regionally, nationally and internationally, or 6). not good at all?" Well, what the heck do you mean by "reputation"? You mean what is the gossip about Dr. Smith in my community? Or do you mean what kind of care they provide in terms of timeliness, evidence, shared decision making, post-operative complications, what? Then came "if you or your family member needed a cardiac procedure, how comfortable would you be going to this facility? 1). very comfortable, 2). somewhat comfortable, 3). somewhat uncomfortable, and 4). not at all comfortable?" How the heck should I know? I have not researched all the local facilities, I have not checked on their outcomes, I have not interviewed all of their cardiac surgical teams (yes, including anesthesia), I do not know what their infection control track records are, and, most importantly, how willing they are to treat me as an individual rather than a source of income. And then, for every hospital she mentioned (and there were quite a few), she went through the same litany of meaningless questions.

And then she asked me if I am familiar with some of the well-known quality-rating organizations. And she included US News and World Report Hospital Ratings! And I don't even believe the CMS got it anywhere near right!!! Oy! What do the answers to these questions from someone who is not steeped in the data mean anyway? If researchers and providers have not arrived at the appropriate metrics for quality, how meaningful are the lay public's opinions on these matters?

And finally, a group of questions that let the cat out of the bag as to the purpose of the survey. She told me a story first, of a large regional medical center in the area building a new multi-million dollar state-of-the-art cardiac care facility. Sexy new equipment, individual patient rooms, targeted and individually-tailored treatment plans, all the buzzwords of the brave new world of medicine. And then she asks me would I be comfortable going to this facility. What am I supposed to say? I have no idea! How do I tell this poor child that the questions are written in an absurd way and smack of marketing? How do I explain to her that this facility will probably need to recoup their capital investment, and, therefore, has a conflict of interest when it comes to caring for me? How do I teach her that this is the problem with American medicine, this very over-reliance on reputations and expertise to tell us to over-indulge in interventionism at the expense of our health and budgets?

Anyway, I will not belabor this further. My advice to survey fielders: If you want to market to the gullible, go ahead and call people randomly and ask your market-building question. And if a person tells you she is a physician and a health services researcher to boot, run, don't walk, the other way.

The beautiful uncertainty of science

I am so tired of this all-or-nothing discussion about science! On the one hand there is a chorus singing praises to science and calling people who are skeptical of certain ideas unscientific idiots. On the other, with equal penchant for eminence-based thinking, are the masses convinced of conspiracies and nefarious motives of science and its perpetrators. And neither will stop and listen to the other side's objections, and neither will stop the name-calling. So, is it any wonder we are not getting any closer to the common ground? And if you are not a believer in the common ground, let me say that we are only getting farther away from the truth, if such a thing exists, by retreating further into our cognitive corners. These corners are comfortable places, with our comrades-in-arms sharing our, shall we say, passionate opinions. Yet this is not the way to get to a better understanding.

Because I spend so much time contemplating our larger understanding of science, the title "Are We Hard-Wired to Doubt Science" proved to be a really inflammatory way to suck me into thinking about everything I am interested in integrating: scientific method, science literacy and communication and brain science. The author, on the heels of doing a story on the opposition to smart meters in California, was led to try to understand why we are so quick to reject science:

But some very intelligent people I interviewed had little use for the existing (if sparse) science. How, in a rational society, does one understand those who reject science, a common touchstone of what is real and verifiable?

The absence of scientific evidence doesn’t dissuade those who believe childhood vaccines are linked to autism, or those who believe their headaches, dizziness and other symptoms are caused by cellphones and smart meters. And the presence of large amounts of scientific evidence doesn’t convince those who reject the idea that human activities are disrupting the climate.

She goes on to think about the different ways of perceiving risk, and how our brains play tricks on us by perpetuating our many cognitive biases. In essence, new data are unable to sway our opinion because of rescue bias, or our drive to preserve what we think we know to be true and to reject what our intuition tells us is false. If we follow this argument to its logical conclusion, it means that we just need to throw our hands up in the air and accept the status quo, whatever it is.

I happen to think that the author missed an opportunity to educate her readers about why we need to come to a better understanding and how to get there. The public (and even some of my fellow scientists) needs to understand what science is and, even more importantly, what it is not.

First, science is not dogma. Karl Popper had a very simple litmus test for scientific thinking: He asked how you would go about disproving a particular idea. If you think that the idea is above being disproved, then you are engaging in dogma and not science. The essence of scientific method is developing an hypothesis from either a systematically observed pattern or from a theoretical model. The hypothesis is necessarily formulated as the null, making the assumption of no association the departure point for proving the contrary. So, to "prove" that the association is present you need to rule out any other potential explanation for what may appear to be an association. For example, if thunder were always followed by rain, it might be easy to engage in the "post hoc ergo propter hoc" fallacy and conclude that thunder caused rain. But before this could become a scientific theory, you would have to show that there was no other explanation that would disprove this association.

So, the second point is that science is driven by postulating and then disproving the null hypotheses. By definition, an hypothesis can only be disproved if we 1). the association exists, and 2). the constellation of phenomena is not explained by something else. And here is the third and critical point, the point that produces equal parts frustration and inspiration to learn more: That "something else" as the explanation of a certain association is by definition informed only by what we know today. It is this very quality of knowledge production, the constancy of the pursuit, that lends the only certain property to science, the property of uncertainty. And our brains have a hard time holding and living with this uncertainty.

The tension between uncertainty and the need to make public policy has taken on a political life of its own. What started out as a modest storm of subversion of science by politics in the tobacco debate, has now escalated into a cyclone of everyday leveraging of the scientific uncertainties for political and economic gains. After all, how can we balance the accounting between the theoretical models predicting climate doom in the future and the robust current-day economic gains produced by the very pollution that feeds these models? How can we even conceive that our food production system, yielding more abundant and cheaper food than ever before, is driving the epidemic of obesity and the catastrophe of antimicrobial resistance? And because we are talking about science, and because, as that populist philosopher Yogi Berra famously quipped, "Predictions are hard, especially about the future," the uncertainty of our estimates overshadows the probability of their correctness. Yet by the time the future becomes present, we will be faced with potentially insurmountable challenges of a new world.

I have heard some scientists express reluctance about "coming clean" to the public about just how uncertain our knowledge is. Nonsense! What we need under the circumstances is greater transparency, public literacy and engagement. Science is not something that happens in the bastions of higher education or behind the thick walls of corporations. Science is all around and within us. And if you believe in God, you have to believe that God is a scientist, a tinkerer, always looking for a more elegant solution. The language of science that may seem daunting and obfuscatory. Yet do not be afraid -- patterns of a language are easy to decipher with some willingness and a dictionary. Our brains are attuned to the most beautiful explanations of the universe. Science is what provides them.

Self-determination is predicated upon knowledge and understanding. Abdicating our ability to understand the scientific method leaves us subject to political demagoguery. Don't be a puppet. We are all born scientists. Embrace your curiosity, tune out the noise of those at the margins who are not willing to engage in a sensible dialogue, leave them to their schoolyard brawling. And likewise, leave the politicians, corporate interests, and, alas, many a journalist, and start learning the basics of scientific philosophy and thought. Allow the uncertainty of knowledge excite and delight you. You will not be disappointed.                   
              

Serendipity

I am always amazed how serendipitous life can be. Oddly I find that when I encounter a new word or concept, it all of a sudden begins to appear wherever I go. Or is it that when you open yourself to the Universe, it comes to your doorstep?  Cognitive bias, anyone? Imagine my surprise when after the exchanges of last week I came upon these three ideas in the current issue of Scientific American.

Consider this headline:

Climate Heretic: Judith Curry Turns on Her Colleagues

Why can't we have a civil conversation about climate?

And goes on to say in the fist paragraph:

For most of her career, Curry, who heads the School of Earth and Atmospheric Sciences at the Georgia Institute of Technology, has been known for her work on hurricanes, Arctic ice dynamics and other climate-related topics. But over the past year or so she has become better known for something that annoys, even infuriates, many of her scientific colleagues. Curry has been engaging actively with the climate change skeptic community, largely by participating on outsider blogs such as Climate Audit, the Air Vent and the Black­board. Along the way, she has come to question how climatologists react to those who question the science, no matter how well established it is. 

Now, I certainly do not claim to be the caliber of Judith Curry as a scientist, nor do I claim that the current conversation is nearly as crucial as climate change. Yet, some of the messages are instructive, not the least of which is the clarification of some of the subtle uncertainties in climate science.

I wish I had Scientific American's penchant for creating titillating headlines. How does this one grab you?

Hawking versus God: What Did the Physicist Really Say about the Deity?

The battle for eternity is fought on Larry King Live

And while I am profoundly interested in Hawking's ideas, here is what really spoke to me:

“The theories that Hawking and Mlodinow use to base their arguments on have as much empirical evidence as God,” wrote cosmologist Marcelo Gleiser on an NPR.org blog. Moreover, Gleiser added, “because we don’t have instruments capable of measuring all of nature, we cannot ever be certain that we have a final theory.”

This bears repeating: "Because we don't have instruments capable of measuring all of nature, we cannot ever be certain that we have a final theory." Isn't this beautiful? I love science!

And last, but not least, Christopher Hitchens provides this inimitably lucid observation:

“What can be asserted without evidence can also be dismissed without evidence.”

What a wonderful contradiction to hold, realizing that science is never complete, yet we must always go to the next level to gather the evidence. Science is about questions and discussion. Go forth, question and discuss.  

     

Microbiome: passenger or vehicle?

Remember how you learned in BIO 101 that many fluids in the human body are normally sterile? And you know how you laugh with condescension when someone dares to question this dogma? After all, we know that the gut and the skin are full of microorganisms, but the lung? The bladder? This is inconceivable, heretical!

Well, actually, this, like many other dogmas that seemed like God's word at the time, has been broken open by some very interesting research emerging from multiple laboratories around the world. I was first stopped in my tracks about a year ago, while listening to an NPR story in my car and learning that the number of bugs we carried on and in us was about 10-times greater than the number of our own cells. Then, in a farm house in Cornwall a couple of weeks ago, while on vacation, I read an article in an October 2009 issue of Eureka, The Times monthly science magazine, which once again made me ponder our sojourns in life, the bacteria that cohabitate with us. And finally, today I saw this story in yesterday's NYT, which galvanized me into writing action. This grabbed me in particular:

Scientists are even discovering ecosystems in our bodies where they weren’t supposed to exist. Lungs have traditionally been considered to be sterile because microbiologists have never been able to rear microbes from them. A team of scientists at Imperial College London recently went hunting for DNA instead. Analyzing lung samples from healthy volunteers, they discovered 128 species of bacteria. Every square centimeter of our lungs is home to 2,000 microbes.

It grabbed me for two reasons:
1). I am a lung doctor, and a lot of the research that I do centers around lung infections. One type of this infection, ventilator-associated pneumonia or VAP, is the subject of much scrutiny by policy makers and payers, and is emerging as a quality indicator. There are many challenging issue in VAP, not the least of which is its treatment: because it is likely to be caused by an antibiotic resistant organism, and because at least one-half of all cases of suspected VAP never grows out a specific microbe, the recommended empiric treatment is with broad-spectrum drugs. The clinician treads a fine line between undertreating the patient at the expense of increased mortality or overtreating and promoting antibiotic resistance.
2). I have already written here about my thoughts on evidence and how we tend to approach our knowledge with the arrogance that blinds us to the possibility of being either under-informed or plain wrong. Well, here is the case where we were so convinced of what we knew, yet it was plain wrong! With the scientific skepticism and appropriate tools we are learning something completely opposite to what we thought we knew.

Taking this line of thinking one step further, can we now conceive of the possibility that what we expose our bodies to actually plays a role in our long-term health? To be specific, I mean something as simple as food and the way our food is produced. Monoculture and food engineering have become staples of our food production. Animals are raised in CAFOs in conditions requiring them to be on antibiotics to ward off infections and to help with growth. Additionally, to leverage the subsidized and abundant corn supply, food animals are forced into diets completely unnatural to them. Cows, for example, evolved over millennia to be ruminants meant to eat grass and hay, are fed a corn-based grain diet, which, by altering their gut environment, makes them much more likely to carry toxigenic E. coli. Although there is much nay-saying about whether antibiotics (and hormones, by the way), are passed on to humans through meat or milk, I have never seen any convincing scientific evidence to the contrary. So, if we eat meat that is microbiologically altered, and may also be exposed to antimicrobial residues that may in turn change our own native microbiome, is it possible that at least some of the emerging health issues are due to the travesty of monoculture production?

I am not saying that I have all the answers -- clearly, this research has a long way to go to establish causal pathways and potential points of intervention. But let's exercise our common sense and at least ask the questions. Do it early and often. And let's please agree that there is a certain burden of proof here that resides with the food production oligopoly. Oh, yeah, and let's not forget to continue limiting the use of antibiotics in human health too. And have a little bit of dirt... for lunch... every day...  

Evidence: What the bleep do we really know?

So I know that my blogging has fallen off a bit, and I am sure you are all sorely disappointed (or not). Let me try to explain why.

There are several things going on to nudge me toward the next rung in the evolution of my thinking about healthcare and evidence. The event most responsible for this re-examination of my assumptions is the previously-mentioned illness of a loved one, along with my need to be closely involved with his care decisions. I think that I have generally adequately voiced my frustration with how decisions are made in our healthcare system, and where the switches for these choices should be as opposed to where they are. I am even more convinced now that by the time the physician and the family are considering expensive options with marginal effectiveness, the cat is already out of the bag: how can either the distraught family or the committed healthcare provider not consider those, despite shaky evidence of the value?

But this is not all. I have actually been cogitating the entire way we do evidence and evidence-based medicine. When we invoke evidence, we generally talk about some scientific study's findings, or a group of studies either showing similar or differing results. Let's keep stepping back and looking at the components. The scientific community, based on some statistical and other methodological considerations has come to a consensus around what constitutes valid study designs. This consensus is based on a profound understanding of the tools available to us to answer the questions at hand. The key concept here is that of "available tools". As new tools become available, we introduce them into our research armamentarium to go deeper and further. What we need to appreciate, however, is that "deeper" and "further" are directional words: they imply the same direction as before, only beyond the current stopping point. This is a natural way for us to think, since even our tools are built on the foundation of what has been used previously.

So, what emerges is a picture of being emotionally committed to a certain way of viewing the question, the processes of answering it and the actual answers. And what if by narrowing ourselves to this one particular way of examining the world, to this one particular way to collect and interpret the evidence, we have effectively ignored all other possibilities?

Let me try to clarify what I mean a little further. We spend our days walking through waves. Some of these waves we can detect through our own senses: sound, light, for example. Others we need special external "receptors" to detect, such as radio and micro-waves. Is it possible that there are some other, heretofore unknown waves (or other phenomena) that are around us that we are unable to attune to because of our biology? Is it kooky to think that this is possible, or is it simply blind to walk away from such possibility?

The answer may be that it is both. Nevertheless, it is highly likely that there are many biological phenomena that are not amenable to being examined through our narrow prism of current methodology. We as scientists and clinicians need to be open to this possibility. The Buddha said that both preachers and scholars are blind because they constantly commit themselves to partially-seen truths. Given the shambles in which we find not only our nation's healthcare system, but also the health of its citizens, this would be a great time for this epiphany to penetrate our collective psyche and strive toward a broader view of possibilities. Let's give up the arrogance of ignorance in favor of the humbleness of enlightenment.

And these are my latest thoughts. I am not saying that they are the only way or that they are even remotely correct. But here they are nevertheless. Do with them what you will.            

Tinkering and innovation: You can't have one without the other

Last week I blogged about the empty black box of evidence in medicine. Afterwards I got to thinking about black boxes in other parts of our lives in the 21st century -- seems to me they are everywhere and getting more and more prevalent.

To appreciate their proliferation, we must travel back in time to around the middle of the 19th century. I know, it is a bit anxiety-provoking, what no blogging, no tweeting, no iPhones... And yet, let's persist. The landscape is still dominated by small rural communities with farming as the major industry. Around these farming villages some small local businesses are making a successful run: furniture makers, dress makers, healthcare providers. Schools are also local and classrooms are multi-age and co-ed. School day follows the agrarian cycle, allowing the kids to maximize their farm time in every season. Most goods and services are created locally, if not in the home itself. When a shovel breaks, it is fixed. When socks rip, they are darned. When a child is having trouble with homework, she is helped by her parents and siblings. Evening entertainment is provided in the home by family members' stories and the stars in the sky. OK so far?

In comes industrialization. Furniture and dress making are automated, farmers begin to move into cities and schools become more regionalized and no longer need to observe the constraints of the agrarian lifestyle. The second half of the 20th century ushers in unprecedented advances in technology. Instead of fixing our broken shovels, we chuck them and buy new ones. Ditto for socks, cars, computers, couches. We used to be able to look under the hood of our 1967 Dodge Dart and fix the strange rattling noise. We used to bring our shoes to a shoe repair shop to re-sole when necessary. Today we discard the old and buy the new model.

I know what you are thinking: there she goes again railing against consumerism! Well, yes, but that is not where I am going with this. A byproduct of our rampant consumerism is the loss of the art of tinkering. Tinkering is our legacy. What were Thomas Edison and Ben Franklin if not tinkerers? How about Steve Jobs and Michael Dell? How many of us can build or fix a transistor radio? What about next generation life-altering technologies we cannot even conceive of yet?

Along with this loss of tinkering came the ceding of expertise. What I mean is that, since everything today seems infinitely more complicated, the public defers to the expert class, built around the black boxes clearly inaccessible to mere mortals. And guess what is the biggest black box... If you guessed our educational system itself, you are correct. Reading, math and social studies have been taught in the home, then in small school houses, for centuries. (And if you think that farmers did not learn math, think again: you cannot get around a farm without being facile with things mathematical.) Parents used to know how to help their children learn. Now, we defer our children's education to "experts" behind closed doors, who are likely to recommend pharmacologic solutions where individualized approach is needed. And what has this gigantic black box yielded? If you read award-winning secondary school teacher John Taylor Gatto's angry diatribe against our educations system, "Weapons of Mass Instruction", aside from a big headache, you will walk away with the understanding that we are now, as a nation, more cynical and dumber than we were 80, 50, and even 30 years ago. Thanks to experts.

So, we in effect have given away transparency and control over most of our daily lives. Distant industrial farms provide what passes for our food, distant manufacturing plants in China deliver our clothes, furniture, computers. Locked down institutions beyond our inquiry and constructive criticism have a hold on our children for 8 hours each day, and wish to prolong this hold by contracting vacation time to do what? Produce even more pervasive boredom, cynicism and ignorance? Even our entertainment has passed into the black box realm: distant producers selling their advertisers' wares by creating inane and irrelevant "entertainment" for the masses.

People, we are no better informed than our ancestors banging their drums to ward off solar eclipse. Our fancy gadgets separating us from what's under the hood of reality give us a false sense of complacency. If we want true innovation, we need to get back to our tinkering roots. Learn to darn your socks, help your child to read and teach her to tinker, so that she can stay curious. Question "experts": most of the time the mountains of complexity behind their concepts are useless or unnecessary, or created for the purpose of exclusion by obfuscation. Throw open these black boxes and shine a light in them. Play with stuff. Play with ideas. Tinker!              

Why medicine is not like a parachute

At the recent meeting of the American College of Chest Physicians I went to a session on hospital rapid response teams. This was a talk given by a senior and respected member of the Pulmonary and Critical Care academic community, well published and known. Now, if you are wondering what the heck is a rapid response team, I will tell you: it is a team composed variously of an ICU physician (or a hospitalist, depending on local patterns), a nurse or two, a pharmacist and sometimes other ancillary personnel. The role of this team is to be at the beck and call of the ward nursing staff to evaluate urgently any floor patient whose clinical condition may appear to be deteriorating. The ultimate goal, of course, is to avert a catasprophic emergency called a code, where the patient deteriorates to the point of needing emergent resuscitation.

While such a team makes a lot of sense, no study has convincingly shown that it does what it is meant to do -- either prevent codes or save lives. The presenter, a man quite invested in the sensibility of such teams, was making the point that perhaps we just have not focused on the correct outcomes to measure for this intervention. Ultimately, he said, we should continue to support such teams because of our gestalt that they work. After all, he went on, nobody has tested the effectiveness of the parachute in a randomized controlled trial.

When I heard this, I realized that I had been hearing this comparison a lot lately, of medical interventions to parachutes. And since his statement seemed somehow wrong to me, I started to think about why such an analogy is completely fallacious. And here is the startlingly obvious answer: the number needed to treat (NNT) for a parachute is 1! What this means is that one only needs to "treat" one person with a parachute (or strap it on before jumping out of a plane) to save one life. Now, is there really anything that we do in medicine whose margin of effectiveness comes even close? Last week we argued about a screening test with the NNT of 2,000. And we also talked about a therapy with a NNT of 16 which is not being utilized because of high costs.

So, let's not be intellectually lazy and let's not resort to comparing anything we do in medicine to a parachute. The parachute is made to mitigate the laws of physics, which are inherently more predictable, stable and generalizable than the laws of biology. Instituting a rapid response team may feel good to us, but, in addition to lending no measurable improvements to patient care, it may have the "unintended consequences" of abandoning patients who really need our attention at that moment to better their outcomes.

A similar argument can be made about many interventions whose mere face validity we take as God's own word. We and others have cautioned against such gullibility, particularly in a world that runs with half-truths and misinformation under the banner of quality measures. None of us has the cause to be so self-congratulatory as to compare our wares to the parachute. Now more than ever, experts need to think and present critically, soberly and objectively, lest we continue the trend of parochialism in our out-of-control disease care system.    

The lost art of evidence-based debate

When people first find out that I was not born in the US (that's right, there go my presidential aspirations!), their question is about culture shock. They want to know what I perceived to be the biggest differences in cultures in the old vs. the new country. Over the nearly 35 years that I have spent in the US my thinking on this has evolved. Initially I was flabbergasted that teachers in high school tolerated students' putting their feet up on the desks. In my old country such behavior would have been punished corporally.

Today I consider myself to be a more sophisticated judge of cultural differences. I have come to appreciate that the US is not a monoculture, but a conglomeration of many different philosophies. And I do not mean just based on race, ethnicity and other obvious cultural drivers. Take for example politics as a profession: its culture is quite distinct from many others. Social scientists tell us that successful politicians are necessarily narcissistic, which is a blessing and a curse. This makes them not only über confident and decisive, but also prone to philandering and other forms of deception. I am also struck by how well they have perfected the art of not responding to questions by taking every opportunity to drive their talking points. Particularly astonishing is their ability, derived from Karl Rove's playbook, to turn a complete fabrication into what is accepted as reality by some. Just look at the death panels rhetoric.

I have to say that over the years I have cynically come to expect this and other bad behaviors from politicians. What is concerning to me is the pervasiveness of the same Rovian tactics in other sectors. Remember the recent E. coli outbreaks linked variously to Mexican peppers, tomatoes and spinach? Well, keeping this in mind, let's consider the work by the Center for Science in the Public Interest on FDA-regulated foods that have been implicated in E. coli outbreaks. This work, citing such seemingly innocuous and even desirable foods as lettuce, tuna and potatoes, has apparently incensed some of the corporate food lobby. Indeed, the Chicago Tribune reports that

...the Produce Marketing Association disagrees with putting them on the list, saying it is committed to providing safe produce and is working with the FDA and CDC to create "food safety solutions to help safeguard public health" and that "fruits and vegetables are necessary for better health."

All points well taken. But how does this address the issue at hand? Take back your data because we are committed and working on doing better? Why this is like a doctor saying "take this malpractice suit off my record because I am committed and working on doing better". Absurd logic? I think so.

The article further cites objections from the tuna organizations thusly:

But the National Fisheries Institute was even more scathing the study (sic) in their statement, saying "Seafood is a safe and healthy product that is an essential part of the American diet. CSPI has a history of attempting to scare consumers by playing fast and loose with definitions that might lead the casual reader to think people are getting sick left and right from seafood...Consumers should take this report with a grain of salt and a heaping helping of perspective."

Scare? Fast and loose? Have they ever heard of scombroid? No, it is not fatal, just unpleasant, but it happens despite the FDA regulation to gut and stuff the fish with ice immediately upon catching. Instead of addressing the evidence, they are quick to deny the allegation by casting doubt on the source.

And this brings me to my central point. Do these tactics of killing the messenger originate in relentless pursuit of market share even at the expense of public's health? Or is this just ignorance of how to debate a legitimate point? Is it possible that these executives and lobbyists do not have the skills to structure a logical evidence-based argument? Perhaps our educational institutions have lost sight of this critical skill as well. My guess is that it is a little bit of the latter and a lot of the former. Seems like public debate continues to deteriorate into voyeuristic entertainment, aided by shallow reporting.

But it does not have to be this way. The public needs to be empowered to make their own decisions based on a critical look at the evidence presented by both sides. If we can vet this evidence ourselves, there will be no room for corporate deception. Their antics will take their rightful place in the comics section of the paper.