Tipping a sacred cow: QI under the microscope

So much media and journal space has been devoted to financial conflicts of interest, particularly within and related to pharma and device manufacturers, that to write any more about it may be redundant. On this site we have also intermittently addressed COI from other perspectives, such as financial interest of the members of the American College of Radiology in maintaining mammography screening status quo, thinly veiled in its own version of the pernicious "death panel" language. We have also spoken a bit about the non-financial COI. And even though we are so very much aware of COI's potential to lurk around every corner, there are still some surprises.

Take the sacred cow of "quality improvement" in healthcare. Even the name, much like the "pro life" moniker, suggests that it is untouchable in its purity and nobility of purpose. So necessary is it because of the epic magnitude of morbidity and mortality attributed to healthcare itself, that the billions of dollars spent on it seem unquestionably justified. Indeed, much like our public education system, the QI movement garners higher and higher allocations simply due to the sheer face validity of the assumption that more of it is better. And the most fascinating aspect is that, in our current zeal for sensible economic allocation through evidence, QI, much like education, appears immune to scrutiny. This is the very definition of politics driving policy.

I return to the case of ventilator-associated pneumonia, or VAP, as the poster child for this movement. I have already alluded to the fact that definitionally VAP is a slippery slope: its diagnosis varies based not only on the tools used to diagnose it, but also depending on who is doing the diagnosing. Yes, indeed, what one clinician calls VAP another may call absence of VAP. I have also dissected the weak evidence behind some of the strongest purportedly evidence-based recommendations aimed at VAP prevention. But what if VAP itself is the wrong endpoint? What if we are spending untold dollars and other resources on a futile pursuit?

Do you feel yourself bristling yet? If you said "yes", it is a normal response I get from my colleagues and people who read my scholarly papers. Because how can anyone be against QI? Well, I am not against QI. I am simply against sanctifying QI as a sacred cow and thus shielding it from a sensible and rational evaluation.

So, if you are over the initial shock, allow me to explain myself. I am sure you have heard of surrogate endpoints. Here is a definition from Wikipedia:

In clinical trials, a surrogate endpoint (or marker) is a measure of effect of a certain treatment that may correlate with a real clinical endpoint but doesn't necessarily have a guaranteed relationship. The National Institutes of Health (USA) defines surrogate endpoint as "a biomarker intended to substitute for a clinical endpoint".^[1][2]

Surrogate markers are used when the primary endpoint is undesired (e.g., death), or when the number of events is very small, thus making it impractical to conduct a clinical trial to gather a statistically significant number of endpoints. The FDA and other regulatory agencies will often accept evidence from clinical trials that show a direct clinical benefit to surrogate markers. ^[3]

This begs the question of what constitutes a "real" clinical endpoint. Well, in my simplemindedness I think of them as endpoints that matter to the patient or in the long run. So, death, disability, quality of life, functionality, these are the real endpoints. Something that alters one's life or threatens it is a real endpoint. Thus, blood pressure and cholesterol are surrogate endpoints, since they usually, but not always, correlate with the risk of a myocardial infarction or death. But what if such a correlation did not exist? Furthermore, what if a cholesterol level was measured with, say, tea leaves, and therefore was subject to a tremendous variation in detection? Would we then spend hundreds of billions of dollars on trying to alter this factor or would we calmly and rationally walk away and look for something that truly impacts the real outcome of a heart attack or death? I think I am making my point fairly clearly.


Let me explain why I think that VAP is but a surrogate outcome, and, given its diagnostic challenges, not a sensible one in the least. VAP by definition occurs in patients on mechanical ventilation (breathing machine), whose quality of life is fairly badly damaged in the short term. The literature would suggest that not all VAP impacts mortality adversely, but some forms of VAP indeed do, particularly VAP that develops late in the course of illness. So in this VAP does correlate with a real endpoint. Also, there is very little doubt that getting VAP prolongs one's dependence on mechanical ventilation, and increases the duration of the stay in the ICU and hospital overall. So, this can be considered not a very good, albeit real, outcome. An additional point to remember is that VAP engenders the use of additional, usually broad spectrum, antibiotics, putting both the individual and the society at risk for such unwanted consequences as the emergence of highly resistant microorganisms.


So, even though VAP is a surrogate endpoint, it certainly seems to fit the bill for something we would want to prevent. But here is the monkey wrench in this argument: what seem to be great surrogate endpoints do not always end up correlating with clinical reality. The association of VAP with morbidity and mortality has been detected in mostly retrospective observational studies. Trials of VAP prevention rarely, if ever, report any endpoint other than VAP. And, given how elusive VAP diagnosis is, there is plenty of room for gamesmanship so pervasive in the real world to make any data fit our preconceived hypotheses and political needs. 


So, what is my point? My point is that if QI wants to be a science, it needs to be subject to the same rules that all other science is guided by. Since we do not even know how much money we are spending on the ubiquitous QI efforts (likely hundreds of billions), and since we are not sure what they are accomplishing (see my many prior posts on the lack of validity of current claims in VAP prevention), we need to pause and ask ourselves whether the cheering alone justifies such an investment. I hate to say it, but can we really trust those with most to lose, financially and politically, if in reality QI does little more than lather the masses, to be the oracles of truth about the results of these efforts? The cognitive biases alone should disqualify them from being the arbiters of their own success. So, if we do not want to continue to indulge the principle of diminishing returns in QI, we need to take a sober look at what we have invested and what this investment has accomplished. Then and only then can we claim to practice evidence- rather than politics-or dogma-based policy.

Is it the science or is it the message?

As you know, I have been thinking a lot about cognitive biases lately. Apropos this, I have had this nagging thought that even smart people (I flatter myself by including myself in this group) cannot escape this trap. What sent me down this rabbit hole is a tweet I saw in my stream from someone whom I respect very much, a science writer of the highest caliber with an international reputation. This person said:

I'm skeptical of her; hearing same from many others: Ntl Acad of Sci researcher: "Disconnect: Truth Abt Cell Phone Rads http://bit.ly/dxpTtG

Now, there is really nothing fundamentally wrong with this statement -- we seek out the opinions of those we trust and align ourselves with them in the absence of evidence to the contrary; this is how friendships and other relationships tend to work. But what is striking to me is the disparity of effort to seek out real evidence when it comes to stuff we agree with vs. stuff that we do not like.

The link is to a radio interview with Devra Davis, whom I have mentioned on several occasions on this blog. I read her book on cancer with great interest, and the history that she told in a measured and well documented way sent shivers up and down my spine. Yes, there was in fact a conspiracy to suppress the science behind the link between smoking and cancer. She described the anatomy of this conspiracy in minute detail, well researched and referenced. The continued legacy of the tobacco strategy is our callousness toward the bulk of chemicals on the market and their health effects, our allegiance leaning more towards saving the economy than lives.

In her new book, which I have not yet read, Davis turns her quill to the cell phone and electronics industry. If this book is as well researched and footnoted and damning as the Cancer book, we should all pay attention. Of course, I can see how the message can become quite unpopular -- the devices that we all rely on, even are addicted to, the devices that make the world go round and add trillions of dollars to our beloved economy, are on trial! Radiation from cell phones, brain cell death and proliferation, especially in children -- not a comfortable line of logic to follow. Yet, if we are to be balanced and scientific about it, should we not examine the science that this author presents, rather than knee-jerk into the ad hominem?

Now, I am singling out this particular tweet unfairly -- the person who wrote it probably did not give much thought to the words. But this is an influential individual, and, as I already mentioned, we often join an opinion bandwagon for reasons other than rational thought or preponderance of evidence, so one influential individual may blind the less critical masses to the very opinion nature of this remark. This is a potential broken telephone game, which will end up in "oh, her, she is a crank, a pseudoscientist and not credible". In fact, the only questions of her credibility that I could find was in comments from listeners like this:

I'm really disappointed that Leonard didn't ask this woman any tough questions, and pretty much just let her rant using his show as her soapbox. I just looked up her CV, and this woman does not have one degree in science. Her background is in sociology and history. Where's the science?

Well, if one bothers really to look up her credentials, this is what we find:

Dr. Davis holds a B.S. in physiological psychology and a M.A. in sociology from the University of Pittsburgh, 1967. She completed a Ph.D. in science studies at the University of Chicago as a Danforth Foundation Graduate Fellow, 1972  and a M.P.H. in epidemiology at the Johns Hopkins University as a Senior National Cancer Institute Post-Doctoral Fellow, 1982.  She has also authored more than 190 publications in books and journals ranging from the Lancet and Journal of the American Medical Association to Scientific American and the New York Times and blogs in Freakonomics for the New York Times, Huffington Post and elsewhere.

And her honors and appointments span the entire page. So, in terms of pedigree, she is impeccable, she has authored more respectable peer-reviewed publications than many giants I know, and she has been on all the "right" commissions and boards.

So, why are we skeptical of her? I think we need to be very honest with ourselves here and really examine our objections. And I would go even further to say that we not only need to examine the science, but we need to be cognizant of whether we are examining this science with the same amount of ferocious scrutiny as we examine science that aligns with our preconceived notions of the world. For it is this magnitude of ferocity ("ferocity quotient"?) that will tell us whether we are biased or completely objective. Ultimately, the question each of us needs to answer for herself is this: Is it the science or is it the message?

Please, don't kill the messenger!

     

Assume a spherical cow

You know that joke about the farmer whose cows are not producing enough milk? A university panel gathers under the leadership of a theoretical physicist. They analyze each aspect of the problem thoroughly and carefully, and after much deliberation produce a report, the first line of which is "First, assume a spherical cow in a vacuum". This joke has become short-hand for some of the reductionist thinking in theoretical physics, but it can just as easily personify the field of statistics, upon which we rely so heavily to inform our evidence-based practices.

Here is what I mean. Let us look at four common applications of statistical principles. First, descriptive statistics, usually represented by Table 1 in a paper, in which we are interested in understanding the measures of central tendency such as the mean and median values. I would argue that both measures are somewhat flawed in the real world. As we all learned, a mean is a bona fide measure of central tendency in a normal or Gaussian distribution. And furthermore, in such a distribution 95% confidence intervals are bounded by 2.5x the standard deviation from the mean. Although this is very convenient, few things about human biology are actually distributed normally, while many cluster to the left or to the right of the center creating a tail on the opposite side. For these skewed distributions a median is the recommended measure to be used, bracketed by the boundaries of the 25% to 75% range of the values, or the interquartile range. But in a skewed distribution it is exactly the tail that is its most telling feature, as described so eloquently and personally by Stephen J. Gould in his "The Median Isn't the Message" essay. This is especially true when more specific identification of characteristics precipitates the numbers dwindle; e.g., if instead of such general measures as blood pressure among all-comers you want to focus on morning blood pressures in such specific group as African-American males with a history of diabetes and hypercholesterolemia who attend a smoking cessation program, say. And this is important because at the level of the office encounter each patient is a universe of his/her own risk factors and parameters that does not necessarily obey the rules of the herd.  

Second, analytic modeling relies on the assumption of normality. To overcome this limitation we transform certain non-normal distributions to their log forms, for example, to normalize them in order to force them to play nicely. Once normalized, we perform the prestidigitation of a regression, reverse-transform the outcome to its anti-log, and, voila, we have God's own truth! And even though statisticians tend to argue about such techniques, in the real world of research, where evidence cannot wait for perfection, we accept this legerdemain as the best we can do.

The next example I will use is that of pooled analyses, specifically meta-analyses. The intent of meta-analyses is to present the totality of evidence in a single convenient and easily comprehended value. One specific circumstance in which a meta-analysis is considered useful is when different studies are in conflict with one another, as for example, when one study demonstrates that an intervention is effective, while another does not show such effectiveness. In my humble opinion, it is one thing to pool data when studies suffer from Type II error due to small sample sizes. However, what if the studies simply show opposite results? That is, what if one study indicates a therapeutic advantage of treatment T over placebo P, but another shows the exact opposite? Is it still valid to combine these studies to get at the "true" story? Or is it better to leave them separate and try to understand the potential inherent differences in the study designs, populations, interventions, measurements, etc.? I am not saying that all meta-analyses mislead us, but I do think that in the wrong hands this technique can be dangerous, smoothing out differences that are potentially critical. This is one spherical cow that needs to be milked before it is bought.    

The last cow is truly spherical, and that is the one-on-one patient-clinician encounter, wherein the doctor needs to cram the individual patient with his/her binary predispositions into the continuous container of evidence. It is here that all of the other cows are magnified to obscene proportions to create a cumbersome, at times incomprehensible and frequently useless pile of manure. It is one thing for a clinician to ignore evidence willfully; it is entirely another to be a conscientious objector to what is known but not applicable to the individual in the office.

But let's remember that I, as a health services researcher and a self-acknowledged sustainability zealot, am in awe of manure's life-giving properties. Extending the metaphor then, this pile of manure can and should be used to fertilize our field of knowledge by viewing each therapeutic encounter systematically as its own experiment. The middle way is that between the cynicism of discarding and blind acceptance of any and all evidence. That is where the art of medicine must come in, and the emerging payment systems must take it into account. Doctors need time, intellectual curiosity and skills to conduct these "n of 1" trials in their offices. If the fire hose of new data insists on staying on, this is the surest way to direct "conscientious and judicious" application of the resulting oceans of population data in the service of our public's health. And that should make the farmer happy, even if his cows are spherical.                  

Evidence: What the bleep do we really know?

So I know that my blogging has fallen off a bit, and I am sure you are all sorely disappointed (or not). Let me try to explain why.

There are several things going on to nudge me toward the next rung in the evolution of my thinking about healthcare and evidence. The event most responsible for this re-examination of my assumptions is the previously-mentioned illness of a loved one, along with my need to be closely involved with his care decisions. I think that I have generally adequately voiced my frustration with how decisions are made in our healthcare system, and where the switches for these choices should be as opposed to where they are. I am even more convinced now that by the time the physician and the family are considering expensive options with marginal effectiveness, the cat is already out of the bag: how can either the distraught family or the committed healthcare provider not consider those, despite shaky evidence of the value?

But this is not all. I have actually been cogitating the entire way we do evidence and evidence-based medicine. When we invoke evidence, we generally talk about some scientific study's findings, or a group of studies either showing similar or differing results. Let's keep stepping back and looking at the components. The scientific community, based on some statistical and other methodological considerations has come to a consensus around what constitutes valid study designs. This consensus is based on a profound understanding of the tools available to us to answer the questions at hand. The key concept here is that of "available tools". As new tools become available, we introduce them into our research armamentarium to go deeper and further. What we need to appreciate, however, is that "deeper" and "further" are directional words: they imply the same direction as before, only beyond the current stopping point. This is a natural way for us to think, since even our tools are built on the foundation of what has been used previously.

So, what emerges is a picture of being emotionally committed to a certain way of viewing the question, the processes of answering it and the actual answers. And what if by narrowing ourselves to this one particular way of examining the world, to this one particular way to collect and interpret the evidence, we have effectively ignored all other possibilities?

Let me try to clarify what I mean a little further. We spend our days walking through waves. Some of these waves we can detect through our own senses: sound, light, for example. Others we need special external "receptors" to detect, such as radio and micro-waves. Is it possible that there are some other, heretofore unknown waves (or other phenomena) that are around us that we are unable to attune to because of our biology? Is it kooky to think that this is possible, or is it simply blind to walk away from such possibility?

The answer may be that it is both. Nevertheless, it is highly likely that there are many biological phenomena that are not amenable to being examined through our narrow prism of current methodology. We as scientists and clinicians need to be open to this possibility. The Buddha said that both preachers and scholars are blind because they constantly commit themselves to partially-seen truths. Given the shambles in which we find not only our nation's healthcare system, but also the health of its citizens, this would be a great time for this epiphany to penetrate our collective psyche and strive toward a broader view of possibilities. Let's give up the arrogance of ignorance in favor of the humbleness of enlightenment.

And these are my latest thoughts. I am not saying that they are the only way or that they are even remotely correct. But here they are nevertheless. Do with them what you will.            

Lies, big lies and... epidemiology?

Now, as you know, I am a big fan of epidemiology. I do not believe that a randomized controlled trial is the be-all-and-end-all in evidence generation, and a well done observational study can add to our reservoir of knowledge much more efficiently. Of course, I, as many others, acknowledge certain limitations of epidemiologic design. However, many of them can be overcome with careful design and analysis.

I have to confess, though, that over the last week I have bumped into two news stories that have made me cringe. The first, reported a couple of days ago and based on a Kaiser study, showed that people

drinking at least four cups of coffee a day were 18 percent less likely to be admitted with a heart rhythm disturbance than those who drank no coffee at all.

So, great, the public may take away the message that drinking coffee prevents a-fib. Well, I have to say that the reporting of this was measured and tried to avoid this unfortunate inference of causality. Yet, it left enough room to imply that yes, perhaps there is a causal link. So, what's wrong with that?

Bear with me while I bring in the second example of a study that bugged me this week, based on the Women's Health Initiative. You may recall that the WHI is the large NIH-sponsored study that a few years ago turned hormone replacement therapy on its head. The study had a randomized component and an observational component. So, the newest analysis shows that women

who drank the equivalent of one to two drinks a day -- be it beer, wine, or liquor -- were 30% less likely than non-drinkers to become overweight or obese.

 Do you see the similarities? So why am I bothered? To me this is the classic case of a high potential for confounding by indication. What's that you say? That is a situation in which a subject that has the exposure in question (in these two cases coffee and alcohol) is inherently different from one who does not, and this difference is what determines the probability of the exposure itself. Why should this present a problem in a study where the authors carefully adjusted for confounding, which is true for both of the studies? It is a problem because the kind of confounding that this represents is impossible to tease out without real-time attention to the subject.

Here is how it would work in the case of coffee study. Say I am a person with paroxysmal (occasional) a-fib, and I have noticed that if I drink so many cups of coffee per day, I get into brief episodes of palpitations. Not enough to send me to the doctor's office or the hospital, but enough to start thinking about cutting out caffeine. So, I stop drinking caffeine, and continue with my baseline frequency of a-fib attacks. You see the problem? Is it possible (or even probable) that those people who drink four or more cups of coffee per day somehow have an inherently higher threshold for slipping into their a-fib than those who do not? And if the answer is "yes", then the four cups become a marker for someone who can tolerate them, rather than the cure for a-fib. You can construct a similar explanation with the two drinks and weight.

So, while I love epidemiology and its methods, I am wary of hanging my hat on associations that may likely be explained by confounding by indication. And although the stories were reported with many caveats, human nature may prevent us from hearing the nuance. It is clear that in both these instances the burden of proof is on the researchers to show me that I am wrong.      

 

  

When the black box of evidence is empty

A close friend of mine gave birth a few days ago to her second child. My friend is a slight woman whose infant came out at 6 lb 9 oz, even though she was calculated to be at 36 weeks' gestation. The infant and the mom have done great with absolutely no complications.

This is why, when I was talking to the Dad this morning, I was shocked to hear that the baby had to undergo a test I had never heard of: something to do with putting the child in its car seat and monitoring its heart rate and oxygen saturation for one hour. Mind you, my youngest is 9 years old, and I had never hear of this test, so I looked it up.

Turns out, this is a test formally known as a "pre-discharge car seat challenge". I went to the American Academy of Pediatrics web site to look for their take on it, and here it is:

Hospitals should develop a policy to ensure provision of a period of observation in a car safety seat before hospital discharge for each infant born at <37 weeks' gestation to monitor for possible apnea, bradycardia, or oxygen desaturation.

I went on a circuitous journey to trace the origin of this recommendation. Turns out, it is based on a consensus statement from an APP committee from 1996, the reference for which is this paper from 1993 about implementation of the 1990 AAP recommendations. Looking at the 1990 version, however, does not help in the least, as there is no mention at all of the test in question, or anything to do with infants under 37 weeks' gestation. So, this is where the dubious evidence trail ends. Period. No evidence near as I can see. [If someone knows of a more complete statement that does include the information I am interested in, please, point me to it].

Why did I go after this piece of information? Well it seemed to me to arise out of the "more is better" fallacy, and I wanted to understand a) the evidence that led to the recommendation, b) how the test alters practice, and finally, and most importantly, c) does this practice impact the desired outcomes (i.e., prevent something bad).

So, I went to one of the most trusted sources in evidence synthesis, the Cochrane Collaborative, to see if they have examined this issue. Lo' and behold, I found this 2006 review, whose findings were summarized thusly:

There is no evidence that undertaking a pre-discharge "car seat challenge" benefits preterm infants. The "car seat challenge" assesses whether preterm infants who are ready for discharge home are prone to episodes of apnoea (stopping breathing), bradycardia (slow heart rate), or desaturation (low oxygen levels) when seated in their car seat. However, it is not clear whether the level of oxygen desaturation, apnoea, or bradycardia detected in the car seat challenge is actually harmful for preterm infants. Additionally there is concern that the use of the car seat challenge may cause undue parental anxiety about the safety of transporting their infant in a car seat. Despite these uncertainties, and despite the widespread use of the test, we have not identified any randomised controlled trials that assessed whether undertaking a car seat challenge is beneficial or harmful to preterm infants.

So, here is a kid, born at supposedly 36 weeks gestation, but looking and acting like a full-term infant, having to sit in her car seat for 1 hour to undergo monitoring that has absolutely no evidence behind it. Putting aside the inconvenience, the false positives, the parental anxiety, let's think about the costs. I could not find published reimbursement rates for this useless test. However, if you think about the person-power, the equipment used, and the potential prolongation of hospitalization that it may induce, the price tag is likely not trivial.

This case is instructive to me. As I have said before, a pitiful minority of medical practice has good evidence behind it. It is precisely this type of testing, done to comply with what looks to be a self-propagating yet outdated recommendation, that healthcare reformers in Washington need to be addressing. Furthermore, when evidence is kept in a black box, we are likely to find that the box is indeed empty.a,

bradycardia, or oxygen desaturation. An appropriate

hospital staff person should conduct the observation.

Hospitals should develop policies to

  

Evidence-based free-market economics?

Seeing as I am an outcomes researcher, I thought I'd float this question to all the zealots of free market economic theories: Have their outcomes ever been examined in an evidence-based manner? Here is what I mean.

In medicine, the highest quality evidence comes from randomized controlled trials (RCT). In these deliberately and painstakingly planned and executed studies subjects are enrolled randomly into either a treatment group or a placebo (or standard of care) group, and certain relevant outcomes over time are compared between the two groups. This design works well for drug trials, for example, but not so well for something like environmental exposure studies. Thus, in order to understand the adverse effects of smoking on health, no RCT could be performed ethically, so these data come to us from epidemiologic studies. Though deemed by the EBM community to be somewhat inferior to an RCT, they have their advantages: they can enroll more and varied patients, they are cheaper, they are more feasible to do over long periods of time, and they do not require alterations in naturalistic practice. The flip side is that we have to be more careful about introducing such inaccuracies as bias, misclassification and others, though their most important limitation as cited by some is their inability to prove cause and effect.

Putting the limitations of epidemiologic methods aside, their practicality makes them attractive and ubiquitous in helping us understand healthcare. One of the most important issues we face in EBM is understanding the risk-benefit profile of what we do. To get at this, our studies have to be planned carefully to incorporate not only the important exposures, but also the relevant, valid and complete endpoints (or outcomes). Thus, it is not enough to pat ourselves on the back by saying that a drug decreases the risk of a clot; we also have to show that this benefit outweighs the attendant risk of bleeding in the specific population of patients.

Let's apply these principles to the free-market theory. Have its advocates examined its risk-benefit profile in a scientifically rigorous manner? According to Harvard's Stephen Marglin, they have not. In his book "The Dismal Science" he makes a strong argument that economists, by pushing their free market theories with religious fervor, have undermined the very structures of our communities. Furthermore, this noted professor of Economics maintains that free market theories have never been impartially evaluated. So, there you have it: the single unifying thread of the entire Western civilization today is just as faith-based as a rain dance around a fire. So, when we talk about evidence for climate change and peak oil and how a concerted effort to mitigate these phenomena would affect our economy, let's not fool ourselves: the evidentiary standards we demand for free market's detrimental effects are much more stringent than those for its benefits, which we have essentially swallowed on faith. Hook, line and sinker.      

Shaky evidence for flu vaccination policies

This in the British Medical Journal from a Cochrane Center -- evidence for policies is shaky at best.

BMJ  2006;333:912-915 (28 October), doi:10.1136/bmj.38995.531701.80

Analysis and comment

Public health

Influenza vaccination: policy versus evidence

Tom Jefferson, coordinator^{1
1 Cochrane Vaccines Field, Anguillara Sabazia, Roma 00061, Italyjefferson.tom@gmail.com}


QUOTING:
Summary points

Public policy worldwide recommends the use of inactivated influenza vaccines to prevent seasonal outbreaks

Because viral circulation and antigenic match vary each year and non-randomised studies predominate, systematic reviews of large datasets from several decades provide the best information on vaccine performance

Evidence from systematic reviews shows that inactivated vaccines havelittle or no effect on the effects measured

Most studies are of poor methodological quality and the impact of confounders is high

Little comparative evidence exists on the safety of these vaccines

Reasons for the current gap between policy and evidence are unclear, but given the huge resources involved, a re-evaluation should be urgently undertaken

(A hat tip to @cebmblog)

Absence-of-evidence based medicine

The term evidence-based medicine has made it into our colloquial language over the last decade. David Sackett, considered one of the founding fathers of the movement, defined it as

... the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research." (Sackett D, 1996)

The reality is that only about 5% of all medical practice recommendations have good quality evidence behind them, and perhaps another quarter have some evidence. The rest of medicine is practiced by "the seat of the pants" approach, or "the 15 years of experience" method. Clearly, this is not wrong when there is absence of evidence.

It is a fallacy, however, to confuse this absence of evidence with the evidence of absence of an association or an effect. What I mean by this is that in the former no one has bothered to generate the evidence, whereas in the latter there is evidence that does not support an association between the exposure and outcome. The latter is in fact evidence, while the former is absence of any evidence.

The distinction may be subtle, but is worth making. Take some of our environmental policies, for example. There are 80,000 potentially toxic chemicals on the market, including household cleaners, pesticides, etc., of which only 200 have been evaluated for toxicity by the government agencies. Does this represent the evidence of absence of any harm to human health? Of course not! This is a clear case of absence of evidence -- we have not deemed it fit to put resources into investigating the association between exposures to these chemicals and health. So to say that these chemicals are harmless would be a fallacy in the absence of a concerted effort to study them in the context of public's health.

Similarly, because our adverse drug reactions reporting system has traditionally been voluntary, the absence of reporting does not equal the absence of adverse events. Again, a classic situation of absence of evidence and NOT evidence of absence of these events.

So, given what I have just said, it is in fact difficult to practice evidence-based medicine or evidence-based public health. Unfortunately, even the evidence that exists is slow to wend its way into practice. As the RAND researchers demonstrated several years ago, the median penetration of evidence into practice is on the order of 50%, which is no better than a coin toss! So, we can certainly do better than that. When there is evidence. But, let's not be fooled: most of the time what we end up practicing is absence-of-evidence based medicine. Question your doctors closely to distinguish between the two, as this distinction is the key to rational decision making.