Why "efficacy" does not equal "effectiveness"

As you may have noticed, one of the biggest medical conferences is in progress right now in Chicago -- the American Society of Clinical Oncology. It is the premier meeting of cancer doctors, where new data are the order of the day. This stream of data is reflected in a constant barrage of media coverage. One of the stories I saw today serves as a great illustration of the title of this post: why efficacy does not equal effectiveness. Although I do discuss this in my book, I thought it might be nice to apply these ideas to a current story.

The MedPage story opens with this:

Older patients with advanced cancers treated in the community had survival that fell short of that for patients who received the same regimens in clinical trials, analysis of a large government database showed.

Most of the differences were small in absolute terms, except for patients with stage IV colorectal cancer treated with the FOLFIRI chemotherapy regimen. Patients in the community had a median survival of 16.1 months, which was 30% lower than that of patients treated with the same regimen in a cooperative-group randomized clinical trial.

Moreover, survival among FOLFIRI-treated older patients was 4 months shorter than that of older patients treated with FOLFOX chemotherapy, Elizabeth B. Lamont, MD, reported here at the American Society of Clinical Oncology meeting. 

This is a common conundrum, and a poster child for the distinction between efficacy and effectiveness. So, what are these two "e"s? Efficacy is essentially a statistical distinction between the outcomes of interest among patients who undergo a particular treatment compared to those who do not, or those who get treated with a placebo. Efficacy is measured under the controlled circumstances of a clinical trial, where only very specific patients are enrolled, very specific treatments are administered, and very specific outcomes are monitored. These trials usually randomize patients to either the treatment or the placebo, thus ensuring that treated and untreated patients are the same in all ways save for the treatment under examination. Efficacy is frequently represented by what we call surrogate outcomes, such as laboratory measurements or radiographic tests. In cancer studies, for example, a frequent surrogate outcome that is used is so-called progression-free survival. This refers to the duration of time from treatment that a person is alive AND the tumor has not shown any evidence of growth on a scan. Another frequently used surrogate measure is blood pressure as a marker for heart disease. These are surrogates because, although correlated with such important outcomes as death and heart attacks, respectively, they themselves do not tell us with any precision how our interventions impact these ultimate measures. I will not belabor at this time why we rely on efficacy and surrogate outcomes, since I go into detail about that in the book.

Effectiveness, on the other hand, is something altogether different. Effectiveness tells us exactly what happens in the real messy world to outcomes that matter, such as death and quality of life, in conjunction with the treatment in question. We have known for a long time that the outcomes we see in naturalistic studies are often much less spectacular than those reported in RCTs of efficacy. Why is this? And more importantly, which do we believe? The second question is easier to answer than the first: we believe what happens in the real world, because it is precisely what happens in the real world rather than in the laboratory of clinical research that matters. As to why this difference exists, there are many reasons for this, most of which I have discussed elsewhere on this blog and in Between the Lines. Some of the reasons may have to do with patient selection, which in real life tends to be less restrictive than in RCTs. For example, individuals who are more ill may get the intervention that was intended to be given to those with lesser illness severity. In this population the intervention may not prove to be as effective as in those who are not as ill. This is called "confounding by indication," and we talked about it most recently here. Other reasons may be that other conditions patients have in real life, which tend to be excluded from RCT populations, attenuate the impact of the intervention. When looking at mortality in cancer, patients with end-stage heart disease may be excluded from the RCT, but treated in the wilds of clinical practice. And this treatment may give us a Pyrrhic victory, where cancer is indeed held at bay, but the patient dies of his heart disease. And here is yet another reason for the efficacy-effectiveness disconnect: attribution. In clinical trials there is a meticulous process that has to be followed in order to attribute the cause of death to a particular disease. In real life -- not so: death certificates are a notoriously dicey source of information on the causes of death.

So here are some of the challenges with applying RCT data to the real world, illustrated so palpably in the story we started out with. Please, do not misunderstand my message: I am not saying that RCTs are useless. What I am saying (I must sound like a broken record by now) is that we need different types of studies to see the whole picture. RCTs by their nature are exclusive undertakings whose findings are only narrowly translatable to the real world. Naturalistic observational data are key components of building the entire jigsaw puzzle of how our interventions really work.                

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Could our application of EBM be unethical?

As you may have noticed, I have been thinking a lot about the nature of our research enterprise and its output as it relates to practical decisions that need to be made in the real world by physicians and policy makers. I have come to the conclusion that it is woefully inadequate in so many ways, and in fact it may even be borderline unethical. My thinking on this has been influenced at least in part by some of my reading of the work by the Tufts group led by David M. Kent, who has written a lot about the impact of heterogeneous treatment response on the central measures we report in trials -- I urge you to look their work up on Medline. (My potential COI here is that I did all of my Internal Medicine and Pulmonary and Critical Care training at Tufts in the 1990s, but did not know or work with Kent or his group). But admittedly I have been thinking about a lot of this stuff on my own as well, as I have advocated risk stratification for quite some time now. So, here is what I have been thinking.

The randomized controlled trial is what puts "evidence" into evidence-based medicine. It is the sine qua non of EBM, and is the preferred way to resolve questions of "does it work?" The rigorous experimental design, the blinding and placebo controls are all meant to maximize what we call internal validity, or assure that what we are detecting is not due to bias or chance or confounding or placebo effect or some other invalidating reason. But the one downfall of these massive undertakings is that we arrive at some measure that tells us what the response was on average, and how that differed on average from placebo. The beauty of such a measure of central tendency is also its curse: it tends to smooth out any of what we call "noise" in the signal, but at the same time this smoothing cannot differentiate between noise and important population differences in both baseline characteristics and responses. Therefore, I contend that it lacks what I have termed "individualizability", a threat to validity most felt at the bedside.

Take for an example, a cholesterol-lowering medication. Let us assume that the mean or median lowering that it provides is 20 points over 6 months. This number conflates those patients who respond vigorously (say with a 40 point lowering) with those with much more sluggish to no responses. So, while the total patient group may represent a broad spectrum of the population, we walk away from the data not really understanding who is likely to respond well and who not so well. Make sense?

Now, this becomes potentially even more important when looking at adverse responses to interventions. Just because on average we do not detect more deaths, for example, in the treatment than in the placebo arm, there may be important differences between those susceptible to this outcome in the two groups. That is, while in the placebo group the deaths may be due to the disease under investigation, in the treatment group the deaths may be directly attributable to the experimental treatment, and thus may impact different subjects. And although the balance of the accounting does not differ between groups, we may be creating a variant of the trolley problem.

What would be helpful, but is less accepted by statisticians and regulators, is exploration of subgroups of patients within any given trial. I will go even further to say that the trial design should a priori include randomization stratified by these known potential effect modifiers, so as to give the power and validity to understand the continuum of efficacy and harm. Because we rarely do this, the instrument of evidence becomes a blender of forced homogenization, lumping together to create results we like to call generalizable, but lacking the individualizable data that a clinician needs in the office. And it is this lack of individualizability that to me creates a possibility of an ethical breach at the point of treatment.

It has been asserted here that 90% of all medicines work only in 30-50% of the people who qualify for them. So, for the majority of drugs, we have less than a coin-toss chance that it will work. While at 50% we have equipoise, below 50% equipoise disappears. And may I remind you that equipoise is a requirement for an experimental protocol. In other words, if we have data that leans in either direction away from the 50-50% proposition of efficacy, the ethics of an investigation are brought into question. In the current situation, where the probability of a drug working is less than or equal to 50%, does not the office trial of such therapy qualify as investigational at best and potentially unethical at worst? Of course, this would depend, as always, on the risk-benefit balance of treating vs. not treating a particular condition. But my overarching question remains: should such a drug not be considered investigational in every patient in the office when the more individualizable data are not available?

Some of these issues may be remedied by the adaptive trial designs that the FDA is exploring. But knowing the glacial pace of progress at the agency, this solution is not imminent. Also, I am by no means a bioethicist, but I do spend a lot of time thinking about clinical research as it applies to patient care. And this situation seems ripe for a more well informed multi-disciplinary discussion. As always, my ruminations here are just a manifestation of how I am thinking through some of the issues I consider important. None of this is written in stone, and all is evolving. Your thoughtful comments help my evolution.  



 

Furthering the discussion

My original intent was to go through Dr. Novella's and Orac's criticisms individually and take it from there. On second thought I decided not to take that approach. Instead, here is my response.

Firstly, I am grateful that there has been so much discussion about our views. Amid many valid points in their posts made with a skillful turn of the phrase, I saw quite a lot of sarcasm as well. I am sure that the tone of my original post is what incited it, and for that I am sorry: I really do want to have a civil discussion about these ideas, as I realize that we are all learning all the time, and the only way to gain a better understanding of a topic is through discourse. So, again, I apologize for setting the confrontational tone, and will try to avoid it in the future.

I do believe our views are more same than different. We both (SBM group and I) understand that science evolves, that evidence is not stagnant and the sense of certainty frequently conveyed to the lay public by the media is oftentimes misplaced. We simply disagree on the extent to which there is uncertainty in evidence. While it is true that the oft-cited 5-20% number representing the proportion of medical treatments having solid evidence behind them is very likely outdated, the kind of evidence we are talking about is a different matter.

In the hierarchy of evidence, depending on where you look, it is either meta-analyses or the randomized controlled trial that is the gold standard. The latter is a great proof of concept tool, but it is necessarily limited in its external validity, or generalizability. The reason for this is that these trials, frequently done for regulatory purposes, very limited types of patients, exert extremely stringent controls on the total care of the patient (or else are criticized for not doing so if they fail to do so), focus on short-term and surrogate outcomes (hence, the use of cholesterol lowering as a marker for cardiac mortality, for example), and do a fairly abysmal job as a rule considering the sources of heterogeneity of response. The interventions come to market and are typically used in a much broader population based on the evidence of the RCT. This paper, one of many in the same vein, is a nice illustration of a perennial problem with trial evidence, where real-world use of a therapy goes far beyond the available evidence. And although this paper addresses issues with evidence used for reimbursement, these are the same studies that feed guideline recommendations. Certainly meta-analyses, which are a way to combine the data from multiple RCTs in a systematic way, when done well can give us greater confidence of the direction and the magnitude of the treatment effect, but they in no way overcome the generalizability issues of their component RCTs.

The next rung of the evidence ladder is observational data, specifically cohort studies first prospective, then retrospective. I am actually a great fan of observational data, as I have mentioned in the past. Cohort studies give us the opportunity to examine what happens in the real world without imposing artificial conditions necessary in a clinical trial. Observational data can be great when describing epidemiology of a particular disease, the frequency of a given exposure, how different characteristics can modify the relationship between the exposure and the outcome. One of the most attractive features of cohort studies is that the population can be observed over long period of time -- just look at the Nurses' Study, the Framingham Cohort, and others. But these types of studies also have important limitations, and these are readily acknowledged as a heightened susceptibility to bias (especially in the retrospective studies), the possibility of misclassifying important events, and, despite our best efforts to adjust for it, residual confounding. I will come clean and admit my affection for observational data, even despite the fact that it is lower on the totem pole of evidence than an RCT. I really love this paper by Rothman and Greenland that takes a bird's eye view of our research debates. The whole paper really tickles the brain, but I will quote from a section of it briefly here:

Impossibility of Proof

Vigorous debate is a characteristic of modern scientific philosophy, no less in epidemiology than in other areas. Perhaps the most important common thread that emerges from the debated philosophies stems from 18th-century empiricist David Hume’s observation that proof is impossible in empirical science. This simple fact is especially important to epidemiologists, who often face the criticism that proof is impossible in epidemiology, with the implication that it is possible in other scientific disciplines. Such criticism may stem from a view that experiments are the definitive source of scientific knowledge. Such a view is mistaken on at least two counts. First, the nonexperimental nature of a science does not preclude impressive scientific discoveries; the myriad examples include plate tectonics, the evolution of species, planets orbiting other stars, and the effects of cigarette smoking on human health. Even when they are possible, experiments (including randomized trials) do not provide anything approaching proof, and in fact may be controversial, contradictory, or irreproducible. The cold-fusion debacle demonstrates well that neither physical nor experimental science is immune to such problems.

Some experimental scientists hold that epidemiologic relations are only suggestive, and believe that detailed laboratory study of mechanisms within single individuals can reveal cause–effect relations with certainty. This view overlooks the fact that all relations are suggestive in exactly the manner discussed by Hume: even the most careful and detailed mechanistic dissection of individual events cannot provide more than associations, albeit at a finer level. Laboratory studies often involve a degree of observer control that cannot be approached in epidemiology; it is only this control, not the level of observation, that can strengthen the inferences from laboratory studies. Furthermore, such control is no guarantee against error. All of the fruits of scientific work, in epidemiology or other disciplines, are at best only tentative formulations of a description of nature, even when the work itself is carried out without mistakes.

What follows in the hierarchy of evidence are case-control studies, done for some very specific reasons, then case reports and finally expert opinion. When evidence-based guidelines are developed, a comprehensive systematic literature review is undertaken, and all the evidence is examined and ranked. Based on these papers, a recommendation is made and a strength of this recommendation is reported based on the quality of the underlying evidence. This is an arduous and costly process, and it is commendable that it is undertaken. At the same time, given the limitations of the components of the guideline, the final product can be quite inconclusive or even misleading (I hate to bring it up, but look at the screening mammography debate, as well as the recent HRT recommendation reversal). I think it is obvious that I believe in the scientific method, I am simply not convinced that we have done such a great job generating trustworthy evidence in many instances. At the same time, I am not totally nihilistic about what we know, but am somewhere between thinking we have good evidence for a lot of stuff vs. not having any for anything at all.

Allow me one more piece of evidence, if you will, though this is merely anecdotal coming from my dual experience as an author and peer reviewer. I am occasionally floored by the quality of peer review. I have had reviews say on the one hand that of course the paper should be accepted, since it comes from such a reputable group, and on the other reject out of hand papers based on the reviewers' profound lack of understanding of the methods employed. And lest I sound like a crybaby, let me say that I welcome a well-reasoned rejection. What I am talking about is not that. And this is not a surprise, since pretty much anyone can sign up to be a peer reviewer, since, to the best of my knowledge, there is no set of qualifications that journals ask for in their reviewers. And this, so far as I know, applies even to such high caliber publications as JAMA.

So, these are my thoughts on evidence-based medicine. I welcome responses to this, as my understanding of this science is constantly evolving, and differing well-reasoned opinions really help me get a better handle on this stuff.

I will try to tackle my CAM argument next. If I in any way implied in my remarks that I encourage allopathic physicians (by the way, I am not using it in a derogatory way, but merely as it is defined here; in fact, until today I was blissfully unaware of its negative connotation) to be purveyors of CAM, I sincerely apologize. I am pretty sure, however, that I have never made such a statement, as this is not what I believe. My belief is that all modalities that may impact what happens to public's health need to be evaluated for safety, not question. I think we both agree, since there is really no reason to think that something like homeopathy has anything that can help, by the same token we do not believe that it have anything that can hurt. Same with healing crystals, reiki and prayer. So, if a person wants to engage in these activities, and they are perfectly safe physically, be my guest. Other modalities, such as chiropractic, acupuncture, herbalism and the like, definitely need to be evaluated more stringently, as there is reason to think that they may cause harm. And the decisions about their use must be made based on the probability of harm vs. the perceived probability of benefit. Why do I think that these should not be regulated the same way as allopathic treatments? Well, herbs grow naturally and I dare say we have little to say about what our patients grow in their back yards, unless of course the thought is to regulate them the way we do marijuana. As for chiropractic, it is already regulated, though to what extent I am not sure, and would love to hear from someone who knows. Since its techniques most resemble surgical interventions, the level of evidence for them should perhaps be the same as that which we demand in the surgical literature. This is just a thought, and I am not sure that I am correct in this, so other views are, as always, welcomed. What is coming through for me is that perhaps my call to equipoise was a little over the top, as I do not seem to be approaching the above CAM issues in a frequentist, but more in a Bayesian way (though I remain committed to equanimity). Yet, there is something to be said about the frequentist approach, even though it is not my way generally. The frequentist approach, which is what underlies the bulk of our traditional clinical research, does not rely on differential prior probabilities for different possible associations, but treats them all equally. Despite many disadvantages, one obvious advantage is that we do not discount potential associations that do not have biologic plausibility, given our current understanding of biology, and sometimes help us stumble on brand new hypotheses. So, clearly, there is a tension here, and I am still working on what is the better way, if any.

My final words will be about vaccination. It is disheartening to be lumped with "anti-vaxers", as has been done in the comments to Dr. Novella's and Orac's posts. While my bruised ego will survive this insult, I would like to question this assertion. Nowhere have I said that vaccinations are a bad idea or present a real danger to our children. The hype surrounding the vaccination-autism "debate" is abhorrent to me. What I have stated, however, is that I am of the opinion that we have gone a bit overboard with some of them, one being the chicken pox vaccine. Now, this does not make me an "anti-vaxer"; this just makes me a bit skeptical. The way I view the data is that the advantages for this vaccine are mostly economic, in that they prevent parents from missing days at work. Now, I am certainly not opposed to making such a vaccine available to parents who desire it, but I am not convinced that it should be a prerequisite for my kid to go to school. Given that there is always a possibility of an adverse reaction, no matter how small that possibility is, if the risk of it may outweigh the benefit (and here I do not mean the benefit of having mom show up at work), it has to be weighed very carefully. And even though the question asked by one of the comments raises the issue of an immunocompromised child worrying about potentially being exposed to chicken pox, given the known issues with breakthrough disease, I am not sure that immunizing all of his/her contacts would produce anything other than a false sense of security. My sentiment about the HPV vaccine is similar, to respond to another comment. For reasons laid out here and elsewhere on my blog, I am pretty convinced that it would be a compete subversion of the intent of vaccinations to make it into another mandated shot. To date HPV is merely a recommendation, whose validity I am free to question, though last I heard there was movement afoot to make it mandatory. If in your informed opinion your daughter should get vaccinated against HPV, well then I have very little to say about it. But if I were to counsel an individual patient in the context of my understanding of the data, I would be very upfront with my view.

To me the fact that there are such heated debates about this stuff is a testament exactly to how NOT straightforward our science is. I do understand that as a researcher I can afford a certain amount of analysis paralysis that is unacceptable at the bedside. However, I think we (and the press) do a disservice to the patients, to ourselves and the science if we are not upfront about just how uncertain much of what we think we know is. I could not have said it better than this story about Dr. Devereaux's presentation at the recent ASA meeting did here:

It would be nice if we could all agree that science is not static, but rather progresses and regresses. We learn, and then find out that some of what we thought we had learned was wrong, and set about using that information to seek the next level of truth. Repeat, ad infinitum. Personally, I’d love it if my doctors couched every bit of advice with, “Here’s what we think we know today.”

But I suspect that wouldn’t sit well with many patients, who want certainty (as if there is such a thing). And it especially seems like a difficult proposition in our contentious society, where anti-science nay-sayers like to jump on contradictory findings to challenge the basic value of science overall.

That is exactly NOT what I am trying to do. I am merely reflecting on many of the issues that threaten the validity of what we think we know. I am confident that disclosure and transparency not only lead to better science, but they also lead to science that can withstand the test of nay-sayers.

I have come to the end. I am not sure that I have addressed each and every one of the criticisms, though I hope that I have addressed the majority; I am sure you will point out what I missed. A couple of things about comments: Tomorrow I will be in the air most of the day and may not have the opportunity to sign on to approve comments. So, please, if your comment does not go up until Wednesday, do not think that it has been rejected. Also, I really would like to keep it civil and, though I did not apply this rule today, I will not accept any overt insults or name calling from either side of the debate.

I am sincerely looking forward to continuing this discussion!