Early radical prostatectomy trial: Does it mean what you think it means?

Another study this week added to the controversy about early prostate cancer treatment. The press, as usual, stopped at citing the conclusion: Early prostatectomy does not reduce all-cause mortality. But the really interesting stuff is buried in the paper. Let's deconstruct.

This was a randomized controlled trial of early radical prostatectomy versus observation. The study was done mostly within the Veterans' Affairs system and took 8 years to enroll a little over 700 men. This alone should give us pause. Figure 1 of the paper gives the breakdown of the enrollment process: 5,023 men were eligible for the study, yet 4,292 declined participation, leaving 731 (15% of those who were eligible) to participate. This is a problem, since there is no way of knowing whether these 731 men are actually representative of the 5,023 that were eligible. Perhaps there was something unusual about them that made them and their physicians agree to enroll in this trial. Perhaps they were generally sicker than those who declined and were apprehensive about the prospect of observation. Or perhaps it was the opposite, and they felt confident in either treatment. We can make up all kinds of stories about those who did and those who did not agree to participate, but the reality is that we just don't know. This creates a problem with the generalizability of the data, raising the question of who are the patients that these data actually apply to.

The next issue was what might be called "protocol violation," though I don't believe the investigators actually called it that. Here is what I mean. 364 men were randomized to the prostatectomy group, and of them only 281 actually underwent a prostatectomy, leaving nearly one-quarter of the group free of the main exposure of interest. Similarly, of the 367 men randomized to observation, 36 (10%) underwent a radical prostatectomy. We might call this inadvertent cross-over, which does tend to happen in RCTs, but needs to be minimized in order to get at the real answer. What this type of cross-over does is, as is pretty intuitively obvious, blend the groups' differences in exposure, resulting in a smaller difference in the outcome, if there is in fact a difference. So, when you don't get a difference, as happened in this trial, you don't know if it is because of these protocol violations or because these treatments are essentially equivalent.

And indeed, the study results indicated that there is really no difference between the two approaches in terms of the primary endpoint (all-cause mortality over a substantially long follow-up period was 47% in the prostatectomy and 50% in the control groups [hazard ratio 0.88, 95% confidence interval 0.71 to 1.08, p=0.22]). This means that the 12% relative difference in this outcome between the groups was more likely due to chance than to any benefit of the surgery. "But how can cancer surgery impact all-cause mortality?" you say. "It only claims to alter what happens to the cancer, no?" Well, yes that is true. However, can you really call a treatment like that successful if all it does is give you the opportunity to die of something else within the same period of time? I thought not. And anyway, looking at the prostate cancer mortality, there really was no difference there either: 5.8% attributable mortality in surgery group compared to 8.4% in the observation group (hazard ratio 0.63, 95% confidence interval 0.36 to 1.09, p=0.09).  

The editorial accompanying this study raised some very interesting points (thanks to Dr. Bradley Flansbaum for pointing me to it). He and I both puzzled over this one particularly unclear statement:

...only 15% of the deaths were attributed to prostate cancer or its treatment. Although overall mortality is an appealing end point, in this context, the majority of end points would be noninformative for the comparison of interest. The expectation of a 25% relative reduction in mortality when 85% of the events are noninformative implies an enormous treatment effect with respect to the informative end points.

Huh? What does "noninformative" mean in this context? After thinking about it quite a bit, I came to the conclusion that the editorialists are saying that, since prostate cancer caused such a small proportion of all deaths, one cannot expect this treatment to impact all-cause mortality (certainly not the 25% relative reduction that the investigators targeted), the majority of the causes being non-prostate cancer related. Yeah, well, but then see my statement above about the problematic aspects of disease-specific mortality as an outcome measure.

The editorial authors did have a valid point, though, when it came to evaluating the precision of the effects. Directionally, there certainly seemed to be a reduction in both all-cause and prostate cancer mortality in the group randomized to surgery. On the other hand, the confidence intervals both crossed unity (I have an in-depth discussion of this in the book). On the third hand (erp!) the portion of the 95% CI below 1.0 was far greater than that above 1.0. This may imply that with a study that could have achieved greater precision (that is, narrower confidence intervals) we might have gotten a statistical difference between the groups. But to get at higher precision we would have needed either 1) a larger sample size (which the investigators were unable to obtain even over an 8-year enrollment period), or 2) fewer treatment cross-overs (which is clearly a difficult proposition, even in the context of a RCT), or 3) both. On the other hand (the fourth?), the 3% absolute reduction in all-cause mortality amounts to the number needed to treat of roughly 33, which may be clinically acceptable.

So what does this study tell us? Not a whole lot, unfortunately. It throws an additional pinch of confusion into the cauldron already boiling over with contradiction and uncertainty. Will we ever get the definitive answer to the question raised in this work? I doubt it, given the obvious difficulties implementing this RCT.  

                  

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Peeling the cabbage of "works" in treatment interventions

What exactly does it mean when we say that a treatment works? Do we mean the same thing for all treatments? Are there different ways of assessing whether and how well a treatment works? I am sure you've guessed that I wouldn't be asking this question if the answer were simple. And indeed, the answer is "it depends."

What I am talking about is examining outcomes. I did a post a couple of years ago here, where I use the following quote from a Pharma scientist:

"The vast majority of drugs - more than 90 per cent - only work in 30 or 50 per cent of the people," Dr Roses said. "I wouldn't say that most drugs don't work. I would say that most drugs work in 30 to 50 per cent of people. Drugs out there on the market work, but they don't work in everybody."

Here is that word "work" again. What does this mean? well, let's take such common condition as heart disease. What does heart disease do to a person? Well, it can do many things, including give him/her such symptoms as a chest pain, shortness of breath, dizziness and palpitations, to name a few. These symptoms may have at least two sets of implications: 1) they are bothersome to the individual, and in this way may impair his/her enjoyment of life, and 2) they may signal either a present or a future risk of a heart attack. Why are heart attacks important? Well, they are important because one may kill the person who is having it, or one (or several) may weaken the heart to the point of a substantial disability and thus a deterioration in the quality of life. So, there certainly seems to be a good rationale to prevent heart disease either from happening in the first place or from at least worsening when it's already established.


Now, what's available to us to prevent heart disease? Well, some think that lowering one's cholesterol is a good thing. OK, let's go with that. What is the sign that the statins (cholesterol-lowering drugs) "work"? What would it look like if it was about lowering the cholesterol? Say, your total cholesterol is 240. You go on a statin and in 6 months your total cholesterol is 238. Your cholesterol was lowered, it worked! Well, yes, but if you are asking what this 2-point drop really accomplishes, you are beginning to understand the meaning of "work." So, just intuitively we can say that there needs to be a certain, perhaps "clinically significant," drop in the total cholesterol in order for us to say that the drug "worked." 


Great! Now we are sidling up to the real issue: What constitutes a "clinically significant" drop in cholesterol? Is it some arbitrary number that looks high enough? Probably not. How about some drop that correlates to a drop in the risk of the actual condition we are trying to impact, heart disease? Say, a 40-point drop, or getting to below 200, may be the right threshold for the "works" judgment. Ah, but there is yet another question to ask: How often does this type of a drop lead to a reduction in heart disease? Is it always (not likely), or is it the majority of the time (rarely) or at least some of the time (most likely in clinical medicine)? And what portion of that time do we consider satisfactory -- 60%? 40%? 20%? 2%? 


Let me bring just one more layer into this discussion. Many people walk with heart disease and don't know that they have it. Some of these people are destined to have a heart attack and/or die from it. Many others are likely to die from something else before they ever experience any symptoms or signs of their heart disease. This raises the question of whether the statins' ability to reduce cholesterol and hence reduce the risk of heart disease is enough to say that the drugs "work." Perhaps "work" means that by lowering cholesterol (say in the majority of those who take it) they reduce the risk of hear disease in some proportion of those who are at risk for it, and among that proportion whose risk is reduced they also reduce the risk of a heart attack in a few, and of death in even fewer. 


So, to sum up, "works" is a loaded term. For the case we are discussing, there is what I call a "dwindle" effect, where the main outcome, cholesterol lowering, is likely to show a somewhat robust result. On the other hand, this (surrogate) outcome itself is not all that interesting when divorced from what we really care about -- symptoms, heart attacks and death. And I haven't even gone into the side of the equation where the patient gets to decide what "work" means for him/herself. The layers of the possible "works" are a cabbage that we all need to peel when discussing treatment plans with our clinicians and when reading news about new technologies.                     
 

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

How to game mortality data

There is a great illustration in this BMJ article of what I discuss in Chapter 2 of my book: the type of mortality that matters. In the figure below from the paper, note that, as the new diagnoses of each of the cancers rise (green lines), the attendant "Deaths" (red lines) stay unchanged. If you look at the Y-axis of each graph, it tells us that the unit of measurement is "Rate per 100,000 people." So the red lines represent population mortality.

"Population mortality" means that the denominator for this value is all people in the population who are at risk for the disease in question. This means that for prostate cancer, for example, we include only those men who have a prostate and exclude all women and men who have had a prostatectomy. Population mortality stands in contradistinction to case fatality. The latter is defined as deaths among all the people diagnosed with the disease. So, for prostate cancer, case fatality would be deaths among all men who have been diagnosed with prostate cancer.

It is not difficult to see how case fatality is a somewhat circular, even self-referential, measure of our diagnostic prowess, but says very little about how well we are doing with disease treatment. If we have tests that are capable of picking up the most minute of diseases, those that are not likely to cause death in the first place, then the denominator becomes inflated with this noise, while the numerator, the actual fatalities, does not change. This leads of course to an apparent reduction in deaths from the disease, but a reduction that is an artifact of overdiagnosis. Population mortality, on the other hand, cannot be gamed this way, as you can see in the figure above. This is the only mortality that gives us honest feedback without a bias about how we are doing with our early detection and other interventions. In the case of the cancers in the figure, the answer is "not so well."

You can learn more about this in Chapter 2, where I even have a figure that illustrates this critical distinction.        

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Three wrong reasons to dismiss the azithromycin data

There is a lively discussion over at KevinMD about the azithromycin study in the New England Journal of Medicine, which I blogged about here. A certain ethos has emerged in the comments that bears unpacking. There are 3 distinct points that I am feeling unsetlled by:

#1. As a retrospective cohort study it should be ignored.
This is an intellectually indefensible position in my mind -- it is a study that uses a set of methods developed and validated for this type of data. Yes, it's complex; yes, it's messy; no, it's not to be ignored. The discipline of pharmacoepidemiology relies heavily on observational data. To expect anything more is to indulge in misapprehensions that a). it is feasible to run a RCT to detect such rare signals, and 2). that a RCT like that would give us a definitive answer.

#2. They really had to add a lot of zeros to the denominator to make the numerator seem impressive.
This is a baseless accusation, since 1 million prescriptions is not that difficult to generate. Z-pak has been on the market for over a decade, and according to this article, last year 55 million prescriptions for azithro were handed out in the US. So, just a back-of-the-envelope calculation for excess deaths per year at this rate is well over 2,000. And this is just in one year! So, as a safety signal this is not something to be trivialized.

#3. A concern that this information will keep patients from doctors' offices and delay needed treatment.   
I find this to be rather a hollow concern (though I am sure that the person putting it forward believes it wholeheartedly). As another commenter pointed out, the overuse and misuse of antibiotics is completely out of control! And yes, cardiac deaths from azithromycin are but a small part of the issue, where the elephant in the room is the evolution of resistance. It is not just these latest data that should keep patients as far away as possible from unnecessary healthcare encounters, seeing as how these encounters are the third leading cause of death in the US. Why aren't we worried that this entire monster is keeping patients away? And quite frankly, why isn't it?

So, all in all, I am very glad that Rob Lamberts chose to blog the study, and the discussion has been worthwhile. The comments have really confirmed for me that it is not only the lay public, but also healthcare professionals, who have a hard time interpreting data. And when a study is somewhat challenging, it is generally easier to let our cognitive biases run amok.      

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Septic shock doesn't need my rescue bias

Rescue bias and auxiliary hypothesis bias are tempting distractions when it comes to dealing with data that are counter to our preconceived notions. In general, the role of our cognitive biases in all kinds of discourse, including clinical and scientific, is under appreciated. For this reason I devoted fully two chapters of my book to cognitive biases.

Which makes it even more frightening that, as I read this NEJM paper on the failed drotrecogin PROWESS-SHOCK study, I am still looking for reasons why it failed, other than that just doesn't work. After so many years of trials and tribulations with Xigris, and hopes that this lone therapy ever to have been approved for treating this deadly disease, I am having a hard time letting go.

Yet the rial was meticulous, as all trials designed and overseen by B. Taylor Thompson are -- he is just a brilliant clinical researcher that we all need to learn from. The design considered all the right issues a priori -- multiple interim looks at the data, a possible need to increase the power, heterogeneous treatment effect, and others. Although there were a few numerical imbalances between the treatment and the placebo groups -- blood cultures were positive 4% more frequently and the offending pathogen overall was 5% more likely to be identified in the Xigris than in the placebo group -- I have to accept that these are not the reasons for the observed failure to improve either the 28-day or the 90-day mortality.

I have seen and even blogged about these data before -- the press release had some of them, but, more importantly, Taylor presented them at the Society of Critical Care Medicine meeting back in January. So, this is nothing new. Yet reading all of the details in the pages of the NEJM brings back that pre-conscious cognitive wall that I have to climb over in order to get to reality. And the reality is that the drug just did not work.

But the reality is also that sepsis patients have a better chance of surviving this deadly assault than they did 10 years ago. In the original PROWESS study 28-day mortality in the placebo group was 31%, and these were all kinds of sepsis patients. In the current study, among most severe sepsis patients, those with septic shock, the 28-day mortality was on the order of 25% in both arms -- that's a staggering reduction in this most ill septic population! We really need to appreciate this. In part this trend may be due to some evolution of the disease-host interaction. But in part it has to be because of the concerted effort to understand sepsis better, to study various treatment options, and, most importantly, to implement these learnings at the bedside. In no small part we owe these advances in sepsis care to the short life of drotrecogin alpha.

The data are clearly anticlimactic, and for this reason the current paper has no sex appeal -- just look at the (lack of) press coverage about it. Yet, this is a clear example of countering the traditional publication bias: Here is a manufacturer-funded negative study published in a premier peer-reviewed journal. I realize that it was always a high-profile undertaking, but let's pause for a moment to enjoy this small victory for those who have railed against the spread of biased information in the medical literature.

I don't know if and when another therapy will emerge courageous enough to brave the rough waters of sepsis and septic shock. But one thing is clear: Xigris has served out its purpose, and no amount of rescue bias from me will save it. Rest in peace.    

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Why I have a propensity to believe the azithromycin data

Before I take off for the Health Foo Camp in Cambridge, MA, where I will be rubbing elbows with such luminaries as Susannah Fox, Ted Eytan, e-Patient Dave deBronkart, Regina Holliday, Nick Christakis and Nancy Etcoff, I thought I'd say a few words about one of my favorite topics: antibiotics. In particular I want to talk about azithromycin. A lot has been said about it this week already in the wake of the NEJM study (alas, the full paper is behind a pay wall) indicating an increase in the risk of cardiac death among patients exposed to this drug compared to those either not getting antibiotics or those receiving a different class of antimicrobials. And a lot more will be said in the future as the FDA reviews this risk. This is not specifically what I wanted to talk about. I wanted to focus on the methods.

The study was done meticulously, as far as I can see. It was a large Medicaid database from Tennessee (this may present a generalizability problem, of course, though I cannot think of a specific reason why it would). The study design was a retrospective cohort, which, as we already know, sets the study up for all kinds of biases. So, why should we believe anything the study showed? And particularly given many people's distaste for invoking causality from observational data?

There are at least 2 reasons why we should pay attention to these results. The first is that we are talking about the ultimate harm: death. When it comes to harm, my philosophical approach leans in the direction of caution. What this means scientifically is that I accept a much lower threshold for the certainty that the data convey than when it comes to evidence of benefit. This is an extension of the precautionary principle, where the burden of proof of safety now lies with the drug.

But there is a second, possibly more important reason that I am inclined to believe the data. The reason is called succinctly "propensity scoring." This is the technique that the investigators used to adjust away as much as feasible the possibility that factors other than the exposure to the drug caused the observed effect. In my book I briefly discuss propensity scoring in Chapter 21. And here is what I say:

Propensity scoring is gaining popularity as an adjustment method in the medical literature. A propensity score is essentially a number, usually derived from a regression analysis, describing the propensity of each subject for a particular exposure. So, in terms of smoking, we can create a propensity score based on other common characteristics that predict smoking. We take advantage of the presence of some of these characteristics also in people who are non-smokers to yield a similar propensity score in the absence of this exposure. In turn, the outcome of interest can be adjusted in several ways for the propensity for smoking. One common way is to match smokers to non-smokers based on the same (or similar) propensity scores and then examine their respective outcomes. This allows us to understand the independent impact of smoking on, say, the development of coronary artery disease. 

And if you are able to access Table 1 of the paper, you will see that their propensity matching was spectacularly successful. So, although it does not eliminate the possibility that something unobserved or unmeasured is causing this increase in deaths, the meticulous methods used lower the probability of this.

One final word about azithromycin. There are data that suggest that macrolides (the class of drugs that includes erythromycin, clarithromycin and azithromycin) are actually associated with improved outcomes in the setting of community-acquired pneumonia, or CAP. This is why these drugs are in the CAP treatment guideline. The point is that again, as in everything, the benefit of using azithromycin in any individual case will have to be weighed against this newly-identified risk.

  

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Fast science: No time for uncertainty

Reading Barbara Ehrenreich's "Bright-Sided" has been liberating in that is has given me permission to let my pessimistic nature out of the closet. Well, it's not exactly that I am pessimistic, but certainly I am not given over to brightness and cheer all the time. My poison is worry. Yes, I am a worrier, in case you had not noticed. So, imagine how satisfying it is for me to find new things to worry about. As if climate change were not enough, lately I started to worry about science.

No, my anxiety about how we do clinical science overall is not new; this blog is overrun with it. However, the new branch of that anxiety relates to something I have termed "fast science." Like fast food it fills us up, but the calories are at best empty and at worst detrimental. What I mean is that science is a process more than it is a result, and this process cannot and should not be microwaved. Don't believe me? Let me give you a couple of instances where slow science may be the answer to our woes.

1. Lies and damned lies
Remember this story in the Atlantic that rattled us with its incendiary message? Researcher John Ioannidis has been making headlines with his assertion that most, if not all, of what we know in medicine is in doubt, given how we do and publish research. And how we do and publish research has everything to do with the speed of "progress." Academic careers are made with positive results, to sell news the media demand positive results, and to respond to this demand academic journals prefer only to publish positive results (this last phenomenon is referred to as "publication bias," and is something Ben Goldacre rails against at length). A further manifestation of this fast science is that "no replicators need apply." I am, of course, referring to an extension of the publications bias, whereby journals are not interested in publishing even a positive study that replicates a previous finding -- this is simply not sexy. Thus, results have to be quick and positive to grab a share of our attention and sell academic prestige, journals and news.

2. Science output to drive business profits
In his book Supercapitalism, Robert Reich describes the growing demand by investors over the last several decades to squeeze ever-growing profits. It is clear that this chase after short-term profits has resulted in job loss in the US through outsourcing, the widening of the economic gap, and even the crash of the world economy following the collapse of the mortgage-backed securities house of cards. Much of the profit can be counted on to come through scientific innovations which may or may not improve our quality of life.
In medicine, where scientific progress is applied to our fragile being, being reasonably sure of our findings seems pretty important. Yet speed is once again the order of the day. I will grant you that speed is of importance in such diseases as advanced cancer, for example, where we may and should accept a level of uncertainty that we would ordinarily run away from in other circumstances. But doesn't it make sense to be much more cautious before broadly accepting an intervention that happens before one gets sick, one that is meant to diagnose either early disease or a precursor to one? Should we not demand slower science before we allow anyone to medicalize such normal events in life as menopause and aging? Should this caution also not apply to screening for diseases that may or may not impact us in the long term, yet the chase could hurt us substantially in the immediate future?
But this is not the way to stimulate the economy or to make a profit. The half-life of a medical device, for example, is less than 1 year. After that a new "improved" version of the device is expected, whether it does or does not improve outcomes. For decades we were told to get screening mammography after the age of 40, only to find out now that the risks of this may well outweigh its benefits for many. The American Lung Association has just endorsed CT screening for lung cancer among current or former heavy smokers, yet the jury on its risk-benefit-uncertainty equation should still be in the thick of deliberations.

3. Science denialism
We hear a lot about how people are turning away from science. The state of Tennessee is about to descend back into the dark ages when superstitions instead of scientific theories dominated the classroom. A strong and largely anti-scientific lobby wants to bury any mention of human-driven climate change; fortunately, it looks like they are not succeeding. The anti-vaccination groups are getting more instead of less vocal following repeated debunking of any link between vaccination and autism. Science denialism is so rampant that there was even a need for a conference on how to address it. What gives?
While blaming everything on fast science alone may be reductionist, fast science in the setting of our growing societal innumeracy is a recipe for disaster, as we are seeing unfold. Our schools have failed spectacularly in their duty to educate kids about the process of science, while at the same time arming them with the "single-right-answer-to-every-question" attitude toward knowledge. This pernicious combination, along with the publication and reporting of sexy science at the expense of the more thorough analytic and introspective approach, seals the impression that the roller coaster of scientific knowledge represents not the very essence of how science should be done, but that science (and scientists) has failed.
Is slow science the answer to this fiasco? Only in part, I am afraid. Without altering fundamentally how we teach science at all levels, it would not be the cure, even if it were possible to execute. No, I am afraid that without teaching what science is, it is not even possible to get it to slow down.

Let me reiterate: the pace of scientific discovery is slow. This does not mean that we need to hide every step of it from view until we get the results that we deem worthy of sharing. On the contrary, I agree with those who think that sharing at the more interim steps can only improve what we do. Yet the innumeracy, fame and fortune are forces that put such free sharing in peril by misrepresenting it as the final answer to everything. And when the answer is changed, which is not only expected, but indeed desired in scientific pursuits, the public opinion punishes science.

Let me end with a quote I read on one of my favorite web sites, Brain Pickings, in a review of the book boldly entitled Ignorance: How It Drives Science:

Are we too enthralled with the answers these days? Are we afraid of questions, especially those that linger too long? We seem to have come to a phase in civilization marked by a voracious appetite for knowledge, in which the growth of information is exponential and, perhaps more important, its availability easier and faster than ever.

[...]There are a lot of facts to be known in order to be a professional anything — lawyer, doctor, engineer, accountant, teacher. But with science there is one important difference. The facts serve mainly to access the ignorance… Scientists don’t concentrate on what they know, which is considerable but minuscule, but rather on what they don’t know…. Science traffics in ignorance, cultivates it, and is driven by it. Mucking about in the unknown is an adventure; doing it for a living is something most scientists consider a privilege. 

So, let's celebrate uncertainty. Let's take time to question, answer and question again. Slow down, take a deep breath, cook a slow meal and think.

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. Thank you for your support!

Coronary CT angiography: More and less

I am still scratching my head over this study that was just published in the NEJM on coronary CT angiography among patients admitted with a suspected coronary syndrome. There are many potentially confusing points about it and how it was reported. The three things that I found most confusing were:
1. The formulation of the null hypothesis
2. The definition of the outcomes
3. The flow of patients through the diagnostic algorithm.
Let's see if we can clear some of that confusion.

The intent of the study was to see how adding the CCTA to the usual diagnostic testing in the ED impacted cardiac mortality or a myocardial infarction within 30 days. The null hypothesis was posed in an interesting way:

The study was powered to test the null hypothesis that the rate of major cardiac events among patients who did not have clinically significant coronary artery disease as assessed by CCTA would exceed 1%. 

This is confusing, and I had to read and reread several times to understand what it meant. I finally realized that what they were saying was that in order to disprove that CCTA is not a useful test (this is the alternative hypothesis) they would have to show that the rates of the primary outcome were not above 1%. Make sense?

The next issue I had trouble with, as I always do in cardiology studies is their choice of endpoints. The primary endpoint was 30-day cardiac death OR MI. This means that anyone who either died of a cardiac cause or had a heart attack within this time period was counted as an event, which would argue against CCTA usefulness if they reached the critical mass of over 1%. Mind you this was limited to those patients whose CCTA did not reveal significant disease. There were some secondary outcomes examined as well, all at the 30-day time point: death, MI, revascularization procedure, and resource utilization. Note these outcomes were not combined, but were examined singly, and the pool of patients for these were all those randomized.

As an aside, cardiology studies frequently use combined outcomes, such as death and MI due to sample size considerations. That is both cardiac death and MI should be rare events in the group examined. When these events are rare, in order to get at their statistical significance exceedingly large sample sizes are needed. For this reason, these trials frequently combine several events, so as to enrich the frequency and commensurately drop the needed sample size.

But here is where it gets a little confusing. Looking at the "Safety" section of the Results, the authors state that no one who received a CCTA had a cardiac death or an MI within the 30-day period. However, 1% of the patients randomized into the CCTA arm did have a MI within 30 days. How can this be? Well, we need to keep in mind that not all those randomized to CCTA (n=908) actually got a CCTA (n=767), and that the majority of those who did have one did NOT have significant coronary disease (n=640). Thus, both the numerator and the denominator for the primary and the secondary outcomes are different.

Then there is this quick sentence in the "Efficiency and Use of Resources" section:

Coronary disease was more likely to be diagnosed in patients in the CCTA group than in patients in the traditional-care group (9.0% vs. 3.5%; difference, 5.6 percentage points; 95% CI, 0 to 11.2).

It is almost an afterthought, but it is important. It is puzzling that the outcomes in the two groups are completely identical, and this is not limited to those in the primary endpoint pool, namely people without significant disease. The question arises about what this means. Since there is no difference in cardiac death or MI, this increase in the diagnostic rate may imply overdiagnosis in the group receiving CCTA. There is a slight increase in the revascularization rates in the CCTA group (3%) over the standard care group (1%). This endpoint is much more subjective that most people realize, as a lot of judgment by the cardiologist and the surgeon goes into the decision. So this endpoint does not rule out overdiagnosis as a possibility. On the other hand, the study was not powered to detect a difference in the secondary outcomes, so it may be that the diagnoses are valid, but we do not have enough events to judge.

One final point of frustration with the study reporting was hitting dead ends in the flow of patients through the respective algorithms. I was, of course, looking for the rates of false positive CCTA tests and their outcomes. Unfortunately, I kept getting stuck at the catheterization and stress test steps, not knowing who exactly went on to have this testing. Since in the CCTA out of the 767 tests 47 were indeterminate and 80 were indicative of moderate-to-severe coronary disease, the 37 catheterizations performed were likely among these patients, though we are not told for sure. Of these catheterizations, 9 were negative for a significant stenosis, but again we are not told how this reflects back to the CCTA results. In fairness it is worth noting that fewer people in the CCTA arm (18%) underwent a follow-up test, compared to the standard care arm (62%). But again, this is just one test replacing another, or even being added on top of others. In addition, more of the former group were discharged from the ED (50%) than among the latter (23%) without being admitted.

So what does it all say to me? Well, CCTA seems to aid in the diagnosis of coronary disease among certain people presenting to the ED with chest pain. Given such a low pre-test probability of significant coronary disease (17%, derived by dividing the negative CCTA [640] by the total CCTA tests [767] and subtracting that from 1), I have to wonder about the performance of the test that is quite sensitive but not that specific (high risk of a false positive in a low-risk population). And even though fewer patients needed to be admitted from the ED in the CCTA group, I wonder if this does not have more to do with the differential availability of the testing rather than with its superiority.

Given the context that I laid out above for the results presented, I would not be rushing to adopt CCTA for all patients who present to the ED with a certain type of chest pain. And after all, though a "black swan" event, catastrophes do occur in follow-up catheterization even when the patient is disease-free. This is a good reason to be careful and circumspect in our quest for primum non nocere.      

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have non-profit status. 
Thank you for your support!

Aspirin or bias?

Update 4PM Eastern, 3/21/12

Wanted to append a couple of thoughts I tweeted earlier about these studies, in case you don't follow me on Twitter or just missed them:

Around 2:30 PM Eastern:

And then closer to 3:00 PM Eastern: 

Thoughts?


There is a fascinating review by Cancer Research UK of the new and old aspirin data with respect to its effects on cancer and cardiovascular complications in the context of a heightened risk for bleeding. The review is full of fabulous information about what we know and the uncertainties that remain, all with practical suggestions at the end, so go there and read it.

But here is what I wanted to highlight in the graph that I am reproducing from a screen shot:

There is something very interesting going on here. Just as the risk of bleeding begins to drop, so does the risk of developing cancer. This could be a complete coincidence, but perhaps not. An alternative explanation is that people who already have cancer, though it may not yet be diagnosed, may be at a higher risk for a bleeding complication. Those who develop a bleeding complication presumably are taken off aspirin. But remember, they may already be harboring a cancer that will rear its head in the near future. But what about those who do not bleed and therefore are able to tolerate aspirin for a longer time? They also seem to have a drop in their risk of incident cancer. But of course this may have nothing to do with aspirin's preventing cancer, so much as with its ability to unmask a cancer that is already present and essentially weed them out from the future risk pool for cancer development. And when you weed out those at a higher risk for clinical cancer, by definition you have a group with a lower than standard risk, creating the potential for a selection bias. Make sense?

Conversely, the risk of a cardiac event starts to increase roughly at the same time as the risks for cancer and bleeding begin to drop. This to me suggests confirmation that aspirin may prevent cardiovascular events early in the course of taking it. Furthermore, given my hypothesis above about aspirin's weeding out those with an early cancer, perhaps its cardiovascular impact is for some reason limited to those with an early cancer or with another reason for aspirin-induced bleeding.

All-in-all the data do not convince me to start taking aspirin -- I am still at odds with Dr. Agus on that. The selection bias that I described above may very well mean that aspirin's role is not as a cancer prevention, but more likely as a sort of a stress test for those with a subclinical cancer. So we are left again with the the chicken-and-egg question. But isn't that, after all, what makes science exciting?

Would love to know what others think -- does this make sense? Are there other possible explanations?      

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. 


 Thank you for your support!

How medicine is like quantum physics

When a patients goes to his doctor to get fixed a pivotal triad of presentation-diagnosis-treatment ensues. The three steps are as follows:
1. History, physical examination and a differential diagnosis
When the patient shows up with a complaint, a constellation of symptoms and signs, a good clinician collects this information and funnels it through a mesh of possibilities, ruling certain conditions in and others out to derive the initial differential diagnosis.
2. Diagnostic testing
Having gone through the exercise in step 1, the practitioner then decides on appropriate diagnostic testing in order to narrow down further the possible reasons for the person's state.
3. Treatment
Finally, having reviewed all the data, the clinician makes the therapeutic choice.

These three steps seem dead simple, and we have all experienced them, either as patients or clinicians or both. Yet the cause for the current catastrophic state of our healthcare system lies within the brackets of each of these three little domains.

The cause is our failure to acknowledge the vast universe of uncertainty dotted sparsely with the galaxies of definiteness, all shrouded in false confidence. And while the cause and the way to address it are conceptually simple, the remedy is not easy to implement. But I am jumping ahead; first I have to convince you that I have indeed discovered the cause of this ruin.

Let's examine what goes on in step 1, the compilation of history and physical to generate a differential diagnosis. This is usually an implicit process that takes place mostly at a subconscious level, where the mind makes connections between the current patient and what the clinician has learned and experienced. What does that mean? It means that the clinician, within the constraints of time and the incredible shrinking appointment, has to listen, examine, elicit and put together all of the data in such a way as to cram them into a few little diagnostic boxes, many of which contain much more material than a human brain can hold all at once, even if that brain is at the right tail of human cognition (or not). What overtakes at this step is a bunch of heuristics and biases. Have we talked about those enough here? Just to review, heuristics are mental shortcuts that can serve us well, but can also lead us astray, particularly under conditions of extreme uncertainty, as in a healthcare encounter. If you want to learn more about this, read Kahneman, Slovic and Tversky's opus "Judgement under uncertainty: Heuristics and biases." As for cognitive biases, I will not belabor them, as there is enough material about them on this web site and elsewhere to overload a spaceship.

The picture that emerges at this step is one of fragments of information gathered being fit into fragments of studies and experience, stirred with mental shortcuts and poured into a bunch of baking tins shaped like specific diagnoses. Is there any room in this process for assigning objective probabilities to any of these events? Well, there is an illusion of doing so, but even this step is done by feel, rather than by computation. So while there is some awareness of a probabilistic hierarchy, it is more chaos than science. Given this picture, it's a wonder it actually works as well as it does, don't you think?

The next step in this recipe is the diagnostic workup. What ensues here is utter Wild West, particularly as new technologies are adopted at breakneck speed without any thought to the interpretation of data that they are capable of spitting out. Here the confusion of the first step gets magnified exponentially, just as it seduces us into further illusion of certainty. The uncertainties in arriving at the differential get multiplied by the imperfections of diagnostic tests to give the encounter truly quantum properties: you may know the results or you may know the patient, but you may not know both at the same time. What I mean is what I have always said on this blog: no test is perfect, and because of this simple truth, unless we know the pre-test probability of the disease in a particular patient, as well as the characteristics of the test, we have no idea about the context of these results. Taking them at face value, as we know, is a grave error.

What follows these results is frequently more diagnostic hit-or-misses, as the likelihood of harm and escalating expenditures without any added value rises. Then comes the treatment, with its many uncertainties and the potential for adverse events, and what are we left with? A pile of costly and deadly steaming manure. So, what's a doc to do?

I think that there is a very simple solution to this, and in its simplicity it will be incredibly hard to implement: education. And I don't just mean medical education. Everything that I have talked about in this post echoes back to the concept of probability. In the secondary education, at least as I remember it, probability is left to Advanced Math. By the time a student becomes eligible to take this course, she has been made to feel that she does not have the facility for math, and that, furthermore, math is boring and useless. So, while my friends in education may have a much better idea of what percentage of kids leave high school having been exposed to some probability, my guess is that it is woefully small. And those that do get exposure to it walk out of class perfectly able to bet on a game of craps or a horse race, but no clue how to apply these ideas to the world they live in.

And so those who progress into healthcare and those who don't have heard the word "probability," but cannot quite understand how it impacts them beyond their chance of winning the lottery. And unfortunately, I have to tell you that, if I relied on what I learned in medical school about probability, well, let's just say it is highly improbable that we would be having this discussion right now. This is why I do now and will for the foreseeable future harp on all of these probabilities, so that when you are faced with your own medical decisions, you will at least know the right questions to ask.

I know I need to wrap this up -- I saw that yawn! Here is the bottom line. First, we need to acknowledge the colossal uncertainties in medicine. Once we have done so, we need to understand that such uncertainties require a probabilistic approach in order to optimize care. Finally, such probabilistic approach has to be taught early and often. All of us, clinicians and patients alike, are responsible for creating this monster that we call healthcare in the 21st century. We will not train it to behave by adding more parts. The only way to train it is to train our brains to be much more critical and to engage in a conversation about probabilities. Without this shift a constructive change in how medicine is done in this country is, well, improbable.              

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. 

Thank you for your support!

A good week

So, it has been a great week here at Healthcare, etc, and I am grateful.  Here are the highlights:

1. I did a guest post on plagiarism at Retraction Watch, and not only did it generate a fantastic conversation there, but it also brought other stories of plagiarism to us right here on this site. The story is not finished, and I will update when I can.

2. Gary Schwitzer of Health News Review cited 2 posts from Healthcare, etc. on his extremely popular and well-respected site here and here. The corresponding stories that he referenced were "Unpacking the meat data" (which incidentally is the most popular post of all time on this blog) and "PSA screening: Does it or doesn't it" (a niche post with a lively discussion in the comments).

3. The crowning glory of the week was being cited by Mark Bittman in his New York Times column on May 16th, which drove hordes of readers to Healthcare, etc.

4. Finally, my book is progressing through editing and design, and it is all beginning to seem more real. It feels like I am realizing my dream of bringing a critical approach to evaluating medical evidence to all who are interested. If you want to get news about it in your inbox, please sign up here.

In the next few weeks you can expect to hear about my experience attending and presenting at the Ignite Boston conference at MIT on March 29, as well as coverage from TEDMED 2012 in DC, which I am thankful to be attending as a Frontline scholar.

Thanks again to everyone for visiting, tweeting, linking and sharing in all other ways. Looking forward to seeing you here again.

The flu and likelihood ratios

An interesting study was just published in the Annals of Internal Medicine. It was a meta-analysis of rapid influenza diagnostic tests (RIDT) and their characteristics. Since we have been spending so much time talking about test characteristics, this study provides a nice opportunity to discuss another way of looking at the values of positive and negative tests. This is going to be a fairly short post, since I simply want to discuss these additional tools.

In the current study the investigators evaluated how well these RIDTs predicted the disease. As we have discussed in multiple places on this blog (here, here and here, to name a few), it matters whether the test is used for screening or diagnosis. In this case, the testing was done in symptomatic populations, so for diagnostic reasons. The authors report that there was quite a bit of heterogeneity in their findings, but the ultimate result is reported as a positive (34.5) and negative (0.38) likelihood ratios. What are they and how do we interpret them?

A positive likelihood ratio, or LR+ is the ratio between sensitivity and 1-specificity (LR+=[sensitivity]/[1-specificity]). Sensitivity is the proportion of patients with the disease who are identified as having the disease, or true positives (TP), and specificity is the proportion of persons without the disease who are identified as not having the disease, or true negatives (TN). The opposite of TN, 1-TN, is the false positives (FP). So, the LR+ equates to the TP/FP, or the odds that a positive indicates true disease. In the current study it is 34, meaning that the odds are 34 to 1 that a positive test indicates the presence of the disease. Another way of putting it is that of the 35 total positive test results, 34 (97.1%) represent true disease. This is essentially equivalent to the positive predictive value (PPV).

Now, let's examine the negative likelihood ratio, or LR-. This is defined as the ratio between the opposite of sensitivity (1-sensitivity) and specificity (LR-=[1-sensitivity]/specificity]). 1-sensitivity is the proportion that are false negative (FN), while the specificity is the proportion of persons without the disease who are identified as such. In the study this LR- was 0.38, meaning that the odds that a negative result truly indicates the absence of disease are about 1 to 2 (0.36:1), or not so great. In other words, out of the total of 3 negative tests, 2 are truly negative, while 1 is a false negative, giving us the negative predictive value (NPV) of about 65% (actually it is 1-0.36=0.64, or 64%).

So, there you have it. The clinical take-away, as the authors noted, is that these RIDTs are good at ruling in the flu, but not at ruling it out. In other words, the problem here is the opposite of what we discussed in all those previous test, or the rate of false negatives. And this makes sense, given that the pre-test probability is reasonably enriched in populations with symptoms, in addition to the relatively poor sensitivities of these technologies.  

  

Some thoughts on denominators

I decided to re-post this piece dating back from May 2009 in the wake of the recent reports about bird flu. As it turns out, this flu may not be quite as deadly as we had previously thought. And, yes, this revelation is all about the denominator.

Let's face it: denominators keep numbers (and people reporting them) honest. Imagine if I said that there were 3,352 cases of a never-before-seen strain of flu in the US. To be sure, 3,352 cases is a large enough number to send us rushing to buy a respirator mask! But what if I put it slightly differently and said that out of the population of roughly 300,000,000 individuals, 3,352 have contracted this strain of flu. I think this makes things a little different, since it means that the risk of contracting this flu to date is about 1 in 100,000, a fairly low number as risks go. Now, I am going to give you another number -- 86. This represents the number of the novel H1N1 flu-related deaths in Mexico reported on April 25, 2009, by the health minister of Mexico, and at that time this flu had been thought to have sickened 1,400 people. This gives us the risk of death with the flu of roughly 6%, a very high risk indeed! Well, that was then. Now that we have all steadied our pulses, and the health authorities have gone back and done some testing, as of yesterday Mexico had confirmed 2,059, cases with 56 fatalities, equating to a 2.7% risk of dying with the disease. Still a high number, to be sure, but lower than what was though before.

In the US, we have had 3 fatalities among 3,352 cases reported as of yesterday, yielding the risk of death from H1N1 in this country of about 1 in 1,000. But, of course, the denominator of 3,352 persons represent only those who sought medical attention and got tested, so probably it is an underestimate of the true burden of this strain of flu, and necessarily also an over-estimate of its attendant mortality. Now, apply this to the situation in Mexico, and it's likely that the risk of death from H1N1 is also lower than what we have observed precisely due to the under-estimation of the denominator. 

So how could we get a true estimate of the numbers of people afflicted with the H1N1 influenza? Well, we could screen absolutely everyone (or more likely a large and representative group of individuals). Then what? Do we treat them all with anti-virals? Do we observe them? Since the Centers for Disease Control and Prevention recommends testing only severe cases and treating only persons at a high risk for complications, universal testing does not seem like a practical approach. So, the bottom line is that we are not likely ever to get at the correct denominator for the risk of dying with this disease, and any number that we get is likely to be an over-estimate of the true risk.

So, what are the lessons here? First, don't let anyone get away with only giving you the numerator, as that is not even a half of the story. Second, even when the denominator appears known, be skeptical -- does it really represent the entire pool of cases that are at risk for the event that the numerator describes? The likely answer will most of the time be "no". Clearly, it is the denominator that is the key to being an educated consumer of health information.

In medicine, beware of what seems too good to be true

Update 2/17/12:
A reader brought to my attention (thanks!) a very slight inaccuracy in the first table below, which I have corrected. I did the calculations in Excel, which, as you may know, likes to round numbers. 

File this under "misleading." Here is the story:

What's the Latest Development? 

A California start up has developed a breath test that can diagnose lung cancer with a 83 percent accuracy and distinguish between different types of the disease. The procedures which currently exist to test for lung cancer, which is the leading cause of cancer deaths worldwide, result in too many false positives, meaning unnecessary biopsies and radiation imaging. The new devices works by drawing breath "through a series of filters to dry it out and remove bacteria, then [carries it] over an array of sensors."  

What's the Big Idea? 

The company is now testing a version of the machine 1,000 times more accurate than its latest model, which could increase the accuracy of diagnoses to 90 percent, the level likely needed to take the device to market. Because the machine is not specific to a particular group of chemicals, the breath tester could, in principle, test for any disease that has a metabolic breath signature, for example, tuberculosis. "A breath signature could give a snapshot of overall health," says the company's founder, Paul Rhodes. 

Am I just being a luddite by not getting, well, breathless about this? I'll just lay out my argument, and you can be the judge.

There is not doubt that lung cancer is a devastating disease, and we have not done a great job reducing its burden or the associated mortality. However, there are several issues with what is implied above, and some of the assumptions are unclear. First, what does "accuracy" mean? In the world of epidemiology it refers to how well the test identifies true positives and true negatives. If that is in fact what the story means, then 83% may not be bad; we'll regroup on that point at the end late in this post. This brings me to my second point: what is the gold standard that the test is being measured against? In other words, what is it that has the 100% accuracy in lung cancer detection? Is it a chest X-ray, a CT scan, a biopsy, what?

The SEER database, the most rigorous source of cancer statistics in the US, classifies tissue diagnosis as the highest evidence of cancer. However, in some cases a clinical diagnosis is acceptable. The inference of cancer when no tissue is examined is possible when weighing patient risk factors and the behavior of the tumor. So, you see where I am going here? The gold standard is tissue or tumor behavior in a specific patient. Is that what this technology is being measured against? We need to know. And here is another consideration. What if the tissue provides a cancer diagnosis, but the cancer is not likely to become a problem, like in the prostate cancer story, for example?

But all of these issues are but a prelude to what is the real problem with a technology like the one described: the predictive value of a positive test. The story even alludes to this, pointing the finger at other current-day technologies and their rates of false positivity, and away from itself. Yet, in fact, this is the crux of the matter for all diagnostics. Let me show you what I mean.

The incidence of lung cancer in the US is on the order of 60 cases per 100,000 population. Now, let us give this test a huge break and say that it yields (consistently) 99% sensitivity (identifies patients with cancer when cancer is really present) and 99% specificity (identifies patients without cancer when they really do not have cancer). What will this look like numerically given the incidence above if we test 100,000 people?

	Cancer present	Cancer absent	Total
Test +	59	999	1,0589
Test -	1	98,941	98,9421
Total	60	99,940	100,000

If we add up all the "wrong" test results, the false negative (n=1) and the false positives (n=999), we arrive at a 1% "inaccuracy" rate, or 99% accuracy. But what is hiding behind this 99% accuracy is the fact that of all those people with a positive test only a handful, a paltry 6%, actually have cancer. And what does this mean to the other 94%? Additional testing, a lot of it invasive. And what does this testing mean for the healthcare system? You connect the dots.

Let's explore a slightly different scenario. Let us assume that there is a population of patients whose risk for developing lung cancer is 10 times higher than the population average. Let us say that their incidence is 600 cases per 100,000 population. Let us perform the same calculation assigning this same bionic accuracy to the test:

	Cancer present	Cancer absent	Total
Test +	594	994	1,588
Test -	6	98,406	98,412
Total	600	99,400	100,000

The accuracy remains at 99%, but the value of the positive test rises to 37%. Still, 63% of all people testing positive for cancer will go on to unnecessary testing. And imagine the numbers when we try to screen millions of people, rather than just 100,000.

Let us do just one final calculation. Let us reflect the data back to the test in question, where the article claims that the accuracy of the next version of the technology will be 90%. Assuming a high risk population (600 cases per 100,000 population), what does a positive result mean?

	Cancer present	Cancer absent	Total
Test +	540	9,940	10,480
Test -	60	89,460	89,520
Total	600	99,400	100,000

From this table, the accuracy is indeed 90%, concealing the very low value of a positive test of 5%! This means that of the people testing positive for lung cancer with this technology, 95% will be false positives! What is most startling is that to arrive at the same mediocre 37% value for a positive test that we saw above in this population, we would need a population where cancer incidence is a whopping 6,000 per 100,000, or 6%!

I do not want to belabor this issue any further. Screening for disease that is not yet a clinical problem is fraught with many problems, and manufacturers need to be aware of these logic pitfalls. What I have shown you here is that even when the "accuracy" of a test is exquisitely (almost impossibly) high, it is the pre-test probability of, or the patient's risk for the disease that is the overwhelming driver of false positives. Therefore, I give you this conclusion: beware of tests that sound too good to be true -- most of the time they are.

h/t to @gingerly_onward for the story link