Aspirin or bias?

Update 4PM Eastern, 3/21/12

Wanted to append a couple of thoughts I tweeted earlier about these studies, in case you don't follow me on Twitter or just missed them:

Around 2:30 PM Eastern:

And then closer to 3:00 PM Eastern: 

Thoughts?


There is a fascinating review by Cancer Research UK of the new and old aspirin data with respect to its effects on cancer and cardiovascular complications in the context of a heightened risk for bleeding. The review is full of fabulous information about what we know and the uncertainties that remain, all with practical suggestions at the end, so go there and read it.

But here is what I wanted to highlight in the graph that I am reproducing from a screen shot:

There is something very interesting going on here. Just as the risk of bleeding begins to drop, so does the risk of developing cancer. This could be a complete coincidence, but perhaps not. An alternative explanation is that people who already have cancer, though it may not yet be diagnosed, may be at a higher risk for a bleeding complication. Those who develop a bleeding complication presumably are taken off aspirin. But remember, they may already be harboring a cancer that will rear its head in the near future. But what about those who do not bleed and therefore are able to tolerate aspirin for a longer time? They also seem to have a drop in their risk of incident cancer. But of course this may have nothing to do with aspirin's preventing cancer, so much as with its ability to unmask a cancer that is already present and essentially weed them out from the future risk pool for cancer development. And when you weed out those at a higher risk for clinical cancer, by definition you have a group with a lower than standard risk, creating the potential for a selection bias. Make sense?

Conversely, the risk of a cardiac event starts to increase roughly at the same time as the risks for cancer and bleeding begin to drop. This to me suggests confirmation that aspirin may prevent cardiovascular events early in the course of taking it. Furthermore, given my hypothesis above about aspirin's weeding out those with an early cancer, perhaps its cardiovascular impact is for some reason limited to those with an early cancer or with another reason for aspirin-induced bleeding.

All-in-all the data do not convince me to start taking aspirin -- I am still at odds with Dr. Agus on that. The selection bias that I described above may very well mean that aspirin's role is not as a cancer prevention, but more likely as a sort of a stress test for those with a subclinical cancer. So we are left again with the the chicken-and-egg question. But isn't that, after all, what makes science exciting?

Would love to know what others think -- does this make sense? Are there other possible explanations?      

If you like Healthcare, etc., please consider a donation (button in the right margin) to support development of this content. But just to be clear, it is not tax-deductible, as we do not have a non-profit status. 


 Thank you for your support!

A good week

So, it has been a great week here at Healthcare, etc, and I am grateful.  Here are the highlights:

1. I did a guest post on plagiarism at Retraction Watch, and not only did it generate a fantastic conversation there, but it also brought other stories of plagiarism to us right here on this site. The story is not finished, and I will update when I can.

2. Gary Schwitzer of Health News Review cited 2 posts from Healthcare, etc. on his extremely popular and well-respected site here and here. The corresponding stories that he referenced were "Unpacking the meat data" (which incidentally is the most popular post of all time on this blog) and "PSA screening: Does it or doesn't it" (a niche post with a lively discussion in the comments).

3. The crowning glory of the week was being cited by Mark Bittman in his New York Times column on May 16th, which drove hordes of readers to Healthcare, etc.

4. Finally, my book is progressing through editing and design, and it is all beginning to seem more real. It feels like I am realizing my dream of bringing a critical approach to evaluating medical evidence to all who are interested. If you want to get news about it in your inbox, please sign up here.

In the next few weeks you can expect to hear about my experience attending and presenting at the Ignite Boston conference at MIT on March 29, as well as coverage from TEDMED 2012 in DC, which I am thankful to be attending as a Frontline scholar.

Thanks again to everyone for visiting, tweeting, linking and sharing in all other ways. Looking forward to seeing you here again.

Unpacking the meat data

So, this story from Harvard on how red meat is bad for you deserves some unpacking. First, allow me to say that all meat is not created equal: the cows that graze on the farm around the corner from where I live make meat that is quite different from that reconstituted slime used by fast-"food" restaurants. Cows that are raised on CAFOs and fed corn-based diets are practically different species from those guernseys down the street.

But putting that aside, let's just look at what the paper reports and what the numbers add up to. The investigators examined two large observational cohort studies totaling over 100,000 subjects and tried to estimate the risk of death associated with red meat consumption. Now, first, it has been widely acknowledged that dietary habit surveys are a difficult beast, and that is how these two studies got at the food history. Next, let us look at some of the numerators and denominators. The paper reports 23,926 deaths among these >100,000 subjects over 22 to 28 years of observation. The denominator for this type of a study is person-years, where you simply multiply the number of persons observed by the corresponding number of years of observation. In this instance, this value is 2,960,000 person-years. So, the roughly 24,000 deaths occurred over 2.96 million years of observation, simplifying to 24,000/2,960,000 = 8 deaths per 1,000 years overall. If we were to translate this to an individual's risk for death over 1 year, it would be 0.008, or under 1%.

The study further reports that at its worst, meat increases this risk by 20% (95% confidence interval 15-24%, for processed meat). If we use this 0.8% risk per year as the baseline, and raise it by 20%, it brings us to 0.96% risk of death per year. Still, below 1%. Need a magnifying glass? Me too. Well, what if it's closer to the upper limit of the 95% confidence interval, or 24%? The risk still does not quite get up to 1%, but almost. And what if it is closer to the lower limit, 15%? Then we go from 0.8% to 0.92%.

Does this effect size matter, even if statistically significant? What if this were a randomized controlled trial for a statin? What would we say to this result? Even if this is a real signal, which is questionable given the observational design (yes, despite holding a special affection for observational studies, I don't think that this cause-effect is completely unconfounded; and this matters greatly in view of this minuscule magnitude), I am far more likely to die next time I get into my car than from eating burgers, even if I do indulge in one a couple of times per week. I am certainly not advocating eating red meat 7 days per week, though this view is driven more by practical concerns for sustainable beef farming than by the data presented in this paper.

There are a few political issues to disentangle. I despise CAFOs and their product; I despise their contempt for the animals and for the environment; and I despise their disregard for human health. I would love to see a study that shows that CAFO-raised beef kills people, as my cognitive biases tell me it must. I would love to see them all shut down, period. And this goes for the meat packing and distributing oligopoly as well. This venom notwithstanding, the current paper gives us no ammunition to this end: it failed to explore this pivotal question. Pity!

Furthermore, a study like this is likely to feed extremist marketing messages to suit someone's agenda that will likely drag us farther away from the moderation that is conducive to our health. But my local farmers should rest easy knowing that this is not by any means a game changer, that there is nothing in this paper to make us any less enthusiastic about their product, and that our New England pastures will not any time soon be devoid of the beautiful sight of these lovely ruminants. Moderation in everything including meat consumption, is probably still the best course of action. If we focus our energies on what is genuinely good for our health, we will do right by the environment.      

The flu and likelihood ratios

An interesting study was just published in the Annals of Internal Medicine. It was a meta-analysis of rapid influenza diagnostic tests (RIDT) and their characteristics. Since we have been spending so much time talking about test characteristics, this study provides a nice opportunity to discuss another way of looking at the values of positive and negative tests. This is going to be a fairly short post, since I simply want to discuss these additional tools.

In the current study the investigators evaluated how well these RIDTs predicted the disease. As we have discussed in multiple places on this blog (here, here and here, to name a few), it matters whether the test is used for screening or diagnosis. In this case, the testing was done in symptomatic populations, so for diagnostic reasons. The authors report that there was quite a bit of heterogeneity in their findings, but the ultimate result is reported as a positive (34.5) and negative (0.38) likelihood ratios. What are they and how do we interpret them?

A positive likelihood ratio, or LR+ is the ratio between sensitivity and 1-specificity (LR+=[sensitivity]/[1-specificity]). Sensitivity is the proportion of patients with the disease who are identified as having the disease, or true positives (TP), and specificity is the proportion of persons without the disease who are identified as not having the disease, or true negatives (TN). The opposite of TN, 1-TN, is the false positives (FP). So, the LR+ equates to the TP/FP, or the odds that a positive indicates true disease. In the current study it is 34, meaning that the odds are 34 to 1 that a positive test indicates the presence of the disease. Another way of putting it is that of the 35 total positive test results, 34 (97.1%) represent true disease. This is essentially equivalent to the positive predictive value (PPV).

Now, let's examine the negative likelihood ratio, or LR-. This is defined as the ratio between the opposite of sensitivity (1-sensitivity) and specificity (LR-=[1-sensitivity]/specificity]). 1-sensitivity is the proportion that are false negative (FN), while the specificity is the proportion of persons without the disease who are identified as such. In the study this LR- was 0.38, meaning that the odds that a negative result truly indicates the absence of disease are about 1 to 2 (0.36:1), or not so great. In other words, out of the total of 3 negative tests, 2 are truly negative, while 1 is a false negative, giving us the negative predictive value (NPV) of about 65% (actually it is 1-0.36=0.64, or 64%).

So, there you have it. The clinical take-away, as the authors noted, is that these RIDTs are good at ruling in the flu, but not at ruling it out. In other words, the problem here is the opposite of what we discussed in all those previous test, or the rate of false negatives. And this makes sense, given that the pre-test probability is reasonably enriched in populations with symptoms, in addition to the relatively poor sensitivities of these technologies.  

  

Some thoughts on denominators

I decided to re-post this piece dating back from May 2009 in the wake of the recent reports about bird flu. As it turns out, this flu may not be quite as deadly as we had previously thought. And, yes, this revelation is all about the denominator.

Let's face it: denominators keep numbers (and people reporting them) honest. Imagine if I said that there were 3,352 cases of a never-before-seen strain of flu in the US. To be sure, 3,352 cases is a large enough number to send us rushing to buy a respirator mask! But what if I put it slightly differently and said that out of the population of roughly 300,000,000 individuals, 3,352 have contracted this strain of flu. I think this makes things a little different, since it means that the risk of contracting this flu to date is about 1 in 100,000, a fairly low number as risks go. Now, I am going to give you another number -- 86. This represents the number of the novel H1N1 flu-related deaths in Mexico reported on April 25, 2009, by the health minister of Mexico, and at that time this flu had been thought to have sickened 1,400 people. This gives us the risk of death with the flu of roughly 6%, a very high risk indeed! Well, that was then. Now that we have all steadied our pulses, and the health authorities have gone back and done some testing, as of yesterday Mexico had confirmed 2,059, cases with 56 fatalities, equating to a 2.7% risk of dying with the disease. Still a high number, to be sure, but lower than what was though before.

In the US, we have had 3 fatalities among 3,352 cases reported as of yesterday, yielding the risk of death from H1N1 in this country of about 1 in 1,000. But, of course, the denominator of 3,352 persons represent only those who sought medical attention and got tested, so probably it is an underestimate of the true burden of this strain of flu, and necessarily also an over-estimate of its attendant mortality. Now, apply this to the situation in Mexico, and it's likely that the risk of death from H1N1 is also lower than what we have observed precisely due to the under-estimation of the denominator. 

So how could we get a true estimate of the numbers of people afflicted with the H1N1 influenza? Well, we could screen absolutely everyone (or more likely a large and representative group of individuals). Then what? Do we treat them all with anti-virals? Do we observe them? Since the Centers for Disease Control and Prevention recommends testing only severe cases and treating only persons at a high risk for complications, universal testing does not seem like a practical approach. So, the bottom line is that we are not likely ever to get at the correct denominator for the risk of dying with this disease, and any number that we get is likely to be an over-estimate of the true risk.

So, what are the lessons here? First, don't let anyone get away with only giving you the numerator, as that is not even a half of the story. Second, even when the denominator appears known, be skeptical -- does it really represent the entire pool of cases that are at risk for the event that the numerator describes? The likely answer will most of the time be "no". Clearly, it is the denominator that is the key to being an educated consumer of health information.

When do diagnostic tests improve mortality?

I thought this post, originally published last May, was worth revisiting apropos this paper that came out in this week's Archives of Internal Medicine. Below I discuss some of the data in the paper, as they were presented in an abstract at a meeting last year, as the context for understanding various mortality statistics. 

The question the title of this post poses is well worth asking, particularly as we argue about the merits of mammography screening. The USPSTF has really stirred up the hornet's nest with this one, and the politicians cannot help but get on their populist pulpit, ignoring the facts completely. Oh well, what else is new?

But the question remains: do screening or diagnostic tests that are more sensitive save lives? A great talk on pulmonary embolism detection and outcomes by a recent graduate from the Dartmouth group at the American Thoracic Society last week prompted me to clarify this. We all hear that mortality from many diseases has decreased over the last few decades. But is this true? In order to answer this question, one has to ask what is meant by mortality. Even people well versed in epidemiology and biostatistics occasionally blur the lines between mortality and case fatality, and to our question the distinction is critical. Case fatality is defined as the proportion of patients with the disease that dies, while mortality is a population-based measure, a proportion of all of the population at risk for the disease that dies. The difference lies in our old friend the denominator, which will always keep us honest.

Let's go through a simple example to illustrate this concept. Let's pretend that the total number of cases of disease D diagnosed using stone-age test T 30 years ago was 100 in a population of 10,000 people. Of these cases, 90 died, giving us the case fatality of 90% and mortality of 9 per 1,000 population. Now, we have a new test for D, a super-Doppler-MRI-PET-cyberscan called über-T, a much more sensitive test than the old "gold standard" test T. And now we detect 1,000 cases of D in the population of 10,000 people. Of the 1,000 cases detected by über-T, 90 have died. The case fatality now has decreased dramatically from 90% to 9%, and we can pat ourselves on the back for a job well done, right? Not so fast, the population mortality from disease D has remained a steady 9 per 1,000 population!

So, what does this mean? Does it mean that über-T, which costs 2 orders of magnitude more than its predecessor, is worthless? Well, decide for yourselves. What it means to me is that the additional cases detected by über-T, though finding earlier stage disease, thus increasing the denominator for the case-fatality calculation, has had no impact on the numerator and therefore has not in fact improved the only mortality that matters: population mortality related to the disease.

So, next time a politician tells you how well we are doing with technological innovation in disease management, ask this simple question: Has all the money and innovation really altered the important outcomes, or is this all smoke in mirrors, a mirage created by our irrational belief that technology is our salvation? This may be an uncomfortable epiphany for some. But think about the 900 excess cases of the pseudo-disease diagnosed in our example above -- how many people could have been saved becoming a chronically ill person, how many complications of follow-up procedures could have been avoided, and yes, how much money could have been spent on something other than healthcare? And asking these questions may help us to identify technological advances that actually improve our lives, as opposed to those that merely create attractive business opportunities and stimulate the economy.            

Why medical testing is never a simple decision

A couple of days ago, Archives of Internal Medicine published a case report online. Now, it is rather unusual for a high impact journal to publish even a case series, let alone a case report. Yet this was done in the vein of highlighting their theme of "less is more" in medicine. This motif was announced by Rita Redberg many months ago, when she solicited papers to shed light on the potential harms that we perpetrate in healthcare with errors of commission.

The case in question is one of a middle-aged woman presenting to the emergency room with vague symptoms of chest pain. Although from reading the paper it becomes clear that the pain is highly unlikely to represent heart disease, the doctors caring for the patient elect to do a non-invasive CT angiography test, just to "reassure" the patient, as the authors put it. Well, lo' and behold, the test comes back positive, the woman goes for an invasive cardiac catheterization, where, though no disease is found, she suffers a very rare but devastating tear of one of the arteries in her heart. As you can imagine, she gets very ill, requires a bypass surgery and ultimately an urgent heart transplant. Yup, from healthy to a heart transplant patient in just a few weeks. Nice, huh?

The case illustrates the pitfalls of getting a seemingly innocuous test for what appears to be a humanistic reason -- patient reassurance. Yet, look at the tsunami of harm that followed this one decision. But what is done is done. The big question is, can cases like this be prevented in the future? And if so, how? I will submit to you that Bayesian approaches to testing can and should reduce such complications. Here is how.

First, what is Bayesian thinking? Bayesian thinking, formalized mathematically through Bayes theorem, refers to taking the probability of disease being there into account when interpreting subsequent test results. What does this mean? Well, let us take the much embattled example of mammography and put some numbers to the probabilities. Let us assume that an otherwise healthy woman between 40 and 50 years of age has a 1% chance of developing breast cancer (that is 1 out of every 100 such women, or 100 out of 10,000 undergoing screening). Now, let's say that a screening mammogram is able to pick up 80% of all cancers that are actually there (true positives), meaning that 20% go unnoticed by this technology. So, among the 100 women with actual breast cancer of the 10,000 women screened, 80 will be diagnosed as having cancer, while 20 will be missed. OK so far? Let's go on. Let us also assume that, in a certain fraction of the screenings, mammography will merely imagine that a cancer is present, when in fact there is no cancer. Let us say that this happens about 10% of the time. So, going back to the 10,000 women we are screening, of 9,900 who do NOT have cancer (remember that only 100 can have a true cancer), 10%, or 990 individuals, will still be diagnosed as having cancer. So, tallying up all of the positive mammograms, we are now faced with 1,070 women diagnosed with breast cancer. But of course, of these women only 80 actually have the cancer, so what's the deal? Well, we have arrived at the very important idea of the value of a positive test: this roughly tells us how sure we should be that a positive test actually means that the disease is present. It is a simple ratio of the real positives (true positives, in our case the 80 women with true cancer) and all of the positives obtained with the test (in our case 1,070). This is called positive predictive value of a test, and in our mammography example for women between ages of 40 and 50 it turns out to be 7.5%. So, what this means is that over 90% of the positive mammograms in this population will turn out to be false positives.

Now, let us look at the flip side of this equation, or the value of a negative test. Of the 8,930 negative mammograms, only 20 will be false negatives (remember that in our case mammography will only pick up 80 out of 100 true cancers). This means that the other 8,910 negative results are true negatives, making the value of a negative test, or negative predictive value, 8,910/8,930 = 99.8%, or just fantastic! So, if the test is negative, we can be pretty darn sure that there is no cancer. However, if the test is positive, while cancer is present in 80 women, 900 others will undergo unnecessary further testing. And for every subsequent test a similar calculus applies, since all tests are fallible.

Let's do one more maneuver. Let's say that now we have a population of 10,000 women who have a 10% chance of having breast cancer (as is the case with an older population). The sensitivity and specificity of mammography do not change, yet the positive and negative predictive values do. So, among these 10,000 women, 1,000 are expected to have cancer, of which 800 will be picked up on mammography. Among the 9,000 without cancer, a mammogram will "find" a cancer in 900. So, the total positive mammograms add up to 1,700, of which nearly 50% are true positives (800/1,700 = 47.1%). Interestingly, the negative predictive value does not change a whole lot (8,100/[8,100 + 200]) = 97.6%, or still quite acceptably high). So, while among younger women at a lower risk for breast cancer, a positive mammogram indicates the presence of disease in only 8% of the cases, for older women it is about 50% correct.

These two examples illustrate how exquisitely sensitive an interpretation of any test result is to the pre-test probability that a patient has the disease. Applying this to the woman in the case report in the Archives, some back-of-the-napkin calculations based on the numbers in the report suggest that, while a negative CT angiogram would indeed have been reassuring, a positive one would only create confusion, as it, in fact, did.

To be sure, if we had a perfect test, or one that picked up disease 100% of the time when it was present and did not mislabel people without the disease as having it, we would not need to apply this type of Bayesian accounting. However, to the best of my knowledge, no such test exists in today's clinical practice. Therefore, engaging in explicit calculations of what results can be expected in a particular patient from a particular test before ordering such a test can save a lot of headaches, and perhaps even lives. In fact, I do hope that the developers of our new electronic medical environments are giving this serious thought, as these simple algorithms should be built into all decision support systems. Bayes theorem is an idea whose time has surely come.                    

Can a "negative" p-value obscure a positive finding?

I am still on my p-value kick, brilliantly fueled by Dr. Steve Goodman's correspondence with me and another paper by him aptly named "A Dirty Dozen: Twelve P-Value Misconceptions". It is definitely worth a read in toto, as I will only focus on some of its more salient parts.

Perhaps the most important point that I have gleaned from my p-value quest is that the word "significance" should be taken quite literally. Here is what Merriam-Webster dictionary says about it:

sig·nif·i·cance

 noun \sig-ˈni-fi-kən(t)s\

Definition of SIGNIFICANCE

1

a : something that is conveyed as a meaning often obscurely or indirectly

b : the quality of conveying or implying

2

a : the quality of being important : moment

b : the quality of being statistically significant

 It is the very meaning in point 2a that the word "significance" was meant to convey in reference to statistical testing. That is "worth noting" or "noteworthy". Nowhere do we find any references to God or truth or dogma. So, the first lesson is to drift away from taking statistical significance as the sign from God that we have discovered the absolute truth, and to focus on the fact that  we need to make note of the association. The follow up to the noting action is confirmation or refutation. That is, once identified, this relationship needs to be tested again (and sometimes again and again) before we can say that there may be something to it.

As an aside, how many times have you received comments from peer reviewers saying that what you are showing has already been shown? Yet, in all of our Discussion section we are quite cautious to say that "more research is needed" to confirm what we have seen. So, it seems we -- researchers, editors and reviewers -- just need to get on the same page.

To go on, as the title of the paper states, we are initiated into the 12 common misconceptions about what p-value is not. Here is the table that enumerates all 12 (since the paper is easy to find for no fee, I am assuming that reproducing the table with attribution is not a problem):

    
Even though some of them seem quite similar, it is worth understanding the degrees of difference, as they provide important insights.

The one I wanted to touch upon further today is Misconception #12, as it dovetails with our prior discussion vis-a-vis environmental risks. But before we do this, it is worth defining the elusive meaning of the p-value once again: "The p-value signifies the probability of obtaining the association (or difference) of the magnitude obtained or one of greater magnitude when in reality there is no association (or difference)". So, let's apply this to an everyday example of smoking and lung cancer risk. Let's say a study shows a 2-fold increase in lung cancer among smokers compared to non-smokers, and the p-value for this association is 0.06. What this really means is that "under conditions of no true association between smoking and lung cancer, there is a 6% or less chance that a study would find a 2-fold or greater increase in cancer associated with smoking". Make sense? Yet, according to the "rules" of statistical significance, we would call this study negative. But is this a true negative? (To the reader of this blog this is obvious, but assure you that, given how cursory our reading of the literature tends to be, and how often I hear my peers discount findings with the careless "But the p-value was not significant", this is a point worth harping on).

The bottom line answer to this is found in the discussion of the bottom line misconception in the Table: "A scientific conclusion or treatment policy should be based on whether or not the p-value is significant". I would like to quote directly from Goodman's paper here, as it really drives home the idiocy of this idea:

This misconception encompasses all of the others. It is equivalent to saying that the magnitude of effect is not relevant, that only evidence relevant to a scientific conclusion is in the experiment at hand, and that both beliefs and actions flow directly from the statistical results. The evidence from a given study needs to be combined with that from prior work to generate a conclusion. In some instances, a scientifically defensible conclusion might be that the null hypothesis is still probably true even after a significant result, and in other instances, a nonsignificant P value might still lead to a conclusion that a treatment works. This can be done formally only through Bayesian approaches. To justify actions, we must incorporate the seriousness of errors flowing from the actions together with the chance that the conclusions are wrong.

When the author advocates Bayesian approaches, he is referring to the idea that a positive result in the setting of a low pre-test probability still has a very low chance of describing a truly positive association. This is better illustrated by the Bayes theorem, which allows us to quantify the result at hand ("posterior probability") to what the bulk of prior evidence and/or thought has indicated about the association ("prior probability"). This implies that the lower our prior probability, the less convinced we can be by a single positive result. As a corollary, the higher our prior probability for an association, the less credence we can put in a single negative result. So, Bayesian approach to evidence, as Goodman indicates here, can merely move us in the direction of either a greater or a lesser doubt about our results, NOT bring us to the truth or falsity.

Taken together, all these points merely confirm my prior assertion that we need to be a lot more cautious about calling results negative when deciding about potentially risky exposures than about beneficial ones. Similarly, we need to set a much higher bar for all threats to validity in studies designed to look at risky rather than beneficial outcomes (more on this in a future post). These are the principles we should be employing when evaluating environmental exposures. This becomes particularly critical in view of the startling revelations of the genome-wide association experiments findings that our genes determine a very small minority of diseases to which we are subject. This means that the ante has been upped dramatically for environmental exposures as culprits, and demands a much more serious push for the precautionary principle as the foundation for our environmental policy.    

 

Getting beyond the p-value

Update 12/8/10, 9:30 AM: I just got an e-mail from Steve Goodman, MD, MHS, PhD, from Johns Hopkins about this post. Firstly, my apologies for getting his role at the Annals wrong -- he is the Senior Statistical Editor for the journal, not merely a statistical reviewer. I am happy to report that he added more fuel to the p-value fire, and you are likely to see more posts on this (you are overjoyed, right?). So, thanks to Dr. Goodman for his input this morning!

Yesterday I blogged about our preference to avoid false positive associations at the expense of failing to detect some real associations. The p value conundrum, where the arbitrary statistical significance is set at <0.05 has bothered me for a long time. I finally got curious enough to search out the origins of the p value. Believe it or not, the information was not easy to find. I have at lest 10 biostatistics or epidemiology textbooks on the shelves of my office -- not one of them went into the history of the p value threshold. But Professor Google came to my rescue, and here is what I discovered.

Using a carefully crafted search phrase, I found a discussion forum on WAME, World Association of Medical Editors, which I felt represented a credible source. Here I discovered a treasure trove of information and references to what I was looking for. Specifically, one poster referred to Steven Goodman's work, which I promptly looked up. And by the way, Steven Goodman, as it turns out is a statistical reviewer the Senior Statistical Editor for the Annals of Internal Medicine and a member of WAME. So, I went to this gem in the journal Epidemiology from May 2001, called unpretentiously "Of P-values and Bayes: A Modest Proposal". I have to say that some of the discussion was so in the weeds that even I have to go back and reread it several times to understand what the good Dr. Goodman is talking about. But here are some of the more salient and accessible points.

The author begins by stating his mixed feelings about the p-value:

I am delighted to be invited to comment on the use of P-values, but at the same time, it depresses me. Why? So much brainpower, ink, and passion have been expended on this subject for so long, yet plus ca change, plus c'ést le meme chose- the more things change, the more they stay the same. The references on this topic encompass innumerable disciplines, going back almost to the moment that P-values were introduced (by R.A. Fisher in the 1920s). The introduction of hypothesis testing in 1933 precipitated more intense engagement, caused by the subsuming of Fisher's significance test into the hypothesis test machinery.^1-9 The discussion has continued ever since. I have been foolish enough to think I could whistle into this hurricane and be heard. ^10-12 But we (and I) still use P-values. And when a journal like Epidemiology takes a principled stand against them, ¹³ epidemiologists who may recognize the limitations of P-values still feel as if they are being forced to walk on one leg. ¹⁴

So, here we learn that the p-value is something that has been around for 90 years and was brought into being by the father of frequentist statistics R.A. Fisher. And the users are ambivalent about it, to say the least. So, why, Goodman asks, continue to debate the value of the p-value (or its lack)? And here is the reason: publications.

Let me begin with an observation. When epidemiologists informally communicate their results (in talks, meeting presentations, or policy discussions), the balance between biology, methodology, data, and context is often appropriate. There is an emphasis on presenting a coherent epidemiologic or pathophysiologic story, with comparatively little talk of statistical rejection or other related tomfoolery. But this same sensibility is often not reflected in published papers. Here, the structure of presentation is more rigid, and statistical summaries seem to have more power. Within these confines, the narrative flow becomes secondary to the distillation of complex data, and inferences seem to flow from the data almost automatically. It is this automaticity of inference that is most distressing, and for which the elimination of P-values has been attempted as a curative.

This is clearly a condemnation of the way we publish: it demands a reductionist approach to the lowest common denominator, in this case the p-value. Much like our modern medical paradigm, the p-value does not get at the real issues:

I and others have discussed the connections between statistics and scientific philosophy elsewhere, ^11,12,15-22 so I will cut to the chase here. The root cause of our problem is a philosophy of scientific inference that is supported by the statistical methodology in dominant use. This philosophy might best be described as a form of naïve inductivism,²³ a belief that all scientists seeing the same data should come to the same conclusions. By implication, anyone who draws a different conclusion must be doing so for nonscientific reasons. It takes as given the statistical models we impose on data, and treats the estimated parameters of such models as direct mirrors of reality rather than as highly filtered and potentially distorted views. It is a belief that scientific reasoning requires little more than statistical model fitting, or in our case, reporting odds ratios, P-values and the like, to arrive at the truth. [emphasis mine]

Here is a sacred scientific cow that is getting tipped! You mean science is not absolute? Well, no, it is not, as the readers of this blog are amply aware. Science at best represents a model of our current understanding of the Universe, it builds upon itself usually in one direction, and it rarely gives an asymptotic approximation of what is really going on. Merely our current understanding of reality, given the tools we have at our disposal. Goodman continues to drive home the naïveté of our inductivist thinking in the following paragraph:

How is this philosophy manifest in research reports? One merely has to look at their organization. Traditionally, the findings of a paper are stated at the beginning of the discussion section. It is as if the finding is something derived directly from the results section. Reasoning and external facts come afterward, if at all. That is, in essence, naïve inductivism. This view of the scientific enterprise is aided and abetted by the P-value in a variety of ways, some obvious, some subtle. The obvious way is in its role in the reject/accept hypothesis test machinery. The more subtle way is in the fact that the P-value is a probability - something absolute, with nothing external needed for its interpretation.

In fact the point is that the p-value is exactly NOT absolute. The p-value needs to be judged relative to some other standard of probability, for example the prior probability of an event. And yet what do we do? We worship at the altar of the p-value without giving any thought to its meaning. And this is certainly convenient for those who want to invoke evidence of absence of certain associations, such as toxic exposures and health effects, for example, when the reality simply indicates absence of evidence.

The point is that we need to get beyond the p-value and develop a more sophisticated, nuanced and critical attitude toward data. Furthermore, regulatory bodies need to get a more nuanced way of communicating scientific data, particularly data evidencing harm, in order not to lose credibility with the people. Most importantly, however, we need to do a better job training researchers on the subtleties of statistical analyses, so that the p-value does not become the ultimate arbiter of the truth.

Of P values, power, tobacco and cell phones

Like everything else political, science, as written in the tobacco playbook, is being used for divisive purposes. A new study from Denmark, as reported by WebMD, indicates that exposure to cell phones in utero may influence the child's behavior. The authors are appropriately tempered in their conclusions and subsequent recommendations, responsibly acknowledging the obvious limitations of their research. Yet, to paraphrase one of the authors, what is the harm in asking a pregnant woman to keep her cell phone away from her uterus? It is at most a slight inconvenience. This sentiment is echoed by Devra Davis, the author of Disconnect, a book summarizing what we know about human health effects of microwave radiation and cellular technology, in her comments to WebMD, wherein she plainly echoes the authors to say that the current study was not perfect. Yet, she urges precaution until we know more about the potential ill effects of cell phones, particularly on the most vulnerable in our population.

Now, I have blogged before about Devra Davis and about my sense that her messages of caution are less than popular even with those who consider themselves to be scientifically skeptical. Perhaps especially with those who consider themselves to be scientifically skeptical. It seems to me that we have gone a little bit overboard on this critical thinking idea, where we demand data on harms to reach the same level of evidentiary standards as salutary effects. This is a fallacy. It is a convenient fallacy, to be sure. In fact, what many people may not realize is that it is much more than an accidental byproduct of our skepticism. No, it is a strategy, as discussed in Davis's book The Secret History of the War on Cancer deliberately developed and skillfully executed by the tobacco industry to cast doubt for decades on the harms of smoking. Look how well it has done -- 80 years of knowing of cancer risks associated with smoking and nearly 50 years after the Surgeon General's report on the dangers of smoking, we are still looking to mass screening for diseases caused by tobacco rather than mass cessation efforts to diminish its impact. This incredibly effective and durable strategy relies on scientific data generated by men with loud reputations funded by the tobacco companies to demand unequivocal causality between the exposure and the outcome. So, inherently, it demands bodies in the street to show unequivocal links.

Predictably, now the cell phone industry is using the same strategy. Here are a couple of quotes from the WebMD article, which I personally find less than amusing:

John Walls, vice president of public affairs at CTIA-The Wireless Association, a trade group representing the wireless industry, tells WebMD that his group “stands behind the research review by independent and renowned public health agencies around the world which states that there are no known adverse health effects associated with using wireless devices.”

Jeff Stier, a senior fellow at the National Center for Public Policy Research, a conservative think tank, says that the new study is full of holes. “For starters, self-reporting of cell phone use makes it impossible to assign any meaning to the exposure,” he says.

“Different phones give off different exposures, and even those who were reported to be not exposed, probably had significant environmental exposure, rendering the study only slightly more than amusing,” he tells WebMD in an email.

Surprise! The science is just not there! So, let's party, right?

Well, wrong. In epidemiology we set a very high bar for avoiding a false positive, and a much lower bar for a false negative. You have all heard of the much misunderstood p value. What this number represents is exactly the probability of chance giving us the result of the magnitude observed or one of a greater magnitude under the circumstances of no real association. By convention (whim?) we usually set the threshold for significance at <0.05, meaning that there is a <5% of obtaining the given result or one of greater magnitude by chance if there is no actual association. This clearly stacks the deck against calling an association positive. In contradistinction, we are much more willing to obtain a false negative, as illustrated by the conventional power of a study to detect a difference where a true difference exists at only 80%. So, colloquially, we want to be 95% sure that a positive result is not a false positive association, yet only 80% sure that a negative result is not a false negative. You see the difference? We are clearly more keen to discard associations that do not really exist than to identify them.  To an untrained eye the difference may be subtle, but it creates a world of philosophical difference in how we approach data.

And it is exactly these in some ways arbitrarily accepted standards of our science that are corrupted to sell false certainty of the negative associations. I have written about my thoughts on proof of harm vs. benefit in prior posts. I firmly believe that it is our interest in being more certain that an association is not spurious at the expense of being more willing to discard associations that are true that is facilitating our national blindness to many potentially adverse health effects from stuff around us. I am convinced that causation of harm needs to be viewed with greater caution and therefore with greater scientific permissiveness. If we do not adopt this precautionary principle, used by the EU liberally in their environmental policies, we will continue to see even more piles of bodies in the streets -- just look at tobacco. This is especially true when all there is is a risk with no benefit (tobacco), or when a simple behavior change can eliminate any concern for an adverse outcome (cell phones).

What is the harm in repeat CDI testing?

This is cross-posted on my new False Positive blog where I plan to curate some of my writing specific to research methods. 

I recently peer-reviewed a manuscript on Clostridium difficile surveillance for a journal and was prompted to reread this paper by Lance Peterson and Ari Robicsek from the Annals of Internal Medicine August 2009. The paper is a great reminder of the high risk of false positive results from repeating a moderately sensitive yet highly specific test for a disease of low prevalence.
The case in point is C. difficile testing, of specific interest to me with the enzyme immunoassays (EIA) for toxins A and/or B. A frequent clinical dilemma is what to do when a patient has symptoms and signs suggestive of C. diff infection (CDI), yet the initial test comes back negative. Frequently the clinician will repeat testing, and possibly more than once. And on occasion, the repeat testing will yield a positive result. But how accurate is this positive result?
To answer this question, Peterson and Robicsek constructed a simple model, depicted in Table 2 of the paper (I have reproduced a part of it below). The authors derived EIA characteristics from the literature to be: sensitivity = 73.3%, specificity = 97.6%. They also assumed a true prevalence of disease of 10% among patients with suspected CDI. So, starting out with a hypothetical 1,000 patient cohort with suspected CDI, the results look as follows:
What comes out loud and clear is that with every round of repeat testing, the positive predictive value degrades dramatically, until the result becomes completely useless. But to go even further, the result may be harmful since we have now gone from having 1/4 of all detected positives being false to a whopping > 1/2! And now we need to think long and hard what to do with these results. Now, one other point: Merely going from a single test to a second round, while adding a substantial number of true positives (18), ends up identifying even a greater number of people without the disease as having CDI (22)!
This illustrates the great potential for misclassification of CDI that we run into in surveillance studies, leading to an inflated disease estimates. The moral of the story is that we always need to know what proportion of positive tests are in fact repeat tests following a negative assay. And I am even beginning to think that we need to limit ourselves at most to two consecutive tests when talking about incidence and prevalence of CDI.

Unidirectional skepticism

I thought I had coined that phrase. I have used it to refer to our tendency to examine ideas that are not aligned with our views of reality with less rigor than those that confirm it. It turns out that I am not the first to notice this phenomenon. In this Harper's article, Gary Greenberg uses the following quote from Mario Beauregard, a neuroscientist at the University of Montreal, writing about the seat of the "self":

The culture of popular science is one of unidirectional skepticism... It is skeptical of any idea that spirituality corresponds to something outside ourselves, but surprisingly gullible about any reductionist explanation for it.

Still, I think that my use of the term is somewhat broader, not simply alluding to the popular science, but also to professional science, and not only to the neurobiology of self. This idea goes hand in hand with the cognitive biases we are subject to.

This has got me thinking about a couple of things. Now, I am a big fan of neurobiology and the burgeoning field of research that it represents. We understand what goes on in the brain so much better now at the molecular level than we did, say, even 20 years ago. But what I am struck by is the comfort that this knowledge gives us about ideas of human behavior we had understood with lesser granularity in the prior era. For example, all parents have appreciated for millennia that teenage behavior is characterized by disinhibition of the higher self, if you will. Now we can say this in scientific lingo, that the area of the prefrontal cortex responsible for executive function lags in development behind other areas of the brain, as demonstrated in fMRI studies. Or how about the evidence from advanced neuroimaging that doing stuff that benefits our community lights up areas that correspond to happiness? We knew this before, only not in these words: we just had an idea that being a part of a community made us feel good. And those of us who subscribe to evolution can even come up with an explanation for why we should have developed this way.

Yet, aside from just gaining fascinating knowledge, we derive an odd sense of satisfaction from having our prior ideas, arrived through experience, observation and philosophical ruminations, confirmed by what we consider to be a more bona fide technique of science. The corollary to this is that, because we are so committed to the scientific method above all other possible ways of knowing, we may also be all too willing to discard rapidly closely-held notions that are not supported by science. But since science is ever-evolving and rather imprecise, in some areas more so than in others, are we always justified in such  prompt disposal of prior beliefs?

In the clinical world we often cite examples of when our intuition was wrong about some disease or another, and only after doing well controlled research studies were we able to arrive at the real understanding of the situation. But how often is it just our willingness to yield to what we think is scientifically sound at the moment? Is it possible that this unidirectional skepticism fueled by the cognitive bias in favor of new science, may not turn out to be the most efficient way to accumulate knowledge?

I do not know the answer to this question, as I am as guilty as my peers of falling in love with new scientific knowledge. But I also enjoy thinking about the nature of knowledge in general, and the many different ways of knowing. Are the materialism of science and other epistemologies mutually exclusive? I hope not.

More thoughts on lung cancer screening

Today I want to continue in the vein of yesterday's post and discuss a slightly different aspect of the NLST study. But first, a story. I was at a meeting a couple of weeks ago, where we were discussing antibiotics in pneumonia. We were talking about mortality, and one of the docs said something that became a constant mosquito buzz in my brain, which I have been unable to swat away. He said that antibiotics never claimed to reduce all-cause mortality, only mortality associated with infection. Now, on the face of it, the statement makes perfect sense, particularly as you think mechanistically. But it also makes an important point from the perspective of population epidemiology. And it just happens to be related to the NLST study.

Yesterday we talked about the characteristics of the screening tests and in what populations it may make sense. But all of these current and future decisions rest on the results for the primary outcome, which was mortality from lung cancer, reaching an impressive 20% relative reduction. But what about all-cause mortality, and why should we care? Well, saving people from cancer deaths is great, and this is what early screening can mechanistically do. But it is only great if they are not at risk of dying of something else and can derive a meaningful benefit from a life not terminated by an undetected cancer. Unfortunately, as you well know, life does not work this way, and there are many other reasons that people die -- old age, cardiovascular disease, and the like. In epidemiology we call these "competing risks", and it is these risks that sometimes dilute the excitement over the effectiveness of our interventions to a Pyrrhic victory.

Let us apply this to the NLST, or at least what we know today. The results reported are so cursory at this time, that we have to make a lot of educated guesses. But when has that ever stopped us? The report, among other data, states "all-cause mortality was 7% lower" in the CT than in the CXR screening group. The report fails to say whether this is a relative or an absolute reduction, and, as you know, this is a critical distinction. It is critical because, if my calculations are correct, the 20% relative reduction in the cancer mortality detected represents only a 0.3% absolute reduction in deaths from cancer. Given a couple of other tid-bits peppering the report, a few numbers can be back-calculated:

So, again, if my calculations are correct, the all-cause mortality is ~6.4% in the CXR group and ~5.5% in the CT group, so no way can there be an absolute 7% reduction in mortality, and the absolute reduction then turns out to be 0.9% over the 5-year follow-up. And here is what it looks like graphically:

There actually is a reduction in non-cancer deaths in the CT arm. Why this is remains unclear so far, but may well be due to the observation bias, as the study was not blinded. If this is the case, the differences should smooth out over further follow-up period.

But here is my bottom line. Since we live in the world of competing risks, the question becomes "What is the value of screening with CT scans to prevent 0.3% of cancer-related deaths over 5 years?" And how much of this 0.3% benefit is due to lead time bias? This question cannot be answered yet, as we do not have the accounting of either the adverse events related to screening and follow-up, or of the financial costs, or of the patient utilities for either strategy. What we can guess is that the risks have to be miniscule in order not to overwhelm the very small benefit detected, even if it is real.

I am personally not that happy that this report was released the way it was -- no mention of absolute risk reductions, no explicit disclosure of the denominators, no hint as to what the comparative risks of the strategies are (both long- and short-term), and most importantly what implications either strategy has for the patients' quality of life. I guess all of these questions will be answered in the peer-reviewed publications that will result from this study. Fully and transparently. Without any politics or obfuscation. I am waiting with baited breath.

CT screening for lung CA: an epidemiology primer

Amid the hype surrounding it, there is an opportunity to ponder the new NCI CT screening study that reduced lung CA deaths by 20% over the 5-year follow-up. The study itself was interesting, as it was a randomized controlled trial of the screening. This means that, although the possibility of lead time bias remains as the explanation for some of the detected difference (this refers to the fact that in the CT scan group the cancers were diagnosed earlier and therefore the deaths may still occur but simply beyond the five-year follow-up period), this is less likely than in any other design. So, the findings, while intriguing, need to be considered with the modicum of caution as recommended by the NCI, particularly given the high number needed to screen to save one life (300), and the high number of false positives requiring invasive testing. Not to mention all the radiation exposure.

So, while the analyses of the trial continue, some groups, such as the Steelworkers' Union, have already jumped on the side of demanding screening for their members. Is this altogether unreasonable and premature? Well, while it may be premature, I do not think that it is altogether unreasonable. And this has to do with screening test characteristics, some of which are immutable and others exquisitely dependent on the population being screened.

The test characteristics that tell us how well a test detects disease are sensitivity, specificity, predictive positive value and predictive negative value. Sensitivity refers to how well the test picks up the disease that is really there, while specificity tells us whether we can trust that a positive test signifies the presence of the particular disease in question. We also calculate a positive predictive value (PPV, or how much we can trust that a positive test really signifies the presence of the disease), and a negative predictive value (NPV, or how sure are we that a negative test result really means the absence of the disease).

In epidemiology, where we are always concerned with denominators, we like to construct so-called 2x2 tables to illustrate. Here is an example:  

 
The values in the squares signify the following: A = true positives, B = false positives, C = false negatives, D = true negatives.

And while both sensitivity and specificity of a test are intrinsic characteristics of the test itself and are not subject to change by the population risk of the disease, not so for PPV and NPV. These measures are sensitive to the pre-test probability of the disease being present. That is, the PPV and NPV of a test are enhanced if the population is enriched for persons with a higher likelihood of the disease.

This brings me to the Steelworkers' Union demand for CT screening. Here is their rationale:

"Millions of workers have been exposed to asbestos, silica, chromium, arsenic, beryllium, cadmium, nickel and combustion products — and all of these exposures are firmly established as causes of human lung cancer.

Union health and safety leaders and others need to meet in the very near future in Washington DC to devise a strategy for assuring that high risk workers are among the first to obtain the benefits of this new screening method."

And here is why it is not altogether unreasonable. Now, it is well appreciated that these workers have high rates of smoking. According to the CDC, smoking alone increases the risk of lung cancer 10-20-fold. It is also well appreciated that occupational exposure to asbestos, for example, is associated with an enormous increase in the risk of lung cancer. More importantly, however, together with smoking these exposures gang up as a team to help each other to promote cancer much more effectively than if each were working alone. This is called potentiation, and it results in a multiplicative increase in lung cancer risk when both smoking and occupational exposure exist. Therefore, the population represented by the Steelworkers' Union is indeed enriched for the risk of lung cancer development, and may indeed be the population where both the PPV and the NPV are enhanced beyond what we see in smokers only. This means a higher degree of certainty that a positive CT finding truly represents cancer, as well as increased comfort that a negative test is not simply missing a cancer that is there. As a corollary, this means a potentially optimized risk-benefit balance for this screening test.

So, while we are awaiting further data from the trial, the Steelworkers' Union may be on to something.    

    

Furthering the discussion

My original intent was to go through Dr. Novella's and Orac's criticisms individually and take it from there. On second thought I decided not to take that approach. Instead, here is my response.

Firstly, I am grateful that there has been so much discussion about our views. Amid many valid points in their posts made with a skillful turn of the phrase, I saw quite a lot of sarcasm as well. I am sure that the tone of my original post is what incited it, and for that I am sorry: I really do want to have a civil discussion about these ideas, as I realize that we are all learning all the time, and the only way to gain a better understanding of a topic is through discourse. So, again, I apologize for setting the confrontational tone, and will try to avoid it in the future.

I do believe our views are more same than different. We both (SBM group and I) understand that science evolves, that evidence is not stagnant and the sense of certainty frequently conveyed to the lay public by the media is oftentimes misplaced. We simply disagree on the extent to which there is uncertainty in evidence. While it is true that the oft-cited 5-20% number representing the proportion of medical treatments having solid evidence behind them is very likely outdated, the kind of evidence we are talking about is a different matter.

In the hierarchy of evidence, depending on where you look, it is either meta-analyses or the randomized controlled trial that is the gold standard. The latter is a great proof of concept tool, but it is necessarily limited in its external validity, or generalizability. The reason for this is that these trials, frequently done for regulatory purposes, very limited types of patients, exert extremely stringent controls on the total care of the patient (or else are criticized for not doing so if they fail to do so), focus on short-term and surrogate outcomes (hence, the use of cholesterol lowering as a marker for cardiac mortality, for example), and do a fairly abysmal job as a rule considering the sources of heterogeneity of response. The interventions come to market and are typically used in a much broader population based on the evidence of the RCT. This paper, one of many in the same vein, is a nice illustration of a perennial problem with trial evidence, where real-world use of a therapy goes far beyond the available evidence. And although this paper addresses issues with evidence used for reimbursement, these are the same studies that feed guideline recommendations. Certainly meta-analyses, which are a way to combine the data from multiple RCTs in a systematic way, when done well can give us greater confidence of the direction and the magnitude of the treatment effect, but they in no way overcome the generalizability issues of their component RCTs.

The next rung of the evidence ladder is observational data, specifically cohort studies first prospective, then retrospective. I am actually a great fan of observational data, as I have mentioned in the past. Cohort studies give us the opportunity to examine what happens in the real world without imposing artificial conditions necessary in a clinical trial. Observational data can be great when describing epidemiology of a particular disease, the frequency of a given exposure, how different characteristics can modify the relationship between the exposure and the outcome. One of the most attractive features of cohort studies is that the population can be observed over long period of time -- just look at the Nurses' Study, the Framingham Cohort, and others. But these types of studies also have important limitations, and these are readily acknowledged as a heightened susceptibility to bias (especially in the retrospective studies), the possibility of misclassifying important events, and, despite our best efforts to adjust for it, residual confounding. I will come clean and admit my affection for observational data, even despite the fact that it is lower on the totem pole of evidence than an RCT. I really love this paper by Rothman and Greenland that takes a bird's eye view of our research debates. The whole paper really tickles the brain, but I will quote from a section of it briefly here:

Impossibility of Proof

Vigorous debate is a characteristic of modern scientific philosophy, no less in epidemiology than in other areas. Perhaps the most important common thread that emerges from the debated philosophies stems from 18th-century empiricist David Hume’s observation that proof is impossible in empirical science. This simple fact is especially important to epidemiologists, who often face the criticism that proof is impossible in epidemiology, with the implication that it is possible in other scientific disciplines. Such criticism may stem from a view that experiments are the definitive source of scientific knowledge. Such a view is mistaken on at least two counts. First, the nonexperimental nature of a science does not preclude impressive scientific discoveries; the myriad examples include plate tectonics, the evolution of species, planets orbiting other stars, and the effects of cigarette smoking on human health. Even when they are possible, experiments (including randomized trials) do not provide anything approaching proof, and in fact may be controversial, contradictory, or irreproducible. The cold-fusion debacle demonstrates well that neither physical nor experimental science is immune to such problems.

Some experimental scientists hold that epidemiologic relations are only suggestive, and believe that detailed laboratory study of mechanisms within single individuals can reveal cause–effect relations with certainty. This view overlooks the fact that all relations are suggestive in exactly the manner discussed by Hume: even the most careful and detailed mechanistic dissection of individual events cannot provide more than associations, albeit at a finer level. Laboratory studies often involve a degree of observer control that cannot be approached in epidemiology; it is only this control, not the level of observation, that can strengthen the inferences from laboratory studies. Furthermore, such control is no guarantee against error. All of the fruits of scientific work, in epidemiology or other disciplines, are at best only tentative formulations of a description of nature, even when the work itself is carried out without mistakes.

What follows in the hierarchy of evidence are case-control studies, done for some very specific reasons, then case reports and finally expert opinion. When evidence-based guidelines are developed, a comprehensive systematic literature review is undertaken, and all the evidence is examined and ranked. Based on these papers, a recommendation is made and a strength of this recommendation is reported based on the quality of the underlying evidence. This is an arduous and costly process, and it is commendable that it is undertaken. At the same time, given the limitations of the components of the guideline, the final product can be quite inconclusive or even misleading (I hate to bring it up, but look at the screening mammography debate, as well as the recent HRT recommendation reversal). I think it is obvious that I believe in the scientific method, I am simply not convinced that we have done such a great job generating trustworthy evidence in many instances. At the same time, I am not totally nihilistic about what we know, but am somewhere between thinking we have good evidence for a lot of stuff vs. not having any for anything at all.

Allow me one more piece of evidence, if you will, though this is merely anecdotal coming from my dual experience as an author and peer reviewer. I am occasionally floored by the quality of peer review. I have had reviews say on the one hand that of course the paper should be accepted, since it comes from such a reputable group, and on the other reject out of hand papers based on the reviewers' profound lack of understanding of the methods employed. And lest I sound like a crybaby, let me say that I welcome a well-reasoned rejection. What I am talking about is not that. And this is not a surprise, since pretty much anyone can sign up to be a peer reviewer, since, to the best of my knowledge, there is no set of qualifications that journals ask for in their reviewers. And this, so far as I know, applies even to such high caliber publications as JAMA.

So, these are my thoughts on evidence-based medicine. I welcome responses to this, as my understanding of this science is constantly evolving, and differing well-reasoned opinions really help me get a better handle on this stuff.

I will try to tackle my CAM argument next. If I in any way implied in my remarks that I encourage allopathic physicians (by the way, I am not using it in a derogatory way, but merely as it is defined here; in fact, until today I was blissfully unaware of its negative connotation) to be purveyors of CAM, I sincerely apologize. I am pretty sure, however, that I have never made such a statement, as this is not what I believe. My belief is that all modalities that may impact what happens to public's health need to be evaluated for safety, not question. I think we both agree, since there is really no reason to think that something like homeopathy has anything that can help, by the same token we do not believe that it have anything that can hurt. Same with healing crystals, reiki and prayer. So, if a person wants to engage in these activities, and they are perfectly safe physically, be my guest. Other modalities, such as chiropractic, acupuncture, herbalism and the like, definitely need to be evaluated more stringently, as there is reason to think that they may cause harm. And the decisions about their use must be made based on the probability of harm vs. the perceived probability of benefit. Why do I think that these should not be regulated the same way as allopathic treatments? Well, herbs grow naturally and I dare say we have little to say about what our patients grow in their back yards, unless of course the thought is to regulate them the way we do marijuana. As for chiropractic, it is already regulated, though to what extent I am not sure, and would love to hear from someone who knows. Since its techniques most resemble surgical interventions, the level of evidence for them should perhaps be the same as that which we demand in the surgical literature. This is just a thought, and I am not sure that I am correct in this, so other views are, as always, welcomed. What is coming through for me is that perhaps my call to equipoise was a little over the top, as I do not seem to be approaching the above CAM issues in a frequentist, but more in a Bayesian way (though I remain committed to equanimity). Yet, there is something to be said about the frequentist approach, even though it is not my way generally. The frequentist approach, which is what underlies the bulk of our traditional clinical research, does not rely on differential prior probabilities for different possible associations, but treats them all equally. Despite many disadvantages, one obvious advantage is that we do not discount potential associations that do not have biologic plausibility, given our current understanding of biology, and sometimes help us stumble on brand new hypotheses. So, clearly, there is a tension here, and I am still working on what is the better way, if any.

My final words will be about vaccination. It is disheartening to be lumped with "anti-vaxers", as has been done in the comments to Dr. Novella's and Orac's posts. While my bruised ego will survive this insult, I would like to question this assertion. Nowhere have I said that vaccinations are a bad idea or present a real danger to our children. The hype surrounding the vaccination-autism "debate" is abhorrent to me. What I have stated, however, is that I am of the opinion that we have gone a bit overboard with some of them, one being the chicken pox vaccine. Now, this does not make me an "anti-vaxer"; this just makes me a bit skeptical. The way I view the data is that the advantages for this vaccine are mostly economic, in that they prevent parents from missing days at work. Now, I am certainly not opposed to making such a vaccine available to parents who desire it, but I am not convinced that it should be a prerequisite for my kid to go to school. Given that there is always a possibility of an adverse reaction, no matter how small that possibility is, if the risk of it may outweigh the benefit (and here I do not mean the benefit of having mom show up at work), it has to be weighed very carefully. And even though the question asked by one of the comments raises the issue of an immunocompromised child worrying about potentially being exposed to chicken pox, given the known issues with breakthrough disease, I am not sure that immunizing all of his/her contacts would produce anything other than a false sense of security. My sentiment about the HPV vaccine is similar, to respond to another comment. For reasons laid out here and elsewhere on my blog, I am pretty convinced that it would be a compete subversion of the intent of vaccinations to make it into another mandated shot. To date HPV is merely a recommendation, whose validity I am free to question, though last I heard there was movement afoot to make it mandatory. If in your informed opinion your daughter should get vaccinated against HPV, well then I have very little to say about it. But if I were to counsel an individual patient in the context of my understanding of the data, I would be very upfront with my view.

To me the fact that there are such heated debates about this stuff is a testament exactly to how NOT straightforward our science is. I do understand that as a researcher I can afford a certain amount of analysis paralysis that is unacceptable at the bedside. However, I think we (and the press) do a disservice to the patients, to ourselves and the science if we are not upfront about just how uncertain much of what we think we know is. I could not have said it better than this story about Dr. Devereaux's presentation at the recent ASA meeting did here:

It would be nice if we could all agree that science is not static, but rather progresses and regresses. We learn, and then find out that some of what we thought we had learned was wrong, and set about using that information to seek the next level of truth. Repeat, ad infinitum. Personally, I’d love it if my doctors couched every bit of advice with, “Here’s what we think we know today.”

But I suspect that wouldn’t sit well with many patients, who want certainty (as if there is such a thing). And it especially seems like a difficult proposition in our contentious society, where anti-science nay-sayers like to jump on contradictory findings to challenge the basic value of science overall.

That is exactly NOT what I am trying to do. I am merely reflecting on many of the issues that threaten the validity of what we think we know. I am confident that disclosure and transparency not only lead to better science, but they also lead to science that can withstand the test of nay-sayers.

I have come to the end. I am not sure that I have addressed each and every one of the criticisms, though I hope that I have addressed the majority; I am sure you will point out what I missed. A couple of things about comments: Tomorrow I will be in the air most of the day and may not have the opportunity to sign on to approve comments. So, please, if your comment does not go up until Wednesday, do not think that it has been rejected. Also, I really would like to keep it civil and, though I did not apply this rule today, I will not accept any overt insults or name calling from either side of the debate.

I am sincerely looking forward to continuing this discussion!