Monday, October 25, 2010

Furthering the discussion

My original intent was to go through Dr. Novella's and Orac's criticisms individually and take it from there. On second thought I decided not to take that approach. Instead, here is my response.

Firstly, I am grateful that there has been so much discussion about our views. Amid many valid points in their posts made with a skillful turn of the phrase, I saw quite a lot of sarcasm as well. I am sure that the tone of my original post is what incited it, and for that I am sorry: I really do want to have a civil discussion about these ideas, as I realize that we are all learning all the time, and the only way to gain a better understanding of a topic is through discourse. So, again, I apologize for setting the confrontational tone, and will try to avoid it in the future.

I do believe our views are more same than different. We both (SBM group and I) understand that science evolves, that evidence is not stagnant and the sense of certainty frequently conveyed to the lay public by the media is oftentimes misplaced. We simply disagree on the extent to which there is uncertainty in evidence. While it is true that the oft-cited 5-20% number representing the proportion of medical treatments having solid evidence behind them is very likely outdated, the kind of evidence we are talking about is a different matter.

In the hierarchy of evidence, depending on where you look, it is either meta-analyses or the randomized controlled trial that is the gold standard. The latter is a great proof of concept tool, but it is necessarily limited in its external validity, or generalizability. The reason for this is that these trials, frequently done for regulatory purposes, very limited types of patients, exert extremely stringent controls on the total care of the patient (or else are criticized for not doing so if they fail to do so), focus on short-term and surrogate outcomes (hence, the use of cholesterol lowering as a marker for cardiac mortality, for example), and do a fairly abysmal job as a rule considering the sources of heterogeneity of response. The interventions come to market and are typically used in a much broader population based on the evidence of the RCT. This paper, one of many in the same vein, is a nice illustration of a perennial problem with trial evidence, where real-world use of a therapy goes far beyond the available evidence. And although this paper addresses issues with evidence used for reimbursement, these are the same studies that feed guideline recommendations. Certainly meta-analyses, which are a way to combine the data from multiple RCTs in a systematic way, when done well can give us greater confidence of the direction and the magnitude of the treatment effect, but they in no way overcome the generalizability issues of their component RCTs.

The next rung of the evidence ladder is observational data, specifically cohort studies first prospective, then retrospective. I am actually a great fan of observational data, as I have mentioned in the past. Cohort studies give us the opportunity to examine what happens in the real world without imposing artificial conditions necessary in a clinical trial. Observational data can be great when describing epidemiology of a particular disease, the frequency of a given exposure, how different characteristics can modify the relationship between the exposure and the outcome. One of the most attractive features of cohort studies is that the population can be observed over long period of time -- just look at the Nurses' Study, the Framingham Cohort, and others. But these types of studies also have important limitations, and these are readily acknowledged as a heightened susceptibility to bias (especially in the retrospective studies), the possibility of misclassifying important events, and, despite our best efforts to adjust for it, residual confounding. I will come clean and admit my affection for observational data, even despite the fact that it is lower on the totem pole of evidence than an RCT. I really love this paper by Rothman and Greenland that takes a bird's eye view of our research debates. The whole paper really tickles the brain, but I will quote from a section of it briefly here:
Impossibility of Proof
Vigorous debate is a characteristic of modern scientific philosophy, no less in epidemiology than in other areas. Perhaps the most important common thread that emerges from the debated philosophies stems from 18th-century empiricist David Hume’s observation that proof is impossible in empirical science. This simple fact is especially important to epidemiologists, who often face the criticism that proof is impossible in epidemiology, with the implication that it is possible in other scientific disciplines. Such criticism may stem from a view that experiments are the definitive source of scientific knowledge. Such a view is mistaken on at least two counts. First, the nonexperimental nature of a science does not preclude impressive scientific discoveries; the myriad examples include plate tectonics, the evolution of species, planets orbiting other stars, and the effects of cigarette smoking on human health. Even when they are possible, experiments (including randomized trials) do not provide anything approaching proof, and in fact may be controversial, contradictory, or irreproducible. The cold-fusion debacle demonstrates well that neither physical nor experimental science is immune to such problems.

Some experimental scientists hold that epidemiologic relations are only suggestive, and believe that detailed laboratory study of mechanisms within single individuals can reveal cause–effect relations with certainty. This view overlooks the fact that all relations are suggestive in exactly the manner discussed by Hume: even the most careful and detailed mechanistic dissection of individual events cannot provide more than associations, albeit at a finer level. Laboratory studies often involve a degree of observer control that cannot be approached in epidemiology; it is only this control, not the level of observation, that can strengthen the inferences from laboratory studies. Furthermore, such control is no guarantee against error. All of the fruits of scientific work, in epidemiology or other disciplines, are at best only tentative formulations of a description of nature, even when the work itself is carried out without mistakes.
What follows in the hierarchy of evidence are case-control studies, done for some very specific reasons, then case reports and finally expert opinion. When evidence-based guidelines are developed, a comprehensive systematic literature review is undertaken, and all the evidence is examined and ranked. Based on these papers, a recommendation is made and a strength of this recommendation is reported based on the quality of the underlying evidence. This is an arduous and costly process, and it is commendable that it is undertaken. At the same time, given the limitations of the components of the guideline, the final product can be quite inconclusive or even misleading (I hate to bring it up, but look at the screening mammography debate, as well as the recent HRT recommendation reversal). I think it is obvious that I believe in the scientific method, I am simply not convinced that we have done such a great job generating trustworthy evidence in many instances. At the same time, I am not totally nihilistic about what we know, but am somewhere between thinking we have good evidence for a lot of stuff vs. not having any for anything at all.

Allow me one more piece of evidence, if you will, though this is merely anecdotal coming from my dual experience as an author and peer reviewer. I am occasionally floored by the quality of peer review. I have had reviews say on the one hand that of course the paper should be accepted, since it comes from such a reputable group, and on the other reject out of hand papers based on the reviewers' profound lack of understanding of the methods employed. And lest I sound like a crybaby, let me say that I welcome a well-reasoned rejection. What I am talking about is not that. And this is not a surprise, since pretty much anyone can sign up to be a peer reviewer, since, to the best of my knowledge, there is no set of qualifications that journals ask for in their reviewers. And this, so far as I know, applies even to such high caliber publications as JAMA.

So, these are my thoughts on evidence-based medicine. I welcome responses to this, as my understanding of this science is constantly evolving, and differing well-reasoned opinions really help me get a better handle on this stuff.

I will try to tackle my CAM argument next. If I in any way implied in my remarks that I encourage allopathic physicians (by the way, I am not using it in a derogatory way, but merely as it is defined here; in fact, until today I was blissfully unaware of its negative connotation) to be purveyors of CAM, I sincerely apologize. I am pretty sure, however, that I have never made such a statement, as this is not what I believe. My belief is that all modalities that may impact what happens to public's health need to be evaluated for safety, not question. I think we both agree, since there is really no reason to think that something like homeopathy has anything that can help, by the same token we do not believe that it have anything that can hurt. Same with healing crystals, reiki and prayer. So, if a person wants to engage in these activities, and they are perfectly safe physically, be my guest. Other modalities, such as chiropractic, acupuncture, herbalism and the like, definitely need to be evaluated more stringently, as there is reason to think that they may cause harm. And the decisions about their use must be made based on the probability of harm vs. the perceived probability of benefit. Why do I think that these should not be regulated the same way as allopathic treatments? Well, herbs grow naturally and I dare say we have little to say about what our patients grow in their back yards, unless of course the thought is to regulate them the way we do marijuana. As for chiropractic, it is already regulated, though to what extent I am not sure, and would love to hear from someone who knows. Since its techniques most resemble surgical interventions, the level of evidence for them should perhaps be the same as that which we demand in the surgical literature. This is just a thought, and I am not sure that I am correct in this, so other views are, as always, welcomed. What is coming through for me is that perhaps my call to equipoise was a little over the top, as I do not seem to be approaching the above CAM issues in a frequentist, but more in a Bayesian way (though I remain committed to equanimity). Yet, there is something to be said about the frequentist approach, even though it is not my way generally. The frequentist approach, which is what underlies the bulk of our traditional clinical research, does not rely on differential prior probabilities for different possible associations, but treats them all equally. Despite many disadvantages, one obvious advantage is that we do not discount potential associations that do not have biologic plausibility, given our current understanding of biology, and sometimes help us stumble on brand new hypotheses. So, clearly, there is a tension here, and I am still working on what is the better way, if any.

My final words will be about vaccination. It is disheartening to be lumped with "anti-vaxers", as has been done in the comments to Dr. Novella's and Orac's posts. While my bruised ego will survive this insult, I would like to question this assertion. Nowhere have I said that vaccinations are a bad idea or present a real danger to our children. The hype surrounding the vaccination-autism "debate" is abhorrent to me. What I have stated, however, is that I am of the opinion that we have gone a bit overboard with some of them, one being the chicken pox vaccine. Now, this does not make me an "anti-vaxer"; this just makes me a bit skeptical. The way I view the data is that the advantages for this vaccine are mostly economic, in that they prevent parents from missing days at work. Now, I am certainly not opposed to making such a vaccine available to parents who desire it, but I am not convinced that it should be a prerequisite for my kid to go to school. Given that there is always a possibility of an adverse reaction, no matter how small that possibility is, if the risk of it may outweigh the benefit (and here I do not mean the benefit of having mom show up at work), it has to be weighed very carefully. And even though the question asked by one of the comments raises the issue of an immunocompromised child worrying about potentially being exposed to chicken pox, given the known issues with breakthrough disease, I am not sure that immunizing all of his/her contacts would produce anything other than a false sense of security. My sentiment about the HPV vaccine is similar, to respond to another comment. For reasons laid out here and elsewhere on my blog, I am pretty convinced that it would be a compete subversion of the intent of vaccinations to make it into another mandated shot. To date HPV is merely a recommendation, whose validity I am free to question, though last I heard there was movement afoot to make it mandatory. If in your informed opinion your daughter should get vaccinated against HPV, well then I have very little to say about it. But if I were to counsel an individual patient in the context of my understanding of the data, I would be very upfront with my view.

To me the fact that there are such heated debates about this stuff is a testament exactly to how NOT straightforward our science is. I do understand that as a researcher I can afford a certain amount of analysis paralysis that is unacceptable at the bedside. However, I think we (and the press) do a disservice to the patients, to ourselves and the science if we are not upfront about just how uncertain much of what we think we know is. I could not have said it better than this story about Dr. Devereaux's presentation at the recent ASA meeting did here:
It would be nice if we could all agree that science is not static, but rather progresses and regresses. We learn, and then find out that some of what we thought we had learned was wrong, and set about using that information to seek the next level of truth. Repeat, ad infinitum. Personally, I’d love it if my doctors couched every bit of advice with, “Here’s what we think we know today.”
But I suspect that wouldn’t sit well with many patients, who want certainty (as if there is such a thing). And it especially seems like a difficult proposition in our contentious society, where anti-science nay-sayers like to jump on contradictory findings to challenge the basic value of science overall.
That is exactly NOT what I am trying to do. I am merely reflecting on many of the issues that threaten the validity of what we think we know. I am confident that disclosure and transparency not only lead to better science, but they also lead to science that can withstand the test of nay-sayers.

I have come to the end. I am not sure that I have addressed each and every one of the criticisms, though I hope that I have addressed the majority; I am sure you will point out what I missed. A couple of things about comments: Tomorrow I will be in the air most of the day and may not have the opportunity to sign on to approve comments. So, please, if your comment does not go up until Wednesday, do not think that it has been rejected. Also, I really would like to keep it civil and, though I did not apply this rule today, I will not accept any overt insults or name calling from either side of the debate.

I am sincerely looking forward to continuing this discussion!                

          

18 comments:

  1. So the definition of herbal medicine is stuff that grows in peoples backyards? Actually I would agree that an unregulated market in herbal medicine sold by the farmer would probably be fine, if only that regulating that would be impossible anyways. Kind of like what we do with unpasteurized apple juice in some states. And it would probably discourage industrial solvents and bleach being called natural herbs.

    ...but overall that's a pretty silly reason to not regulate herbs on the whole! Setting aside 'regulation' which has all sorts of issues, wouldn't you agree that they should be evaluated by the same standard as all other medicine?

    I'm really glad I barely made it into the chicken pox vaccinated generation (either that or I got chicken pox earlier and didn't show symptoms; glad either way).I mean, facial scars aren't uncommon from chicken pox. Talking about how everyone should be able to make their own decision with vaccines is a good example of rugged American individualism, but its completely naive of herd immunity. Even setting aside the immunocompromised, why should a vaccinated classmate of your child have to share the chicken pox hazing you wish to impose on your own children?

    Anyways stuff like this paragraph:
    "To me the fact that there are such heated debates about this stuff is a testament exactly to how NOT straightforward our science is. I do understand that as a researcher I can afford a certain amount of analysis paralysis that is unacceptable at the bedside. However, I think we (and the press) do a disservice to the patients, to ourselves and the science if we are not upfront about jut how uncertain much of what we think we know is."

    makes me wonder if you read SBM much at all. They are constantly talking about what a complicated process science is. And it is a process, not an answer. Of course the media is pretty much a four-letter word on their blog.

    ReplyDelete
  2. Dear Dr. Zilberberg,

    At Respectful Insolence, you posted:

    http://scienceblogs.com/insolence/2010/10/a_fallacy-laden_attack_on_science-based_medicine.php#comment-2880069

    As I cannot leave a comment on Dr. Novella's blog, please, could you let him know as well?


    1. Dr. Novella's blog's URL is
    http://theness.com/neurologicablog/

    It is a WordPress blog. It is configured to require a log-in.

    2. Log-in registration is easy, but many email clients dump the registration confirmation email into a spam filter. So perhaps if you registered but did not receive a confirmation email it may be stuck in your spam filter.

    3. I encourage you to register for Dr. Novella's blog and engage in comments there, as well.

    4. To me, high uptake rates for all vaccine-preventable illnesses, including those you characterize as minor (varicella or chicken pox) are a social justice issue. The social and economic costs burden of vaccine-preventable diseases falls disproportionately on those least able to pay for them: the poor and the working poor.

    ReplyDelete
  3. "That is exactly NOT what I am trying to do. I am merely reflecting on many of the issues that threaten the validity of what we think we know.<"
    Ah, yes, the "just asking questions" gambit.

    ReplyDelete
  4. Litz Ditz says "The social and economic costs burden of vaccine-preventable diseases falls disproportionately on those least able to pay for them: the poor and the working poor."

    So if it's mainly an altruism issue then why do Science Bloggers fervently argue the benefits and risk of the middle class, when you've just stated that it is the poor and working poor who benefit the most from high vaccine uptake?

    In other words, the poor and working poor statistics are applied to all classes as a homogeneous group. You seem to recognize that the risks are not homogeneous but that everyone, including those who aren't at high risk should vaccinate any because of altruistic or social justice issues. This seems to be more of a philosophical issue than a scientific issue.

    ReplyDelete
  5. I totally love the link to the definition of allopathic medicine! Never saw that before, we'll be sure to pass that around.

    Anyway, you provide a very nice post here and the many people who are watching all love the fact that you're standing up to these psuedo- intellectual bullies.

    I too am growing concerned about the piling on of vaccines. Great, let's eliminate polio but ear infections and chicken pox? C'mon, that's not just for massive profit? Please. People are not dumb just because they go to a naturopath or prefer natural health care and think critically about science, medicine, pharma and what gets forced into in their bodies.

    ReplyDelete
  6. ***I will try to tackle my CAM argument next. If I in any way implied in my remarks that I encourage allopathic physicians (by the way, I am not using it in a derogatory way, but merely as it is defined here; in fact, until today I was blissfully unaware of its negative connotation) to be purveyors of CAM, I sincerely apologize.***

    I strongly recommend you stop using the word "allopathic" if you want to be taken seriously by the critics of quackery. The term doesn't just have a negative connotation; as your linked definition shows, it means "that which is not homeopathy" and was coined by Hahnemann. Today, it is used more broadly to mean "medicine which is limited in its outlook" or as a synonym for "reductionist" or "materialist" and is typically used by alternative medicine practitioners. If you don't want to give the mistaken impressoin that you are yourself a homeopath, don't use the term. It's not even reflective of actual conventional medical practice.

    ***I think we both agree, since there is really no reason to think that something like homeopathy has anything that can help, by the same token we do not believe that it have anything that can hurt.***

    Homeopathy, in the traditional sense, should be harmless except insofar as it causes people to delay effective treatment. However, what is sold today is not strictly homeopathy in the traditional sense. Zicam is 2X zinc -- that means the original zinc preparation is only diluted twice, leaving it still far stronger than is considered safe. The FDA is powerless, now that Zicam's manufacturers persuaded the keepers of the Homeopathic Pharmacopeia to add 2X Zinc to their list. (Seriously, that's all it takes to invoke the homeopathic loophole.) Or consider this, over at Todd W's blog: the FDA is recalling Hyland's Homeopathic Teething Tablets because they contain measurable amounts of deadly nightshade.
    http://silencedbyageofautism.blogspot.com/2010/10/recall-of-hylands-teething-tablets.html

    (Note: as your blog does not permit the blockquote tags, I have enclosed quoted text in asterisks.)

    ReplyDelete
  7. Sorry. You've lost. While I did find the 'tone' of your piece on Paternalism overstated, there is no fairness in confrontational representation. The best one can hope for in a situation is to refuse to be be bullied and quietly stand your ground. Since you have already lost patience with being passive while misrepresented you must accept the necessity of 'treating' these disruptions with the contempt they deserve.

    ReplyDelete
  8. I would like to address two of the issues you discussed. First, as to the regulation of herbal medicines I think this story from The PBS News Hour last Thursday, 10/26 does a better job of explaining why herbal medicines should be regulated than I ever could:

    http://www.pbs.org/newshour/bb/health/july-dec10/herbal_10-21.html

    As to the varicella vaccine, I think you have fallen victim to "I have not seen it, so it must not exist" syndrome that is so prevalent today when it comes to vaccine prevenable diseases such as: pertussis, HIB, polio, diptheria, measles, mumps and rubella. The list of possible complications due to varicella include, but are not limited to: pneumonia, cerebellar ataxia, meningitis post herpetic neuralgia and as MRSA has become more common, resistant secondary bacterial infections. As a resident in training I saw examples of each of those devasting complications and now when I speak to new graduates they have almost never seen them. Additionally, as the rate of vaccination has risen, those who choose to remain unvaccinated are at increased risk of reaching adulthood without immunity and subsequently contracting the disease. These are all benefits in addition to the financial benefit to society you described.

    ReplyDelete
  9. @liz ditz
    I've now tried 3 times to register to comment on Dr Novella's blog without success, there is no email sent, there is no email in the spam box, perhaps Dr Novella needs to check his settings.

    So I wanted to post this comment on that blog but here it is instead.

    I'm not 100% sure but I think it's my comments Dr Novella refers to in the post. He has presented these comments which were directly relevant to the issue at hand as a personal insult, 'that his bedside manner is that of a paternalistic ass'. His reasons for doing that are his own but I encourage readers to go to Dr Zilberberg's first post to read what I actually wrote.

    The only other comment worth making at this time is that Dr Novella certainly uses 'post-modern' as an insult in the way he accuses Dr Zilbeberg of using 'allopathic'.

    ReplyDelete
  10. Dr. Zilberberg,

    Between the testable and the untestable; between science and woo; there can be no middle ground. You're going to have to pick a side.

    Are you a scientist or not?

    ReplyDelete
  11. You may not have meant the term "allopathic medicine as derogatory, but that is indeed, what it is. It is a slight, a slur, a marginalization. Many people these days recognize what it refers to and do not (as you) intend it as a slur, but it remains a belittling term. The "all" in it specifically refers to "all" that other stuff which doctors & nurses do which is inferior to homeopathy. Look, we all know what someone means when they use the N-word; there's no lack of clarity there; hell, they may not even be intending it in a hateful way, but it is a slur, nonetheless.

    ReplyDelete
  12. So be brief:

    1) Those are your problems with SBM - what, if anything, do you propose we use to get more reliable results?

    If you don't have an answer, do you agree that SBM is our best way of testing treatments?

    2) You quote a section on 'Proof' which correctly states the elementary idea that we cannot obtain 'absolute' proof of anything. That granted, do you recognise that SBM gives us the best evidence and closest thing to absolute proof, compared to all other 'ways of knowing'

    ReplyDelete
  13. I have another comment to make, this time regarding the varicella vaccine. You say:

    ***What I have stated, however, is that I am of the opinion that we have gone a bit overboard with some of them, one being the chicken pox vaccine. Now, this does not make me an "anti-vaxer"; this just makes me a bit skeptical. The way I view the data is that the advantages for this vaccine are mostly economic, in that they prevent parents from missing days at work. Now, I am certainly not opposed to making such a vaccine available to parents who desire it, but I am not convinced that it should be a prerequisite for my kid to go to school.***

    You are mistaken that the advantages for the vaccine consist primarily of the parents not missing work. There is certainly a financial argument, but it is far from the primary one. The reason that schools, including private schools and daycare centers, tend to require it has nothing to do with the parent missing work. It's to do with the *child* missing *school*.

    You are an educated person; I'm sure you'll agree that it is not good for a child to miss two weeks of school. That's how much I missed when I had chickenpox, and I didn't even need to be hospitalized or anything. My mom was a stay-at-home mom; economically, it really wasn't any skin off our backs. But I missed a lot of school. Oh, they sent homework and stuff home for me, but I missed the lectures and such, and you can't really make that up. Time is one thing you just can't get back.

    Chickenpox is mild, but highly contagious, and results in a significant period out of school. Missing more than a few days in a row of school is associated (for obvious reasons) with reduced grades. If it's just one kid, well, that's too bad. But with a contagious illness, you could be talking half the student body out with it. The outbreak that got me wasn't that bad, because of good compliance with quarantine protocols. But of my close circle of friends, half were sick, one bad enough to be hospitalized. That's a lot of kids missing school, and that throws off the classroom dynamics.

    In short, schools have very good reasons to require vaccination, and it's nothing to do with parental convenience. It's to do with truancy.

    ReplyDelete
  14. Thank you again Dr. Zilberberg!! I am a mom who has obsessively studied both sides of the vaccine argument and I find you voice both refreshing and reasonable.

    ReplyDelete
  15. "Chickenpox is mild, but highly contagious, and results in a significant period out of school. Missing more than a few days in a row of school is associated (for obvious reasons) with reduced grades."
    Then there is the shingles you get in older age, a nasty often very painful episode that at its worst can leave you visually impaired. Having a good chance of avoiding that alone is worth the vaccination. Ask anyone who has had it.

    ReplyDelete
  16. Gunshot injuries to the aorta are rare, and deadly. There are no randomized controlled trials regarding the management of these injuries I am aware of. Just some case reports and expert opinion.

    So when that 20 year-old is in the trauma bay with a blood pressure of 75/40, what should I tell him?

    "There are no RCTs regarding your injury. I think you need emergent life saving surgery, but I have no absolute proof. It is theoretically possible other forms of treatment such as homeopathy or natural herbs may be superior to surgery. I do not wish to force my paternalistic views upon you. Please research the options and let me know how you would like the hole in your aorta treated."

    Trauma is a difficult area to study. Many treatment approaches have changed significantly in recent decades. The science is not perfect and is often debated within the field. Fortunately there are a lot of dedicated people doing the best they can with the data available, including lowley expert opinion. As a result, your chance of surviving a significant injury continues to improve.

    When I tell a trauma patient the need an operation, nobody argues. No anecdotes about relatives who got better with chiropractic therapy. No conspiracy theories about surgical equipment companies. It's easy to understand that you have a hole in the aorta and it must be fixed. I'm an expert in fixing certain injuries, they trust my expert opinion based on training, experience, and science.

    I feel bad for my pediatrician brother when he tells someone they should vaccinate their children. How could he know that? Obviously, big pharma got to him.

    ReplyDelete
  17. Anonymous, exactly! Just like parachutes, your craft does not lend itself to RCTs in all cases. That is exactly the point! Thanks.

    ReplyDelete
  18. In many cases, scientists aren' t sure what specific ingredient in a particular herb works to treat a condition or illness.

    ReplyDelete