Reading Deborah Blum's "The Poisoner's Handbook" is an intellectual treat. Although non-fiction, it paints in understated sepia tones the crevices of New York City at the dawn of the Industrial Revolution, where bootlegged booze and poisons were fare of the day, homicides went unpunished and the corrupt coroner system basked in the glow of its own willful ignorance and political approval. That is until Charles Norris and Alexander Gettler, two single-minded and tireless men, brought science into the lagging American medical jurisprudence and created the now burgeoning field of forensic medicine.
The chapter on radium in particular sparked my interest. Blum describes in vivid detail the well-known misadventure of the "radium girls", a label given to young women in a watch factory in Orange, NJ, in the early 1920s. She sets up the story with the fascinating background of radium discovery by the Curies and Marie Curie's penchant for carrying a "pet" bottle of radium in her skirt pocket, exhibiting its breathtaking beauty in a circus-like fashion (she died a horrible death from aplastic anemia induced by radiation exposure). Once its tumor shrinking properties became known, it did not take long for entrepreneurs, backed by the medical establishment, to create and sell all kinds of tonics and pills containing radium to the clueless public searching for the fountain of youth. The tragic tale of the radium girls, who, because of occupational ingestion of radium used for painting numbers on the faces of watches (according to Blum's account, the girls were encouraged to lick the paint brushes to make them pointy), and playful applications of this glow-in-the-dark paint on their lips and faces, developed debilitating jaw necrosis and other bony complications and early deaths, delivered a dose of sobriety to the public and policy makers about this new health panacea. Even the gifts of radium to Marie Curie were now delivered in a thick lead shield to contain its homicidal particles.
The story of radium raised all sorts of questions for me. When the element was first discovered, even the scientists could not conceive of its deadly health effects on human tissues. And for this reason there was no caution exercised in its use. What I puzzle over, as you may have guessed from many previous posts, is how we can balance our adoption of new glittering technologies, about which we do not have complete information, and keeping a modicum of caution about their currently unknown potentially adverse effects. I particularly wonder about this in the context of how our brains are wired and of our prevailing concerns for the economy even at the expense of our health.
Humans are seekers. I recently read Jonah Lehrer's "How We Decide", and it made me appreciate just how susceptible we are to the pleasurable effects of dopamine, and how craving its effects drives us to perform irrational acts that will soothe our neurons in a bath of dopamine bubbles. Addiction, the ultimate seeking-and-never-finding behavior, is, at least in part, mediated by dopamine. Does this addiction fuel our drive for innovation as well? And does it also make us throw caution to the wind when a desirable new object, like, say, glow-in-the-dark radium or a smart phone, is within grasp?
On the same side of this equation is the corporate voice, thundering in the background about the importance of innovation, injecting doubt about the potential for untoward effects and invoking the reigning rhetoric of Queen Economy as the ultimate justification. You don't believe me? Just look at the tobacco history, rife with denials, manipulation and lies. And this is exactly what our consumer brain wants to hear. So we paint caution as unscientific alarm and walk away from it, shaking our heads, filled with self-righteousness.
This balance that I am describing is once again the baby and bath water problem. We encounter it in every aspect of our modern lifestyle: the environment and the threat of climate change; the healthcare system with its record technology spending without commensurate results in health; our food system and obesity and superbug epidemics; the galloping pace of technological development, far outpacing our cognitive abilities to incorporate these technologies sensibly into our lives. Simply put, the question becomes, how do we harness innovation without demanding corpses (literally and figuratively) as proof of its potential untoward effects?
The first step is clearly understanding our history, and for this read Blum's book -- you won't be sorry! Next we need awareness of how our brains operate and how these biological principles set well known traps in our reasoning. Using metacognition to understand these pitfalls in thinking may at least put us on a smarter course walking this fine line. Finally, as I have advocated before, we need to stop shouting at each other and start listening. Perhaps we are not so drastically different in our views as the press and politicians will have us believe. After all, we are all susceptible to the same poisons. And dopamine.
Showing posts with label precautionary principle. Show all posts
Showing posts with label precautionary principle. Show all posts
Wednesday, January 5, 2011
Monday, December 13, 2010
Can a "negative" p-value obscure a positive finding?
I am still on my p-value kick, brilliantly fueled by Dr. Steve Goodman's correspondence with me and another paper by him aptly named "A Dirty Dozen: Twelve P-Value Misconceptions". It is definitely worth a read in toto, as I will only focus on some of its more salient parts.
Perhaps the most important point that I have gleaned from my p-value quest is that the word "significance" should be taken quite literally. Here is what Merriam-Webster dictionary says about it:
It is the very meaning in point 2a that the word "significance" was meant to convey in reference to statistical testing. That is "worth noting" or "noteworthy". Nowhere do we find any references to God or truth or dogma. So, the first lesson is to drift away from taking statistical significance as the sign from God that we have discovered the absolute truth, and to focus on the fact that we need to make note of the association. The follow up to the noting action is confirmation or refutation. That is, once identified, this relationship needs to be tested again (and sometimes again and again) before we can say that there may be something to it.
As an aside, how many times have you received comments from peer reviewers saying that what you are showing has already been shown? Yet, in all of our Discussion section we are quite cautious to say that "more research is needed" to confirm what we have seen. So, it seems we -- researchers, editors and reviewers -- just need to get on the same page.
To go on, as the title of the paper states, we are initiated into the 12 common misconceptions about what p-value is not. Here is the table that enumerates all 12 (since the paper is easy to find for no fee, I am assuming that reproducing the table with attribution is not a problem):
Even though some of them seem quite similar, it is worth understanding the degrees of difference, as they provide important insights.
The one I wanted to touch upon further today is Misconception #12, as it dovetails with our prior discussion vis-a-vis environmental risks. But before we do this, it is worth defining the elusive meaning of the p-value once again: "The p-value signifies the probability of obtaining the association (or difference) of the magnitude obtained or one of greater magnitude when in reality there is no association (or difference)". So, let's apply this to an everyday example of smoking and lung cancer risk. Let's say a study shows a 2-fold increase in lung cancer among smokers compared to non-smokers, and the p-value for this association is 0.06. What this really means is that "under conditions of no true association between smoking and lung cancer, there is a 6% or less chance that a study would find a 2-fold or greater increase in cancer associated with smoking". Make sense? Yet, according to the "rules" of statistical significance, we would call this study negative. But is this a true negative? (To the reader of this blog this is obvious, but assure you that, given how cursory our reading of the literature tends to be, and how often I hear my peers discount findings with the careless "But the p-value was not significant", this is a point worth harping on).
The bottom line answer to this is found in the discussion of the bottom line misconception in the Table: "A scientific conclusion or treatment policy should be based on whether or not the p-value is significant". I would like to quote directly from Goodman's paper here, as it really drives home the idiocy of this idea:
Taken together, all these points merely confirm my prior assertion that we need to be a lot more cautious about calling results negative when deciding about potentially risky exposures than about beneficial ones. Similarly, we need to set a much higher bar for all threats to validity in studies designed to look at risky rather than beneficial outcomes (more on this in a future post). These are the principles we should be employing when evaluating environmental exposures. This becomes particularly critical in view of the startling revelations of the genome-wide association experiments findings that our genes determine a very small minority of diseases to which we are subject. This means that the ante has been upped dramatically for environmental exposures as culprits, and demands a much more serious push for the precautionary principle as the foundation for our environmental policy.
Perhaps the most important point that I have gleaned from my p-value quest is that the word "significance" should be taken quite literally. Here is what Merriam-Webster dictionary says about it:
sig·nif·i·cance
noun \sig-ˈni-fi-kən(t)s\
Definition of SIGNIFICANCE
1
a : something that is conveyed as a meaning often obscurely or indirectly
b : the quality of conveying or implying
2
a : the quality of being important : moment
b : the quality of being statistically significant
As an aside, how many times have you received comments from peer reviewers saying that what you are showing has already been shown? Yet, in all of our Discussion section we are quite cautious to say that "more research is needed" to confirm what we have seen. So, it seems we -- researchers, editors and reviewers -- just need to get on the same page.
To go on, as the title of the paper states, we are initiated into the 12 common misconceptions about what p-value is not. Here is the table that enumerates all 12 (since the paper is easy to find for no fee, I am assuming that reproducing the table with attribution is not a problem):
Even though some of them seem quite similar, it is worth understanding the degrees of difference, as they provide important insights.
The one I wanted to touch upon further today is Misconception #12, as it dovetails with our prior discussion vis-a-vis environmental risks. But before we do this, it is worth defining the elusive meaning of the p-value once again: "The p-value signifies the probability of obtaining the association (or difference) of the magnitude obtained or one of greater magnitude when in reality there is no association (or difference)". So, let's apply this to an everyday example of smoking and lung cancer risk. Let's say a study shows a 2-fold increase in lung cancer among smokers compared to non-smokers, and the p-value for this association is 0.06. What this really means is that "under conditions of no true association between smoking and lung cancer, there is a 6% or less chance that a study would find a 2-fold or greater increase in cancer associated with smoking". Make sense? Yet, according to the "rules" of statistical significance, we would call this study negative. But is this a true negative? (To the reader of this blog this is obvious, but assure you that, given how cursory our reading of the literature tends to be, and how often I hear my peers discount findings with the careless "But the p-value was not significant", this is a point worth harping on).
The bottom line answer to this is found in the discussion of the bottom line misconception in the Table: "A scientific conclusion or treatment policy should be based on whether or not the p-value is significant". I would like to quote directly from Goodman's paper here, as it really drives home the idiocy of this idea:
This misconception encompasses all of the others. It is equivalent to saying that the magnitude of effect is not relevant, that only evidence relevant to a scientific conclusion is in the experiment at hand, and that both beliefs and actions flow directly from the statistical results. The evidence from a given study needs to be combined with that from prior work to generate a conclusion. In some instances, a scientifically defensible conclusion might be that the null hypothesis is still probably true even after a significant result, and in other instances, a nonsignificant P value might still lead to a conclusion that a treatment works. This can be done formally only through Bayesian approaches. To justify actions, we must incorporate the seriousness of errors flowing from the actions together with the chance that the conclusions are wrong.When the author advocates Bayesian approaches, he is referring to the idea that a positive result in the setting of a low pre-test probability still has a very low chance of describing a truly positive association. This is better illustrated by the Bayes theorem, which allows us to quantify the result at hand ("posterior probability") to what the bulk of prior evidence and/or thought has indicated about the association ("prior probability"). This implies that the lower our prior probability, the less convinced we can be by a single positive result. As a corollary, the higher our prior probability for an association, the less credence we can put in a single negative result. So, Bayesian approach to evidence, as Goodman indicates here, can merely move us in the direction of either a greater or a lesser doubt about our results, NOT bring us to the truth or falsity.
Taken together, all these points merely confirm my prior assertion that we need to be a lot more cautious about calling results negative when deciding about potentially risky exposures than about beneficial ones. Similarly, we need to set a much higher bar for all threats to validity in studies designed to look at risky rather than beneficial outcomes (more on this in a future post). These are the principles we should be employing when evaluating environmental exposures. This becomes particularly critical in view of the startling revelations of the genome-wide association experiments findings that our genes determine a very small minority of diseases to which we are subject. This means that the ante has been upped dramatically for environmental exposures as culprits, and demands a much more serious push for the precautionary principle as the foundation for our environmental policy.
Wednesday, November 17, 2010
Some implications of biologic plausibility
Ever since my... ahem... skirmish... with the folks over at the SBM, I have been contemplating the issue of biologic plausibility. They contend that our tax dollars are wasted by being allocated to the NCCAM to pursue research into CAM. Their reasoning is that there is no biological plausibility to any of it having any therapeutic effect. Now, this is a big bite to swallow. As I have said before there is CAM and then there is CAM. CAM seems to be a convenient wastebasket of modalities that we feel justified in bashing as "woo" since there is limited scientific evidence behind them. But really, I am more willing to give acupuncture and massage the benefit of the doubt than, say, healing crystals (even though I confess I really like rocks!).
So, what of this biologic plausibility, and who came up with it anyway? And is it truly fiscally irresponsible, and possibly even unethical, to test interventions that do not fit our biological plausibility criteria? As a corollary, is there a level of our understanding of biology that makes testing equally wasteful or even unethical? And finally, should plausibility of benefit and harm be required to reach the same evidentiary bar?
For the definition of biological plausibility we apparently thank the milestone 1964 Surgeon General's report linking smoking to cancer. This report was the first official US government document to state that there was enough evidence to implicate cigarette smoking in the rise in lung cancer and cancer deaths. Since the limitations of observational research were used by the critics for decades to derail this definitive statement, the report itself does a nice job laying out the methodologic considerations and the need to rely on the Bradford-Hill criteria. It was in the "coherence" criterion that biologic plausibility entered the picture.
A quick check of my favorite crowd-sourced information site, the Wikipedia, uncovers this treasure from Sir Bradford Hill himself:
Now, on to the question of whether there exist relationships with such high biologic plausibility that they do not require irrefutable proof. Well, how about tobacco and its health effects? How about radiation exposure? Now, how about what we know today about the evolution of microbial resistance to antibiotics? Is it enough that the biologic plausibility for ill-effects of antibiotics in our food chain is strong? Can we now stop the madness? If my colleagues over at SBM are given to the same logic, they would say yes to this. However, extrapolating from this post about organic food production, I somehow think that they would not. So, I am guessing that, although they believe that lack of biologic plausibility should preclude attempts at study, they will nevertheless be reluctant to set a threshold for biologic plausibility that might obviate the need for further research. I am just guessing, and would love to hear what they really think.
And finally, what of the plausibility of benefit vs. that of harm? Should our bar for biologic plausibility for harm be lower than that for benefit? Well, the question really boils down to this: How many bodies do we need to see lying in the streets before we concede that there is a problem? My point is that we Americans have a hard time subscribing to the precautionary principle, applied generously in other parts of the world. If we were a tad less reckless with our need for irrefutable evidence, how many decades of equivocation about tobacco and cancer would we have avoided? How many lives might have been saved? Biologic plausibility for the connection was known even in the 1930s, yet it took another three decades for us to act. What are we obfuscating today that will come back to bite us (and our children) tomorrow? Could it be the cynical injection of doubt that our food production system is causing irreversible damage to us and life around us?
So, what I am saying is that biologic plausibility has several facets. We have to admit humbly that its assumption relies on our necessarily incomplete knowledge, and denying this may prevent us from awe-inspiring discoveries that will advance science in leaps. However, if we feel strongly about the need for it in order to justify our research allocation, some careful soul searching is in order for those thresholds of probability, especially of harm, where we may admit that science makes us sure enough, and, instead of awaiting perfect evidence, we must act promptly.
So, what of this biologic plausibility, and who came up with it anyway? And is it truly fiscally irresponsible, and possibly even unethical, to test interventions that do not fit our biological plausibility criteria? As a corollary, is there a level of our understanding of biology that makes testing equally wasteful or even unethical? And finally, should plausibility of benefit and harm be required to reach the same evidentiary bar?
For the definition of biological plausibility we apparently thank the milestone 1964 Surgeon General's report linking smoking to cancer. This report was the first official US government document to state that there was enough evidence to implicate cigarette smoking in the rise in lung cancer and cancer deaths. Since the limitations of observational research were used by the critics for decades to derail this definitive statement, the report itself does a nice job laying out the methodologic considerations and the need to rely on the Bradford-Hill criteria. It was in the "coherence" criterion that biologic plausibility entered the picture.
A quick check of my favorite crowd-sourced information site, the Wikipedia, uncovers this treasure from Sir Bradford Hill himself:
It will be helpful if the causation we suspect is biologically plausible. But this is a feature I am convinced we cannot demand. What is biologically plausible depends upon the biological knowledge of the day. To quote again from my Alfred Watson Memorial Lecture [1962], there was
- "…no biological knowledge to support (or to refute) Pott’s observation in the 18th century of the excess of cancer in chimney sweeps. It was lack of biological knowledge in the 19th that led to a prize essayist writing on the value and the fallacy of statistics to conclude, amongst other “absurd” associations, that 'it could be no more ridiculous for the strange who passed the night in the steerage of an emigrant ship to ascribe the typhus, which he there contracted, to the vermin with which bodies of the sick might be infected.' And coming to nearer times, in the 20th century there was no biological knowledge to support the evidence against rubella."
In short, the association we observe may be one new to science or medicine and we must not dismiss it too light-heartedly as just too odd. As Sherlock Holmes advised Dr. Watson, "when you have eliminated the impossible, whatever remains, however improbable, must be the truth."[1]Aha, so biologic plausibility is a function of the state of our current knowledge, today. By this litmus test, Marshall and Warren should have been laughed out of all funding agencies. Instead, they rewrote our understanding of what can live in the stomach, and how a microorganism can cause peptic ulcer disease and stomach cancer. And got themselves a cool Nobel to boot. So much for the ethics and finances of biologic plausibility informing meaningful research.
Now, on to the question of whether there exist relationships with such high biologic plausibility that they do not require irrefutable proof. Well, how about tobacco and its health effects? How about radiation exposure? Now, how about what we know today about the evolution of microbial resistance to antibiotics? Is it enough that the biologic plausibility for ill-effects of antibiotics in our food chain is strong? Can we now stop the madness? If my colleagues over at SBM are given to the same logic, they would say yes to this. However, extrapolating from this post about organic food production, I somehow think that they would not. So, I am guessing that, although they believe that lack of biologic plausibility should preclude attempts at study, they will nevertheless be reluctant to set a threshold for biologic plausibility that might obviate the need for further research. I am just guessing, and would love to hear what they really think.
And finally, what of the plausibility of benefit vs. that of harm? Should our bar for biologic plausibility for harm be lower than that for benefit? Well, the question really boils down to this: How many bodies do we need to see lying in the streets before we concede that there is a problem? My point is that we Americans have a hard time subscribing to the precautionary principle, applied generously in other parts of the world. If we were a tad less reckless with our need for irrefutable evidence, how many decades of equivocation about tobacco and cancer would we have avoided? How many lives might have been saved? Biologic plausibility for the connection was known even in the 1930s, yet it took another three decades for us to act. What are we obfuscating today that will come back to bite us (and our children) tomorrow? Could it be the cynical injection of doubt that our food production system is causing irreversible damage to us and life around us?
So, what I am saying is that biologic plausibility has several facets. We have to admit humbly that its assumption relies on our necessarily incomplete knowledge, and denying this may prevent us from awe-inspiring discoveries that will advance science in leaps. However, if we feel strongly about the need for it in order to justify our research allocation, some careful soul searching is in order for those thresholds of probability, especially of harm, where we may admit that science makes us sure enough, and, instead of awaiting perfect evidence, we must act promptly.
Subscribe to:
Posts (Atom)