Reviewing medical literature, part 4: Statistical analyses -- measures of central tendency

Well, we have come to the part of the series you have all been waiting for: discussion of statistics. What, you are not as excited about it as I am? Statistics are not your favorite part of the study? I am frankly shocked! But seriously, I think this is the part that most people, both lay public and professionals, find off-putting. But fear not, for we will deconstruct it all in simple terms here. Or obfuscate further, one or the other.

So, let's begin with a motto of mine: If you have good results, you do not need fancy statistics. This goes along with the ideas in math and science that truth and computational beauty go hand in hand. So, if you see something very fancy that you have never heard of, be on guard for less than important results. This, of course, is just a rule of thumb, and, as such, will have exceptions.

The general questions I like to ask about statistics are 1). Are the analyses appropriate to the study question(s), and 2). Are the analyses optimal to the study question(s). The first thing to establish is the integrity and completeness of the data. If the authors enrolled 365 subjects but were only able to analyze 200 of them, this is suspicious. So, you should be able to discern how complete the dataset was, and how many analyzable cases there were. A simple litmus test is that if more than 15% of the enrolled cases did not have complete data for analysis or dropped out of the study for other reasons, the study becomes suspect for a selection bias. The greater the proportion of dropouts, the greater the suspicion.

Once you have established that the set is fairly complete, move on to the actual analyses. Here, first thing is first: the authors need to describe their study group(s); hence, descriptive statistics. Usually this includes so-called "baseline characteristics", consisting of demographics (age, gender, race), comorbidities (heart failure, lung disease, etc.), and some measure of the primary condition in question (e.g., pneumonia severity index [PSI] in a study of patients with pneumonia). Other relevant characteristics may be reported as well, and this is dependent on the study question. As you can imagine, categorical variables (once again, these are variables that have categories, like gender or death) are expressed as proportions or percentages, while continuous ones (those that are on a continuum, like age) are represented by their measures of central tendency.

It is important to understand the latter well. There are three major measures of central tendency: mean, median and mode. The mean is the sum of all individual values of a particular variable divided by the number of values. So, mean age among a group of 10 subjects would be calculated by adding all 10 individual ages and then dividing by 10. The median is the value that occurs in the middle of a distribution. So, if there are 25 subjects with ages ranging from 5 to 65, the median value is the one that occurs in subject number 13 when subjects are arranged in ascending or descending order by age. The mode, a measure used least frequently in clinical studies, signifies, somewhat paradoxically, the value in a distribution that occurs most frequently.

So, let's focus on the mean and the median. The mean is a good representation of the central value in a normal distribution. Also referred to as a bell curve (yes, because of its shape), or a Gaussian distribution, in this type of a distribution there are roughly equal numbers of points to the left and to the right of the mean value. It looks like this (from wikimedia.org):

For a distribution like the one above it hardly matters which central value is reported, the mean or the median, as they are the same or very similar to one another. Alas, most descriptors of human physiology are not normally distributed, but are more likely to be skewed. Skewed means that there is a tail at one end of the curve or the other (figure from here):

For example, in my world of health economics, many values for such variables as length of stay and costs spread out to the right of the center, similar to the blue curve in the right panel of the above figure. In this type of a distribution the mean and the median values are not the same, and they tell you different things. While the median gives you an idea of the central tendency of the entire distribution, the mean will tell you the central tendency of the majority of the distribution that is tightly clustered at the end opposite the tail. For a distribution similar to the one in the right panel, the mean will underestimate the central measure.

To round out the discussion of central values, we need to say a few words about scatter around these values. Because they represent a population and not a single individual, measures of central tendency will have some variation around them that is specific to the population. For a mean value, this variation is usually represented by standard deviation (SD), though sometimes you will see a 95% confidence interval as the measure of the scatter. Variation around the median is usually expressed as the range of values falling into the central one-half of all the values in the distribution, discarding the 25% at each end, or the interquartile range (IQR 25, 75) around the median. These values represent the stability and precision of our estimates and are important to look for in studies.

We'll end this discussion here for the moment. In the next post we will tackle inter-group differences and  hypothesis testing.      

Getting beyond the p-value

Update 12/8/10, 9:30 AM: I just got an e-mail from Steve Goodman, MD, MHS, PhD, from Johns Hopkins about this post. Firstly, my apologies for getting his role at the Annals wrong -- he is the Senior Statistical Editor for the journal, not merely a statistical reviewer. I am happy to report that he added more fuel to the p-value fire, and you are likely to see more posts on this (you are overjoyed, right?). So, thanks to Dr. Goodman for his input this morning!

Yesterday I blogged about our preference to avoid false positive associations at the expense of failing to detect some real associations. The p value conundrum, where the arbitrary statistical significance is set at <0.05 has bothered me for a long time. I finally got curious enough to search out the origins of the p value. Believe it or not, the information was not easy to find. I have at lest 10 biostatistics or epidemiology textbooks on the shelves of my office -- not one of them went into the history of the p value threshold. But Professor Google came to my rescue, and here is what I discovered.

Using a carefully crafted search phrase, I found a discussion forum on WAME, World Association of Medical Editors, which I felt represented a credible source. Here I discovered a treasure trove of information and references to what I was looking for. Specifically, one poster referred to Steven Goodman's work, which I promptly looked up. And by the way, Steven Goodman, as it turns out is a statistical reviewer the Senior Statistical Editor for the Annals of Internal Medicine and a member of WAME. So, I went to this gem in the journal Epidemiology from May 2001, called unpretentiously "Of P-values and Bayes: A Modest Proposal". I have to say that some of the discussion was so in the weeds that even I have to go back and reread it several times to understand what the good Dr. Goodman is talking about. But here are some of the more salient and accessible points.

The author begins by stating his mixed feelings about the p-value:

I am delighted to be invited to comment on the use of P-values, but at the same time, it depresses me. Why? So much brainpower, ink, and passion have been expended on this subject for so long, yet plus ca change, plus c'ést le meme chose- the more things change, the more they stay the same. The references on this topic encompass innumerable disciplines, going back almost to the moment that P-values were introduced (by R.A. Fisher in the 1920s). The introduction of hypothesis testing in 1933 precipitated more intense engagement, caused by the subsuming of Fisher's significance test into the hypothesis test machinery.^1-9 The discussion has continued ever since. I have been foolish enough to think I could whistle into this hurricane and be heard. ^10-12 But we (and I) still use P-values. And when a journal like Epidemiology takes a principled stand against them, ¹³ epidemiologists who may recognize the limitations of P-values still feel as if they are being forced to walk on one leg. ¹⁴

So, here we learn that the p-value is something that has been around for 90 years and was brought into being by the father of frequentist statistics R.A. Fisher. And the users are ambivalent about it, to say the least. So, why, Goodman asks, continue to debate the value of the p-value (or its lack)? And here is the reason: publications.

Let me begin with an observation. When epidemiologists informally communicate their results (in talks, meeting presentations, or policy discussions), the balance between biology, methodology, data, and context is often appropriate. There is an emphasis on presenting a coherent epidemiologic or pathophysiologic story, with comparatively little talk of statistical rejection or other related tomfoolery. But this same sensibility is often not reflected in published papers. Here, the structure of presentation is more rigid, and statistical summaries seem to have more power. Within these confines, the narrative flow becomes secondary to the distillation of complex data, and inferences seem to flow from the data almost automatically. It is this automaticity of inference that is most distressing, and for which the elimination of P-values has been attempted as a curative.

This is clearly a condemnation of the way we publish: it demands a reductionist approach to the lowest common denominator, in this case the p-value. Much like our modern medical paradigm, the p-value does not get at the real issues:

I and others have discussed the connections between statistics and scientific philosophy elsewhere, ^11,12,15-22 so I will cut to the chase here. The root cause of our problem is a philosophy of scientific inference that is supported by the statistical methodology in dominant use. This philosophy might best be described as a form of naïve inductivism,²³ a belief that all scientists seeing the same data should come to the same conclusions. By implication, anyone who draws a different conclusion must be doing so for nonscientific reasons. It takes as given the statistical models we impose on data, and treats the estimated parameters of such models as direct mirrors of reality rather than as highly filtered and potentially distorted views. It is a belief that scientific reasoning requires little more than statistical model fitting, or in our case, reporting odds ratios, P-values and the like, to arrive at the truth. [emphasis mine]

Here is a sacred scientific cow that is getting tipped! You mean science is not absolute? Well, no, it is not, as the readers of this blog are amply aware. Science at best represents a model of our current understanding of the Universe, it builds upon itself usually in one direction, and it rarely gives an asymptotic approximation of what is really going on. Merely our current understanding of reality, given the tools we have at our disposal. Goodman continues to drive home the naïveté of our inductivist thinking in the following paragraph:

How is this philosophy manifest in research reports? One merely has to look at their organization. Traditionally, the findings of a paper are stated at the beginning of the discussion section. It is as if the finding is something derived directly from the results section. Reasoning and external facts come afterward, if at all. That is, in essence, naïve inductivism. This view of the scientific enterprise is aided and abetted by the P-value in a variety of ways, some obvious, some subtle. The obvious way is in its role in the reject/accept hypothesis test machinery. The more subtle way is in the fact that the P-value is a probability - something absolute, with nothing external needed for its interpretation.

In fact the point is that the p-value is exactly NOT absolute. The p-value needs to be judged relative to some other standard of probability, for example the prior probability of an event. And yet what do we do? We worship at the altar of the p-value without giving any thought to its meaning. And this is certainly convenient for those who want to invoke evidence of absence of certain associations, such as toxic exposures and health effects, for example, when the reality simply indicates absence of evidence.

The point is that we need to get beyond the p-value and develop a more sophisticated, nuanced and critical attitude toward data. Furthermore, regulatory bodies need to get a more nuanced way of communicating scientific data, particularly data evidencing harm, in order not to lose credibility with the people. Most importantly, however, we need to do a better job training researchers on the subtleties of statistical analyses, so that the p-value does not become the ultimate arbiter of the truth.

Paternalism of "science-based" medicine

This is a post I did originally for BMJ's doc2doc site. I am cross-posting it here for your convenience.

Lately I find myself getting more and more incensed with colleagues who seem to preach science-based medicine as the be-all and end-all in the truth about everything. I have blogged about this many times, and some of the references I make below are expanded upon in some of my recent posts. 

The argument goes like this: We who practice science-based medicine are the only ones that are correct in our approach because everything we do comes from a scientifically derived pool of evidence. Anything else is quackery, also referred to as "woo", which includes all modalities that do not fall in line with this EBM paradigm. They rail equally against healing crystal charlatanism, acupuncture, Reiki, and "anti-vaxers", lumping them all into the ignorant masses that dare go against what we "know" at their own and societal peril. They maintain that there is not room for anything in healthcare other than the "proven" scientific interventions.

So, if they advocate for science as the prevailing force in medicine, why should I, a firm believer in evidence as the backbone of healthcare, object? Well, here are some of my reasons, and let's see where you stand.

First of all, it is my belief that all interventions should be approached with equanimity, if not equipoise. Although I am quite dubious that either healing crystals or Reiki can produce actual results, I do not want to confuse the absence of any evidence to this effect with the evidence of absence of the effect. Although I am not that interested in allocating resources to studying these fields, it would be paternalistic of me to bar their further investigation. So the society can decide what it wants to do with them, and in the meantime every individual can make her/his own choice whether to spend their money on them. Interventions in the broad area of traditional Chinese medicine (TCM), including herbs and acupuncture, elicit less visceral aversion in me, and may be shown down the road to have desired health effects, though their study may not be amenable to our current methodologies. Again, at this time, if an individual wants to try them out, they should be keenly aware of how little we explicitly understand about them, including both their risks and potential benefits. 

The vaccine debate is yet another completely different issue altogether. In our privileged society the specter of infectious epidemics is for the most part but a distant memory, yet our concerns for the safety of our children combined with a rampant, albeit not altogether unjustified, distrust of the pharmaceutical industry and the government, have colluded to promote a biased yet durable suite of misinformation about the risk-benefit profile of vaccinations. Some of this prevailing sentiment may be because, in my opinion, we have focused on the wrong outcomes (e.g., the economics rather than morbidity associated with chicken pox), or that we have overstated what we really know and understand, at the risk of sounding too confident and therefore not trustworthy partners in this important area of public health. But these are just my own speculations. What is more important is what we do know about "science-based" medicine.

The uncomfortable truth about clinical science is that by its nature it is inexact. If it were precise, we would be able to describe it through simple mathematics without resorting to statistics, where the wild card is probability of an event. Relying on statistics is wise to understand what happens in large groups of subjects; it is altogether a mistake to take this information for God's own truth for any individual. As I have explained before, statistics are akin to that joke about assuming a spherical cow in order to help a farmer with his milk production quandary. The assumptions of normality, the dark art of log transformations, the cramming of a binary individual into a continuous distribution of population probabilities are all part and parcel of how we develop our understanding in the clinical sciences. No wonder when we bring this knowledge to the bedside a lot of the time it does not seem to apply to the specimen before us, and we are faced with having to use our judgment much more than the science-based medicine mavens are willing to admit. I often wonder how a 2-week prolongation in median survival in a cancer population, even if statistically distinguishable from the much-maligned placebo effect, is in practical terms all that useful. Yet these are effect sizes that we often see in science-based medicine.

My intent is not to belittle what clinical science has and continues to bring to the society. After all, thanks to it, we have a much better understanding of many illnesses now, and we can even take credit for effecting better outcomes for many a sick individual. At the same time we really need to sow humbleness in place of arrogance in all the places where the latter runs rampant. Philosophers of science have understood for decades that there really is no universal truth, and everything we think we know we understand through the prism of our collective human and individual experience. So, instead of wasting time on throwing boulders at our competition, why not take a closer look at the material that our house is made of. Dropping our inveterate paternalism in favor of being truthful about the degree of our uncertainty may make us look more human and fallible, yes. At the same time, it may show our patients and the public at large that we are on the same side. Though this is a risk, it is a necessary one to repair our rapidly crumbling therapeutic relationship. Admitting to not knowing is not only not a weakness, it is the only way to scientific progress.