Data mining: It's about research efficiency.

I have taken a little break from my reviewing literature series -- work has superseded all other pursuits for a little while. But I did want to do a brief post today, since this JAMA Commentary really intrigued me.

First thing that interested me was the authors. Now, I know who Benjamin Djulbegovic is -- you have to live under a rock as an outcomes researcher not to have heard of him. But who is Mia Djulbegovic? It is an unusual enough surname to make me think that she is somehow related to Benjamin. So, I queried the mighty Google, and it spat out 1,700 hits like nothing. But only one was useful in helping me identify this person, and that was a link to her paper in BMJ from 2010 on prostate cancer screening. On this paper (her only one listed on Medline so far), she is the first author, and her credentials are listed as "student", more specifically in the Department of Urology at the University of Florida College of Medicine in Gainesville, FL. The penultimate author on the paper is none other than Benjamin Djulbegovic, at the University of South Florida in Tampa, FL. So, I am surmising from this circumstantial evidence that Mia is Benjamin's kid who is either a college or a medical student. Why does this matter? Well, there seem to be so few papers in high impact journals that are authored by people without an advanced degree, let alone in the first position, that I am in awe of this young woman, now with two major journals to her name -- BMJ and JAMA. This is evidence that parental mentorship counts for a lot (assuming that I am correct about their relationship). But regardless, kudos to her!

Secondly, the title of the essay really grabbed me: what is the "principle of question propagation", and what does it have to do with comparative effectiveness research (CER) and data mining? Well, basically, the principle of question propagation is something we talk about here a lot: questions beget questions, and the further you go down any rabbit hole, the more detailed and smaller the questions become. This is the beauty and richness of science as well as what I have referred to as "unidirectional skepticism" of science, meaning that a lot of the time, building on existing concepts, we just continue down the same direction in a particular research pursuit. This is why Max Planck was right when he said

A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

So, yes, we build upon previous work, and continue our journey down a single rabbit hole our entire career. Though of course there are countless rabbit holes all being explored at the same time. It is really more of a fractal-like situation than a single linear progression. What is clear, as the authors of the Commentary point out, is that this results in the ever-escalating theoretical complexity of scientific concepts. What does this have to do with anything? This, the authors state, argues for continued use of theory driven hypothesis testing, given that medical knowledge will forever be incomplete. And this brings them to data mining.

Here is where I get a little confused and annoyed. They caution the powers that be from consigning all clinical research to data mining, at the expense of more rigorous studies to pursue hypothesis testing. They argue that mining data that already exist is limiting precisely because it is constrained by the scope of our current knowledge, and that we cannot use these data to generate new associations and new treatment paradigms. They further state that emerging knowledge will require updating these data sets with new data points, and this, according to the authors

...creates a paradox, which is particularly evident when searching for treatment effects insubgroups—one of the purported goals of the IT CER initiative. As new research generates new evidence of the importance for tailoring treatments to a given subpopulation of patients, the existing databases will need to be updated, in turn undermining the original purpose to discover new relationships via existing records.

Come agin? And then they say that "consequently, the data mining approach can never result in credible discoveries that will obviate the need for new data collection". Mmhm, and so? Is this the punch line? Well, OK, they also say that because of all this we will still need to do hypothesis testing research. Is this not self-evident?

I don't know about you, but I have never thought that retrospective data mining would be the only answer to our research needs. Rather, the way to view this type of research is as an opportunistic pursuit of information from massive repositories of existing data. We can look for details that are unavailable in the interventional literature, zoom in on the potentially important bits, and use this information to inform more focused (and therefore pragmatically more realistic) interventional studies.

Don't take me wrong, I am happy that the Djulbegovics published this Commentary. It is really designed more as an appeal to policy makers, who, in their perennial search for one-size-fit-all panaceas, may misinterpret our zeal for data mining as the singular answer to all our questions. No indeed, hypothesis testing will continue. But using these vast repositories of data should make us smarter and more efficient at asking the right questions and designing the appropriate studies to answer them. And then generate further questions. And then answer those. And then... Well, you get the picture.

Assume a spherical cow

You know that joke about the farmer whose cows are not producing enough milk? A university panel gathers under the leadership of a theoretical physicist. They analyze each aspect of the problem thoroughly and carefully, and after much deliberation produce a report, the first line of which is "First, assume a spherical cow in a vacuum". This joke has become short-hand for some of the reductionist thinking in theoretical physics, but it can just as easily personify the field of statistics, upon which we rely so heavily to inform our evidence-based practices.

Here is what I mean. Let us look at four common applications of statistical principles. First, descriptive statistics, usually represented by Table 1 in a paper, in which we are interested in understanding the measures of central tendency such as the mean and median values. I would argue that both measures are somewhat flawed in the real world. As we all learned, a mean is a bona fide measure of central tendency in a normal or Gaussian distribution. And furthermore, in such a distribution 95% confidence intervals are bounded by 2.5x the standard deviation from the mean. Although this is very convenient, few things about human biology are actually distributed normally, while many cluster to the left or to the right of the center creating a tail on the opposite side. For these skewed distributions a median is the recommended measure to be used, bracketed by the boundaries of the 25% to 75% range of the values, or the interquartile range. But in a skewed distribution it is exactly the tail that is its most telling feature, as described so eloquently and personally by Stephen J. Gould in his "The Median Isn't the Message" essay. This is especially true when more specific identification of characteristics precipitates the numbers dwindle; e.g., if instead of such general measures as blood pressure among all-comers you want to focus on morning blood pressures in such specific group as African-American males with a history of diabetes and hypercholesterolemia who attend a smoking cessation program, say. And this is important because at the level of the office encounter each patient is a universe of his/her own risk factors and parameters that does not necessarily obey the rules of the herd.  

Second, analytic modeling relies on the assumption of normality. To overcome this limitation we transform certain non-normal distributions to their log forms, for example, to normalize them in order to force them to play nicely. Once normalized, we perform the prestidigitation of a regression, reverse-transform the outcome to its anti-log, and, voila, we have God's own truth! And even though statisticians tend to argue about such techniques, in the real world of research, where evidence cannot wait for perfection, we accept this legerdemain as the best we can do.

The next example I will use is that of pooled analyses, specifically meta-analyses. The intent of meta-analyses is to present the totality of evidence in a single convenient and easily comprehended value. One specific circumstance in which a meta-analysis is considered useful is when different studies are in conflict with one another, as for example, when one study demonstrates that an intervention is effective, while another does not show such effectiveness. In my humble opinion, it is one thing to pool data when studies suffer from Type II error due to small sample sizes. However, what if the studies simply show opposite results? That is, what if one study indicates a therapeutic advantage of treatment T over placebo P, but another shows the exact opposite? Is it still valid to combine these studies to get at the "true" story? Or is it better to leave them separate and try to understand the potential inherent differences in the study designs, populations, interventions, measurements, etc.? I am not saying that all meta-analyses mislead us, but I do think that in the wrong hands this technique can be dangerous, smoothing out differences that are potentially critical. This is one spherical cow that needs to be milked before it is bought.    

The last cow is truly spherical, and that is the one-on-one patient-clinician encounter, wherein the doctor needs to cram the individual patient with his/her binary predispositions into the continuous container of evidence. It is here that all of the other cows are magnified to obscene proportions to create a cumbersome, at times incomprehensible and frequently useless pile of manure. It is one thing for a clinician to ignore evidence willfully; it is entirely another to be a conscientious objector to what is known but not applicable to the individual in the office.

But let's remember that I, as a health services researcher and a self-acknowledged sustainability zealot, am in awe of manure's life-giving properties. Extending the metaphor then, this pile of manure can and should be used to fertilize our field of knowledge by viewing each therapeutic encounter systematically as its own experiment. The middle way is that between the cynicism of discarding and blind acceptance of any and all evidence. That is where the art of medicine must come in, and the emerging payment systems must take it into account. Doctors need time, intellectual curiosity and skills to conduct these "n of 1" trials in their offices. If the fire hose of new data insists on staying on, this is the surest way to direct "conscientious and judicious" application of the resulting oceans of population data in the service of our public's health. And that should make the farmer happy, even if his cows are spherical.