Thursday, December 10, 2009

When the black box of evidence is empty

A close friend of mine gave birth a few days ago to her second child. My friend is a slight woman whose infant came out at 6 lb 9 oz, even though she was calculated to be at 36 weeks' gestation. The infant and the mom have done great with absolutely no complications.

This is why, when I was talking to the Dad this morning, I was shocked to hear that the baby had to undergo a test I had never heard of: something to do with putting the child in its car seat and monitoring its heart rate and oxygen saturation for one hour. Mind you, my youngest is 9 years old, and I had never hear of this test, so I looked it up.

Turns out, this is a test formally known as a "pre-discharge car seat challenge". I went to the American Academy of Pediatrics web site to look for their take on it, and here it is:
Hospitals should develop a policy to ensure provision of a period of observation in a car safety seat before hospital discharge for each infant born at <37 weeks' gestation to monitor for possible apnea, bradycardia, or oxygen desaturation.
I went on a circuitous journey to trace the origin of this recommendation. Turns out, it is based on a consensus statement from an APP committee from 1996, the reference for which is this paper from 1993 about implementation of the 1990 AAP recommendations. Looking at the 1990 version, however, does not help in the least, as there is no mention at all of the test in question, or anything to do with infants under 37 weeks' gestation. So, this is where the dubious evidence trail ends. Period. No evidence near as I can see. [If someone knows of a more complete statement that does include the information I am interested in, please, point me to it].

Why did I go after this piece of information? Well it seemed to me to arise out of the "more is better" fallacy, and I wanted to understand a) the evidence that led to the recommendation, b) how the test alters practice, and finally, and most importantly, c) does this practice impact the desired outcomes (i.e., prevent something bad).

So, I went to one of the most trusted sources in evidence synthesis, the Cochrane Collaborative, to see if they have examined this issue. Lo' and behold, I found this 2006 review, whose findings were summarized thusly:
There is no evidence that undertaking a pre-discharge "car seat challenge" benefits preterm infants. The "car seat challenge" assesses whether preterm infants who are ready for discharge home are prone to episodes of apnoea (stopping breathing), bradycardia (slow heart rate), or desaturation (low oxygen levels) when seated in their car seat. However, it is not clear whether the level of oxygen desaturation, apnoea, or bradycardia detected in the car seat challenge is actually harmful for preterm infants. Additionally there is concern that the use of the car seat challenge may cause undue parental anxiety about the safety of transporting their infant in a car seat. Despite these uncertainties, and despite the widespread use of the test, we have not identified any randomised controlled trials that assessed whether undertaking a car seat challenge is beneficial or harmful to preterm infants.
So, here is a kid, born at supposedly 36 weeks gestation, but looking and acting like a full-term infant, having to sit in her car seat for 1 hour to undergo monitoring that has absolutely no evidence behind it. Putting aside the inconvenience, the false positives, the parental anxiety, let's think about the costs. I could not find published reimbursement rates for this useless test. However, if you think about the person-power, the equipment used, and the potential prolongation of hospitalization that it may induce, the price tag is likely not trivial.

This case is instructive to me. As I have said before, a pitiful minority of medical practice has good evidence behind it. It is precisely this type of testing, done to comply with what looks to be a self-propagating yet outdated recommendation, that healthcare reformers in Washington need to be addressing. Furthermore, when evidence is kept in a black box, we are likely to find that the box is indeed empty.a,
bradycardia, or oxygen desaturation. An appropriate
hospital staff person should conduct the observation.
Hospitals should develop policies to


  1. Really interesting. And I completely agree with your analysis.

    I saw this article the other day on Slate titled Buy Local, Act Evil that wondered if "organic produce and natural shampoo can turn you into a heartless jerk". This study was based on 59 subjects. 59. If that's not a practically empty black box I don't know what is. But I wouldn't be surprised if what people took away was "those crunch granola people are really mean hypocrites".


  2. Thanks for the comment, Liz! I agree that it is hard to draw conclusions from a study of 59 subjects, particularly in the realm of something as complex as the determinants of human behavior. I am not sure that i would want to be generous with $6 with an anonymous imaginary subject -- I may instead opt to give the money to a cause that I consider important and an organization that makes a difference. These highly artificial situations created in a lab that control for nothing lead us into the abyss of erroneous conclusions.

  3. Great blog! I've linked it to the GSEB blog :)

  4. Thanks, Biki, would be delightful to get comments from UMass SPHHS community!

  5. Disturbing: reinforces for me the unfortunate but true paradox that we have the least evidence on the most vulnerable patients (who would potentially benefit the most from EBM): pregnant women, children, and the elderly.

  6. Kennylin, Thanks for your comment! I agree - these happen to be the populations that are perennially left out of regulatory studies as well. But I also wonder re: politics of recs like these and risk aversion by hospitals to ignore them.