Confirm Your Prejudices
A friend of mine had a lament back during the Rathergate scandal: "Why is it that the media never makes dumb errors like this against us?" Needless to say, he's a democrat.
I mention this because there's another "memo"-like scandal coming out. The Lancet released a study this week purporting to show that 100,000 "excess" civilian casualties have occurred because of the invasion of Iraq. The headline numbers have received quite a lot of media coverage, and one's usual suspects (like Prof. Leiter and Prof. Heller) have jumped all over it. States Prof. Leiter: "So much for the pathetic "humanitarian" rationalizations for the war."
Now, the general point to be made about the war--that the mortality rate has probably risen--doesn't seem counterintuitive. Much like the memos scandal, this isn't really controversial. But just as with Rather, the Lancet's overreaching for extravagant "proof" is becoming the story itself.
You see, any casual reader of the report should have had alarm bells ringing the moment they got to the first page. I pointed out to Prof. Heller in his comments section that while I wasn't a statistician, this smelt pretty funny to me. Over the last few days, other commentators have been clobbering the report with more authority. The best analysis is probably in yesterday's Slate. It begins with the same fact as most of the critiques: the 100,000 number is in fact as follows: "98,000 extra deaths (95% CI 8000-194 000)" In other words, the authors are 95% certain that the number of "extra" deaths lies between eight thousand and nearly two-hundred thousand. As Kaplan says, "This isn't an estimate. It's a dart board."
I'll leave it to my readers to link through and look at an accounting of the methodological problems with the survey: use of clustering in a population with highly divergent results; confirmation of few of the deaths; failure to compare results with other reports. And of course, the fact that the lead author is both anti-war and insisted on pre-election publication raises questions of political bias and motivation.
None of this was obscure: it only took reading the report and the surrounding publicity. (Of course, the New York Times noted nothing about the "expedited" peer-review of the article and its author's insistence on pre-November haste.) But neither Heller, Leiter, or their ilk wrote critically about the numbers, instead focusing on the headline figure and from there drawing their preferred political conclusions.
And herein lies the lesson, I think, for bloggers: check your prejudices when they tempt you. There's a strong urge, particularly in a heated electoral season, to quickly grasp those things which seem to support your favored positions. I mean, I'd love to see a headline tomorrow that said something like, "KERRY FOUND IN IOWA AL QAEDA BORDELLO--EVENING SPONSORED BY MOVEON.ORG DONATIONS FROM HEINZ FOUNDATION". But if the headline came from National Review, I would think thrice before linking. I'd probably call friends in Iowa to make sure that the hotel in question existed. Emails to the reporter might be in order, or to prominent members of local law enforcement. I'd certainly search the blogosphere for counterarguments before penning anything. And if, for instance, it turns out that this house of ill-repute were run by an old Jewish madam and the source of the Moveon.org link was Karl Rove's ex-girlfriend... well, let's just say that I'd link to those parts of the argument, if not talk about something else instead of post on the subject. (For instance, I've said very little about the Swift Boats Veterans, because frankly I couldn't evaluate that evidence properly if I took the time, which I won't.)
There's a reason for doing this, and it's summed up in one word: credibility. Anything coming out these days that seems too good to be true probably is. I have to admit that during the Rather scandal, I was continually waiting for the other shoe to drop: the mythical Selectric coming out of hiding and typing out a damning indictment of internet sleuthing, Dan Rather bringing the necessary witnesses back from the grave to testify... something. The idea that CBS would shoot itself in the foot this badly just seemed to be such a gift. I got lucky: my original post on the subject wasn't thoroughly researched enough, but seems to be correct as of this date. But that was luck, and I wouldn't count on it again.
The thing is, as a blogger you've got a continual electronic record of your errors, especially if you get a lot of links. While correcting those mistakes counts for a lot, not making them in the first place is better. And the best way to do that--and preserve a good reputation--is to ensure that your sources are solid before you use them. This Lancet study should caution those who cited it. At least, one would hope.
UPDATE: Add Ambimb to the list of blogs I read that have quoted this study without critically evaluating it. Still, you can hardly blame him, since he reads the Guardian, which should get win, place, and show in the Uncritical Prejudiced Analysis Derby. In light of the above, note the Guardian's introductory paragraph: "About 100,000 Iraqi civilians - half of them women and children - have died in Iraq since the invasion, mostly as a result of airstrikes by coalition forces, according to the first reliable study of the death toll from Iraqi and US public health experts." There is nothing of the critical analysis I've linked here, not even a quote from outside experts. The paper is taken as gospel, the "first reliable study."
Martin, bear in mind the next time that you tell me the Guardian is a "reliable" source that in their mind, "reliable" means somewhere between 8,000 and 196,000.
Comments
Posted by: ambimb | October 31, 2004 7:59 AM
Posted by: A. Rickey | October 31, 2004 11:42 AM
Posted by: martin | October 31, 2004 4:28 PM
Posted by: Anthony | October 31, 2004 4:38 PM
Posted by: Fitz-Hume | November 1, 2004 3:37 PM
Posted by: Tim Lambert | November 6, 2004 11:46 PM
Posted by: Anthony Rickey | November 7, 2004 12:51 AM
Posted by: Tim Lambert | November 7, 2004 10:26 AM
Posted by: Anthony Rickey | November 7, 2004 1:49 PM
Posted by: Simon | November 8, 2004 2:00 AM
Posted by: Anthony Rickey | November 8, 2004 2:16 AM
Posted by: Tim Lambert | November 8, 2004 11:09 AM
Posted by: Simon | November 8, 2004 8:53 PM