My friend Steve likes to proclaim the value of casual intuition — based on one’s day-to-day observations over the course of life — and downplay the value of expertise, analysis, and data in making good judgments. Among other things, he defends Sarah Palin and other less-thinkerly politicians on these grounds.
He also points to Robert McNamara — the king of data analysis — as having failed utterly in his judgments on Vietnam. This putting aside the facts that 1. Steve’s casual intuition would have led him to exactly the same policies (if not worse), and 2. McNamara’s data was not the driving force behind the big decisions and judgments on Vietnam. They were at best excuses, self-justifications, rationalizations, or simple thumb-twiddling. McNamara actually manufactured a system that delivered systematically false data.
Also: systematic, in-depth knowledge — rooted in research, analysis, and frequently, data — is obviously not sufficient to guarantee good judgment. But it is arguably necessary. Or at least, it (greatly?) improves the odds of making good judgments. If the Bush administration, for instance, had had some basic knowledge of the difference between a Shiite and a Sunni…
One of the key books on this field is Philip Tetlock’s Expert Political Judgment. He argues — based on analysis of 82,000 predictions by 284 experts — that political experts perform only slightly better than random dart throws. It’s a pretty damning condemnation of experts.
But as Bryan Caplan has pointed out, there are two fatal flaws in Tetlock’s argument:
1. He only examines questions that are highly controversial among experts. (If 50% believe each way, 50% will inevitably be wrong.) Tetlock explicitly ignores the “dumb” questions that seem to the experts to have obvious answers, but which everyday folks might consider controversial.
2. He doesn’t compare the the experts to the average person on the street. The only such comparison in the book is between experts and Berkeley undergrads — who are darned high on the elite/expert spectrum, in absolute terms. And even in that comparison, the experts win in a landslide. The undergrads aren’t even as good as chimps or dartboards.
This suggests that if you looked at those “obvious” questions — which are often not at all obvious to non-experts — and compared casual to expert opinion, you’d see experts being right far more of the time. As they say in the biz, “more research needed.”
Tetlock does reveal another fact, however, that serves to seriously undermine one’s confidence in the intuitionally inspired beliefs of Sarah and similar: among the experts, “foxes” — those who in Nicholas Kristof’s words are “are more cautious, more centrist, more likely to adjust their views, more pragmatic, more prone to self-doubt, more inclined to see complexity and nuance” — resoundingly beat out the “hedgehogs” — those who “have a focused worldview, an ideological leaning, strong convictions.”
Is this also true of everyday folks? Based on my many years of decidedly non-systematic observation, I would suggest that it is.
Update: Chris’ comment,
This worked well enough back when virtually all information of note was controlled by experts. Now they’re forced to compete with everyone, which has the nasty side effect of forcing people to become steadily more extreme and loud just to be heard.
Reminds me of another takeaway from Tetlock’s research. Again quoting Kristoff because he summarizes it well:
the only consistent predictor [of accuracy] was fame — and it was an inverse relationship. The more famous experts did worse than unknown ones. That had to do with a fault in the media. Talent bookers for television shows and reporters tended to call up experts who provided strong, coherent points of view, who saw things in blacks and whites.
In other words, the loudest, most simplistic, and most dogmatic “experts” — the extreme hedgehogs — 1. are the least accurate, and 2. get the biggest megaphone.
Comments
10 responses to “Do Experts Know Better?”
This worked well enough back when virtually all information of note was controlled by experts. Now they’re forced to compete with everyone, which has the nasty side effect of forcing people to become steadily more extreme and loud just to be heard.
Very true, although the opposite can happen (Fame->Dogmatic), as becoming famous for a viewpoint makes the costs of defection from that viewpoint much greater.
steve
got here by following your link at AB. thanks for your article there, and your reply to my questions.
but now that i am here, let me take provisional issue with your conclusion on this subject.
1) probably mostly you are right.
2) the “non partisan experts” on Social Security are all wrong.
3) the “policy experts” on social security (undisclosed website by invitation only) appear to be a bunch of insecure ignoramuses and ninnies. their idea of a compelling argument is “we are experts.” and “i can’t explain that to the layman.”
so, yeah, Sarah Palin is not chanelling ancient wisdom that provides a better light to see by than the “experts.” but, do not in general trust someone who says, “trust me, i am an expert.”
Thanks for visiting, coberly!
Oh, yeah, ubetcha. Plenty of “experts” predicted that interracial marriage would lead to the decline of Western civilization…
the loudest… most famous experts..
are probably wrong because of the limitations of human intelligence. they became experts … either real experts, or merely experts in what other people said… when they were still young and honest.
as they get older they have repeated their own expert opinions so often they have memorized them and they no longer bother to think. but they do insist upon getting their due respect as experts.
as i have heard it said “physics advances one funeral at a time.” pity that “economics” can’t manage even that.
>as i have heard it said “physics advances one funeral at a time.†pity that “economics†can’t manage even that.
I love that quote. It’s sadly true. (cf. my post on Dawkins and group selection.)
But I think economics is doing that, but much more slowly because it’s a far more difficult science — because you can’t rerun the darned experiments with different settings!
Some may say that makes it an impossible science or non-science, but I’m not so fatalistic regarding the aggregate value of loads and loads of observational research.
sometimes i think the reason that physics “advances” is because there is no political value in lying about it.
possible exception: hitler killed physics in germany by calling it jewish science.
but for the most part physics makes people rich and builds them bigger bombs without regard for their politics.
any time that economics points out that people can do better by limiting their greed, you can count on a bright boy or girl proving that it’s junk science.
>sometimes i think the reason that physics “advances†is because there is no political value in lying about it.
Yeah. Could be also (repeat: also) the reverse causation: because physics is subject to rigorous experimental verification, there’s less play room for our human cognitive failings. Can’t argue stuff away as easily.
oh, well, that too.
but hitler really did stop the advance of physics in germany. can’t get no rigorous experimental verification if you can’t get the money to do the experiments.
similarly, there is no doubt about the truth of Social Security. easily verifiable by the experiment of reading the Reports and doing the arithmetic. But you have a Congress and President who are willing to stand up in front of the entire country and not only lie about it, but make no sense. They can get away with this, because humans in general are not capable of thinking clearly.
Physics advances because it has demonstrated its ability to make money for the people who have the money to pay the very few talented people to think clearly about it.
and with that we will return you to our previously scheduled programming.
[…] Philip Tetlock’s Expert Political Judgment was a groundbreaking look at whether political experts really are expert, as judged by their success at making predictions. His overall conclusion: they aren’t. But (lifted from a previous post): […]