More on Prediction

Expert Political Judgement

I heard a podcast of a speech by Philip Tetlock, a professor at UC Berkeley who does work studying prediction-making ability.  While I’m simplifying considerably, he has done a long study (since 1984) tracking thousands of predictions made by experts in the political arena (all anonymous), and looked at how well they do.  For instance, if a person predicts that some collection of events are 70% likely to occur, then they are a good predictor if in fact 70% of those events do occur.  He also looks at how wide a prediction band a person is willing to commit to; for example a cautious person may say every event is from 40 to 60% likely to occur, whereas others may use a wider range.

He breaks the experts into two main categories, hedgehogs and foxes.  Hedgehogs are ideological – they have a theory, and they tend to see the world in terms of that theory, and fit the evidence into the theory.  Foxes on the other hand tend to pick and choose from various theories, picking what seems to work in each area.

His findings are that overall the foxes have a better record of prediction than the hedgehogs.  Hedgehogs are more likely to make very extreme predictions: that some event is 100% certain or that another event will never occur (0% likelihood). While some of these hedgehog predictions do pan out, many of them don’t.

Interestingly, even the foxes don’t do too much better than some fairly simple statistical prediction models (such as, ‘assume no change’).

I leave it to the reader to draw conclusions from this work.  Here’s one conclusion from an economist, Bryan Caplan.

Here’s a New Yorker review of Tetlock’s last book “Expert Political Judgment: How Good Is It? How Can We Know?”

Trackbacks are closed, but you can post a comment.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: