Today's topic was spurred by some rather unusual college rankings by the Wall Street Journal, in which Texas Tech has a higher rank than Harvard. This has been among the most popular articles on the Online Journal for 3 straight days now. Of course, also very popular are the US News and World Report College Rankings (which give the rather opposite results in which Harvard does slightly better relative to Texas Tech.) We all love rankings.
Talking about ranking methodology, not so much.
(Turns out WSJ rankings were biased towards larger colleges. Texas Tech had undergraduate enrollment of over 22 thousand, compared to less than 7 thousand at Harvard.)
In development, we have all kinds of ranked beauty contests: the Human Development Index (HDI), Doing Business, Governance Indicators, Corruption, etc. etc. I've even been participating in one ranking exercise myself, on, of course, aid agencies.
Just like wacky college rankings, many development rankings are based on methodologies that could use a lot more scrutiny than they usually get. Take the much-loved uber-publicized Human Development Index (HDI). The HDI has a hidden implication pointed out by the World Bank's Martin Ravallion as long ago as 1997. Imagine the reaction to this hidden implication if they publicly announced it:
The UNDP announced today that they consider a human life in the USA to be 70 times more valuable than human life in the Democratic Republic of the Congo.
The other consequence of this hidden implication is that rich countries will get A LOT of credit for higher life expectancy (and not much for income). Scandinavia does very well on the HDI because its life expectancy is higher by 1 or 2 years than the US, even though US income is higher. Bryan Caplan suggests:
Scandinavia comes out on top according to the HDI because the HDI is basically a measure of how Scandinavian your country is.
There's an alternative to constructing murky indexes with dubious assumptions. This would be to be use people's actual choices to infer which places are better. If a college recruiter (the basis of the WSJ rankings) had to choose between otherwise equivalent Texas Tech and Harvard graduates, which one would they pick? Instead of the US News convoluted rankings, why not just ask a student admitted to both Texas Tech and Harvard which one they would pick (or DID pick). These are called "revealed preference rankings" and have actually been done for colleges.
Instead of Human Development Indexes, why not just ask migrants where they would choose to live (or do actually choose to live): US or Iceland? Iceland has a much higher HDI, but actually has out-migration, whereas I think there are a few people trying pretty hard to get into the US.
Instead of Doing Business index-based rankings, why not use the actual choices of businesses where to invest? These also have the added attraction of not being subject to artificial manipulation (which happens with both college index rankings and Doing Business).
Indexes should not be discarded altogether. Sometimes there are no choices to guide rankings (like aid agencies, sadly). They can also summarize information for those making the choices.
But please double check your methodology.