“Evidence-based policies” are in vogue. But how do you synthesize the evidence base? People often engage in “vote counting”: reading the literature and consciously or subconsciously summing up the number of findings for a positive effect, a negative effect, or no effect for a particular program. The group with the greatest number wins.
Unfortunately, vote counting is not an ideal method to synthesize the evidence. The biggest problem is that some “no effect” papers were unlikely to find an effect even if there was one. Many studies in development use too small of a sample to be likely to find an effect, so the fact that their results are insignificant is not actually all that informative.
An alternative technique, meta-analysis, can aggregate many insignificant findings and sometimes transform them into a jointly significant result. It also allows flexibility in weighting studies differently, since all studies are not equal.
In most of the cases in which vote counting and meta-analysis diverge, vote counting reports an insignificant result and meta-analysis reports a significant positive result. For example, both conditional and unconditional cash transfer programs often had several “no effect” results — “cash transfers don’t work!” These types of programs have effects on a very broad range of outcomes, but because some or all of them are only tangentially related to the intervention, it’s harder to see an effect in any one study. But if you aggregate the insignificant results on labour force participation, grade promotion or test scores through meta-analysis then they become significant — “cash transfers work!”
The error of overstating the strength of “no effect” results through vote counting is all the worse given that “no effect” does not really mean no effect. The common misconception is that failure to reject the null hypothesis of no effect means we have accepted the null hypothesis of no effect, but that is simply untrue. Absence of a positive finding becomes a finding of absent effect, but this is not what the test says. Perhaps with a bit more data the result would become significant.
How big is this problem? Preliminary analysis of a database I have assembled of development studies, through a group called AidGrade, suggests that the meta-analysis results for a particular intervention-outcome combination diverge from the results that would have been obtained using vote counting about a third of the time. Vote counting actually gives very similar results as to what one would get by just looking at a single paper selected at random from the entire literature; not a great foundation on which to base policy recommendations. If we want to use rigorous evidence, we have to be rigorous about how we use rigorous evidence.
Last Monday we had the pleasure of hosting a few of our closest friends at Cooper Union’s Great Hall to celebrate the launch of Professor Easterly’s new book, The Tyranny of Experts: Economists, Dictators, and the Forgotten Rights of the Poor. Paul Romer gave a gracious introduction, and many audience members had the chance to question Bill’s audacious theories in a Q&A at the end of the lecture. Below are just a few selected clips from the evening (Paul’s introduction, Bill on his membership in Authoritarians Anonymous, and his answer to the perennial favorite question: “But What Can I Do?”). To hear more, take a look at the author’s speaking schedule for the next few months which will take him to Boston, DC, the West Coast and London, and of course, read the book.
NPR’s The Takeaway asks in an interview with one of our local troublemakers this week, are billionaire philanthropists the true champions in the fight against poverty? Listen to at least part of the audio to get the tone of the critique, as well as read the selected transcripts below.
Bill Easterly: I have nothing to take away from the billionaires who are very generous, who are spending on the poor rather than on private jets – that’s great. But what can actually happen is they can also have too much influence on the way we see the whole problem of global poverty.
Gates has this ‘great man’ approach to development in which he sees great national leaders and great philanthropists like himself doing all the good things that happen. Unfortunately, the Ethiopian government…that he praised a year ago in his Annual Letter from his Foundation is not doing great things. He is very naïve to think that the government is benevolent and is actually contributing to development. They actually are serial human rights abusers that are destroying development.
Before Gates’ annual letter… there was a peaceful blogger named Eskinder Nega who was sentenced to eighteen years in prison simply for advocating more democracy in Ethiopia, for writing about the Arab Democratic Spring.
This kind of democratic activism is what you need to make government leaders benevolent. If you think of our own Chris Christie scandal on the bridge – that’s the sign of democracy working, that we keep Chris Christie from doing something bad. He’ll never do it again. No other governor will ever do it again.
John Hockenberry (host): Can’t you make an argument that you want to be separate from politics? The United Nations and many NGOs try to stay out. For instance, CARE and the Red Cross are completely independent from politics. [They] go into Ethiopia regardless of what the government is doing and get access because of their objectivity, or their detachment from politics.
Bill Easterly: That’s the perpetual temptation in poverty reduction: to think you can do something that’s technically pure that’s free from politics. Unfortunately that’s a delusion. Let me give you one example of that. Famine relief you might think is as a-political it can get. But unfortunately to go back to Ethiopia, the same government Gates was praising was caught red-handed using famine relief to only give it to the supporters of the ruling party. They denied it to the opposition party members. They were starving the opposition – in the middle of a famine they were rewarding their own supporters and staying in power by that means.
Listen to the full program at The Takeaway here.
Monday, March 3, 2014
The Great Hall, Cooper Union, New York City
William Easterly will present a short talk on his new book* released on the same date
Featuring an introduction from NYU University Professor Paul Romer
Doors open at 6:00, the trouble begins at 6:30
“No one who starts this book will be able to put it down, or be able to undo its influence on her thinking about the deep determinants of development progress.” – Nancy Birdsall
“Another striking and original success.” – Tyler Cowen
“Tells the extraordinary story of authoritarian development.” – Angus Deaton
*Independent auditors have certified this book does not mention Jeffrey Sachs, nor does it discuss whether foreign aid works.
Bill Easterly responds to Bill and Melinda Gates’ Annual Letter:
Mr Gates says there has been much progress, but that “we’ll need to apply human ingenuity and act on our compassion” to keep it going. Conversely, he equates the idea that “the world is getting worse” to the idea that “we can’t solve extreme poverty and disease”. For Mr Gates, apparently, much depends on what “we” do. But who are “we”, and who put us in charge? Mr Gates seems to have in mind the global elite whose most prominent representatives were this week assembled in Davos: political leaders, business executives, philanthropists, academics and functionaries from international institutions such as the World Bank.
The progress that Mr Gates celebrates is the work of entrepreneurs, inventors, traders, investors, activists – not to mention ordinary people of commitment and ingenuity striving for a better life. Davos Man may not be ready to acknowledge that he does not hold the fate of humanity in his gilded hands. But that need not stop the rest of us.
Read the whole article in the Financial Times (Note to spotters of irony on Twitter: elitist paywall easily defeated by 1-minute free registration). Also, Chris Blattman grades the letter, giving the Harvard dropout an A-.