April 04, 2011
Alex Counts is President, CEO and founder of Grameen Foundation.
In addition to our commitment to excellence in social-performance management by microfinance institutions, exemplified by our championing of the Progress out of Poverty Index™, Grameen Foundation has long been committed to promoting a fuller understanding of research into the effectiveness of microfinance as a poverty-reduction strategy. We have published two reports, one by Professor Kathleen Odell of Dominican University in 2010 and an earlier one by Nathanael Goldberg (who is now with Innovations for Poverty Action), that attempted to summarize in simple terms what the evidence could tell us. (Though Grameen Foundation commissioned both reports, we exerted no editorial control over what either Odell or Goldberg wrote.) Both reports were well-received by practitioners, researchers, investors and policymakers alike. David Roodman, a leading blogger in the microfinance industry, wrote that Prof. Odell did “a fantastic job” with her report, adding, “I applaud the Grameen Foundation for giving her such autonomy. The report reviews a good set of relevant studies. With concision and clarity, yet without jargon, it explains the pros and cons of various research methods, and the limitations of them all. And it draws balanced judgments. It is a model of public communication about social science research.”
"Measuring the Impact of Microfinance: Taking Another Look" is the latest report published by Grameen Foundation that examines the studies of the effectiveness of microfinance as a tool to alleviate poverty.
One of the most confusing and contentious issues covered in both reports is the controversy regarding the well-known Pitt/Khandker studies on the impact of three major microfinance institutions in Bangladesh, including Grameen Bank. In general, their research found many positive, statistically significant impacts on clients when measured against comparison groups. Despite being one of the few microfinance impact-assessment research studies that has undergone a rigorous peer-review process prior to its publication in an academic journal (the Journal of Political Economy), it has been criticized as being flawed and unreplicable by other researchers using the same data. These criticisms have been leveled by NYU Professor Jonathan Morduch (a respected researcher) and Roodman (microfinance's most widely read and respected blogger). Though I lack a deep understanding of econometrics (on which the Pitt/Khandker study relies), I have probed into this debate, as Odell and Goldberg did in their papers. Interestingly, I was alerted last year that an article that was supposed to conclusively prove that the Pitt/Khandker study was wrong was itself rejected from a peer-reviewed journal. It may still be published, though.
Most recently, Professor Pitt has published a detailed response to the criticisms of his original research with Khandker. He claims that Morduch and Roodman made errors of their own in their analysis of his data, and when those errors are corrected, the original findings stand up. The paper’s abstract states, “This response to Roodman and Mordoch seeks to correct the substantial damage that their claims have caused to the reputation of microfinance as a means of alleviating poverty by providing a detailed explanation of why their replication of Pitt and Khandker (1998) is incorrect. Using the dataset constructed by Pitt and Khandker, as well as the data set Roodman and Morduch constructed themselves, the Pitt and Khandker results standup extremely well, indeed are strengthened ... after correcting for Morduch and Roodman errors.” Roodman has just published a preliminary response on his blog.
What are the main takeaways at this point? First, in the world of social-science research, things are not always as they appear. Individual studies should be taken with a grain of salt, as all have their strengths and limitations. More than ever, I think the safest course is to reflect on more than two decades of research using experimental, quasi-experimental and non-experimental designs, as well as personal observations (for those of us who have spent time with microfinance clients) and qualitative research, such as that found in Portfolios of the Poor, a terrific book co-authored by Professor Morduch, and my book, Small Loans, Big Dreams. Objective efforts to demystify research findings, such as the Odell and Goldberg reports and another solid treatment by Freedom From Hunger President Chris Dunford, should be reviewed by microfinance practitioners, investors and volunteers.
Finally, there is no “gold standard” in research, but rather a growing body of evidence that microfinance is an incomplete but improving strategy to address global poverty – one that can be made more effective by refining it based on available research, as well as by having regular feedback loops involving social-performance data that Grameen Foundation will continue to ensure becomes a part of our industry’s DNA.
Comments
Thank you Alex for this thoughtful take on a confusing situation. I think your caution is wise. Forgive me for emphasizing, however, that the rejection of our critique of Pitt and Khankder and two other studies by the prestigious Journal of Political Economy doesn't say much about the validity of our arguments, no more than would the New York Times's rejection of a letter you sent them, for example. Anyay, the ball is indeed back in our court, the the "game" is not not over.