Dean Karlan on when -- and when not -- to use scientific trials to measure impact

I refer to this as the Goldilocks problem. A lot of donors, of course, "get it" in general and push for the right amount of data. But a lot don’t, and they push for too much or too little data.

Dean Karlan is an economics professor at Yale and president of Innovations for Poverty Action, a non-profit organization that creates and evaluates solutions to social and development problems. His latest book, “More than Good Intentions,” shows how small changes in banking, insurance, healthcare and other development initiatives that take into account human irrationality can drastically improve the lives of the poor. PopTech’s interview with Karlan focused mostly on the use of randomized controlled trials to guide policy.

Dean Karlan at PopTech 2012
  1. Dean Karlan

PopTech: Randomized controlled trials have been around since 600 B.C., but haven’t been applied much to social policy until the last decade. What took you guys so long?

Karlan:I’m not sure the shift is so much about RCTs versus other methods of doing policy evaluations, as much as it is a shift towards doing more policy evaluations. Twenty years ago academic development economists were asking different questions. They weren’t as directed at the policy needs to find out what was not working and why. They were doing more fundamental research, more descriptive research, on the nature of markets and households — good stuff that lies at the foundation of the work the evaluation community is now doing.

There is another practical explanation behind the shift. It is easier to collect data now. It is easier to run analysis on large datasets now. It is easier to fly around the world now, hire people, talk with them and manage them when they are living abroad. Twenty years ago, if you wanted to run an RCT in Ghana, you would have had a hard enough time just managing the staff in Ghana and talking with the local organizations. We just saw much less field work being done back then.

PopTech:What was the most surprising thing you learned from one of these trials?

Karlan: I have been genuinely surprised by the size of different effects. We know that some of these psychological things matter, but it is surprising to me sometimes how much they matter. You know, we tie people’s hands when it comes to commitment savings accounts [mechanisms that commit people to save for the future, the way 401-K plans work] and we see savings go up by 300 percent for those who sign up, compared to a control group. I wasn’t surprised that worked, but the magnitude surprised me.

The other big surprise in many cases is just how often things are not working, particularly things that we think should work. Take smokeless indoor cooking stoves. It is not to say the idea is bad, but clearly they have not figured it out yet, or the ones that have figured it out have not documented well that their stoves work.

PopTech: Is it true that powerful people and institutions sometimes resist trials because they don’t want to learn that somebody’s brilliant idea didn’t pan out?

Karlan: Yes, I think that is why a lot of times people don’t do evaluations. That’s the reality. While they would never say it, they think, “There’s no upside to figuring out if my program works. What do I have to gain from that?” I have been in such conversations. They walk away and they never get evaluated and never figure out with convincing evidence if what they’re doing is worthwhile.

It is depressing, I have to say, when someone really, almost openly says that. I don’t know that I have ever had a leader of an organization say that to me, but I have had an underling say, “My boss sees nothing to gain and lots to be lost. We are getting lots of attention and we are getting lots of money. Nobody seems to mind. If this report comes out badly for us, that’s bad. And if it comes out good, nothing will change.”

PopTech: Are there initiatives that should not be subjected to a randomized controlled trial?

Karlan:Absolutely. There are stages of a project when it doesn’t fit. One stage it doesn’t fit is when something is way too early. You are just figuring out what the program is or what the policy is. Tinker away. Go figure it out. Nobody wants to do a full-blown study if no one is actually going to use the damn thing.

The other is when it is too late. Everybody already has access to it and is using it.

PopTech: Should big funders always insist that they see positive trial data before handing out money?

Karlan: No. Some ideas do have good theory behind them but lack evidence. If that is the case, the big donor should fund the program, and the research to find out if it works and why. If the program has already been shown to work, then sometimes donors push for too much data. I refer to this as the Goldilocks problem. A lot of donors, of course, “get it” in general and push for the right amount of data. But a lot don’t, and they push for too much or too little data. That is something that we are looking at and we are trying to provide guidance to donors when they are asking the NGOs that they support for data, to try to help them figure out what the right level is.

PopTech: What person or organization was way out ahead on the use of trials to guide policy?

Karlan: In microfinance, Freedom from Hunger is an outlier compared to the others. They go out there and test their core question and they see whether it works. For post-conflict work, International Rescue Committee stands out.

PopTech: Critics of trials often argue that the movement has gone overboard, and threatens to stifle innovation and freeze up money while everybody conducts a million trials. What is your response to that?

Karlan: I am amazed what some people claim about people who are proponents of RCTs. They say, “Randomized trial advocates just want to test everything with randomized trials.” No one has ever said that. I can point you to articles where people make this as their main point of attack. All I can say is, “Please show me one place where that has been said by me or any other academic who does this kind of work.” It is annoying, but you just have to move on and learn to ignore it. Folks will say lots of things to defend their turf.

Dean Karlan

Dean Karlan is President of Innovations for Poverty Action, a non-profit organization that creates and evaluates solutions to social and development problems, and works to scale-up successful ideas through implementation and dissemination to policymakers, practitioners, investors and donors. He is a Professor of Economics at Yale University.