One of my old supervisors told me that the Gates Foundation is recruiting for a Senior Program Impact Analyst to lead their Impact Assessment Team. AFAIK none of the other foundations involved in international health have a similar external review setup. Boy, would I hate to be that person. It’d be like serving on Internal Affairs for the LAPD.
Accountability is good — just like evidence-based medicine is good — provided that application of the methods doesn’t stretch their limitations too much…
A trial from a group of Kenyan schools gives us the average effectiveness of flip charts in the experimental schools relative to the control schools for an area in western Kenya, at a specific time, for specific teachers, and for specific pupils. It is far from clear that this evidence is useful outside of that situation. This qualification also holds for the much more serious case of worms, where the rate of reinfection depends on whether children wear shoes and whether they have access to toilets. The results of one experiment in Kenya (in which there was in fact no randomization, only selection based on alphabetical order) hardly prove that deworming is always the cheapest way to get kids into school, as Banerjee suggests.
The comparison with the FDA is very much to the point, but only because exactly the same problems come up. For a specific doctor facing a specific patient, the average outcome of a randomized controlled trial will often be unhelpful. The physician usually has some theory of how the drug works and also an understanding of her patient, who might, for example, be elderly, frail, overweight, and an ex-smoker, with a history of responding to some medications and not others. Therefore the physician will often not prescribe a drug that passed its randomized controlled trial with flying colors and instead prescribe one that did less well but that is a better fit for the patient. Much of medicine is not “evidence-based,†for good reason.
There is no simple way to use randomized controlled trials to eliminate global poverty. They are expensive and technically and politically difficult to do well. We must be careful to apply them only where there is a good chance that the results will be applicable elsewhere. Otherwise, we will be gathering evidence, not knowledge.
–Angus Deaton, “‘Evidence-based aid must not become the latest in a long string of development fads’, Boston Review, Jul/Aug 2006