By Steven Schonberger
Where I come from, in Oakland, California in the US, if you
want to raise the temperature in a room you just have to mention the
“immaculate reception”; the moment in 1972 when the Pittsburgh Steelers
football team scored a winning touchdown against the Oakland Raiders in what we
Raiders fans are sure was an illegal play which eliminated them from the
playoffs and a potential Super Bowl victory (similar to Diego Maradona’s “mano
de Dios” against the English in the 1986 World Cup quarterfinals). In IFAD, I have noticed that a similar
catalyst of passions is the mention amongst CPMs of the Results Indicators
Management System, or RIMS which can turn any meeting into an exchange of
frustrations and complaints. My sense,
however, is that this is not necessarily a response to RIMS, per se, but rather
to the difficulties of implementing M&E systems in general in donor
financed projects. In this case, IFAD’s
RIMS is perhaps “the best of the bad”.
Fortunately, in M&E, as in the case of football, there are always
more games to play and opportunities to do better if we can improve our game. Following are a few ideas of how we might do
that.
In July, we benefitted from a visit by Rachel Glennerster, the Executive Director of
the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachussetts
Institute of Technology which has gained international renown for implementing
and advocating the use of Randomized Control Trials (RCTs) to assess the impact
of development activities ranging from the use and willingness to pay for
improved mosquito nets to incentives to keep girls in school to the household
impacts of access to microfinance. Drawing
on this experience, Rachel emphasized some key points of importance to IFAD,
including:
·
The trend is
that RCTs are becoming more common as the methodological standard expected by
funding agencies;
·
The good news
is that improved approaches to establish control groups are broadening the
situations where RCT can be applied;
·
The bad news
is that some funders are insisting on more use of RCTs than is either useful or
feasible;
·
The challenge
is to know when it makes sense to use rigorous, RCT-type approaches and what
other types of approaches should be used based on what information is needed
and what information can be realistically generated.
A key outcome of the subsequent discussion
was that donors, including IFAD, invest a good deal of funding and effort in
what are called ‘impact evaluations’ (RIMS level 3) but which do not provide credible assessments
of impacts. At the same time, there is a
great deal more that can be done to strengthen the basic output reporting and
more qualitative assessments of how outputs are delivered and how the recipients
respond to this.
Regarding the first point on impact
evaluation, the consensus in the room was moving towards what Chris Blattman, a
Professor at Yale University advocates: ‘Do
R&D, not M&E’. Essentially the
idea is that impact assessment (level 3) should be focused on broader learning
to determine what approaches generally work and under what conditions. This requires a focused research program
which develops carefully defined initiatives and employs rigorous, RCT-type
evaluations of these initiatives conducted under different conditions in
different regions (but not all projects!).
This also requires an open
attitude which does not focus on success or failure but rather on learning from
the results whether they are what was expected or not (in fact, most of the
great scientific advances in history were the unexpected, not predicted,
results of experiments, but that is grist for another blog…) The objective is to
generate something approaching generalizable conclusions regarding questions
such as ‘ what is the best way to encourage increased use of fertilizer by
smallholders to improve yields?’, or ‘what types of institutional conditions
result in sustained empowerment of women in local, rural governance?’. The result of this effort is an enhanced
toolkit of ‘tested’ approaches and the conditions under which they should work
which we can apply with some confidence of success.
The second point, regarding reporting, is that
we need to focus more on monitoring to ensure that projects are doing the basics
of transforming IFAD funding into goods and services for smallholder farmers,
and, perhaps even more importantly, ensuring that these are of the right
quality and delivered in a manner which meets the real needs of the target
group. Geoffrey Livingston, Sara Delaney
and I wrote a paper about this for IFAD’s Smallholder Agriculture Conference in
February of this year where we argued for much more attention to the challenges
of spatial and temporal coordination of development activities, or more colloquially
what we called ‘Right place, right time development’.
So now for the modest proposal: Let’s reorient our project M&E to levels 1 and 2 where we can make a difference
and use the information – specifically in monitoring the timely conversion of
funding into goods and services for the target group through normal project
reporting and due diligence. Let’s then
ensure that local universities or NGOs are hired to do annual ‘client
satisfaction’ surveys regarding farmer’s views regarding the timeliness and
quality of what they receive as well as how they are using the assistance
provided. As Geoffrey, Sara and I
argued, simply addressing this part of the log frame or results chain would
already likely lead to much better outcomes.
And what about impact studies? Let’s ask those RCT-oriented members of the Board
(and perhaps some interested foundations) to finance an internal IFAD fund for
carrying out RCT and similar, high quality research on key issues for IFAD
programs. The IFAD fund could plan to
provide finance and direct technical support for 2 to 4 RCT evaluations per
year per Region which respond to priority issues for IFAD as a whole. We can then share these results with
partners, such as J-PAL, IFPRI, FAO, the World Bank and others in building up
the collective rural development tool kit which program design could draw
upon. Of course we will continue to
innovate and experiment beyond what is in the toolkit, but with more attention
to designing these innovations in ways that facilitates evaluating if they work,
and under what circumstances, we can strengthen the basis for the type of
scaling up which is emphasized under IFAD 9.
And what will be the indicator for monitoring
and evaluating this new M&E approach?
Simple - unlike Oakland Raider fans, we will ‘get over’ our frustrations
with M&E and focus that energy on something more useful, like how we get
effective agricultural finance working…. What do you think?