• Home
  • IFAD website
  • Subscribe to posts
  • Subscribe to comments

Whither M &E in IFAD projects? – A Modest Proposal

Posted by Roxanna Samii Tuesday, September 13, 2011


By Steven Schonberger

Where I come from, in Oakland, California in the US, if you want to raise the temperature in a room you just have to mention the “immaculate reception”; the moment in 1972 when the Pittsburgh Steelers football team scored a winning touchdown against the Oakland Raiders in what we Raiders fans are sure was an illegal play which eliminated them from the playoffs and a potential Super Bowl victory (similar to Diego Maradona’s “mano de Dios” against the English in the 1986 World Cup quarterfinals).  In IFAD, I have noticed that a similar catalyst of passions is the mention amongst CPMs of the Results Indicators Management System, or RIMS which can turn any meeting into an exchange of frustrations and complaints.  My sense, however, is that this is not necessarily a response to RIMS, per se, but rather to the difficulties of implementing M&E systems in general in donor financed projects.  In this case, IFAD’s RIMS is perhaps “the best of the bad”.  Fortunately, in M&E, as in the case of football, there are always more games to play and opportunities to do better if we can improve our game.  Following are a few ideas of how we might do that.

In July, we benefitted from a visit by  Rachel Glennerster, the Executive Director of the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachussetts Institute of Technology which has gained international renown for implementing and advocating the use of Randomized Control Trials (RCTs) to assess the impact of development activities ranging from the use and willingness to pay for improved mosquito nets to incentives to keep girls in school to the household impacts of access to microfinance.  Drawing on this experience, Rachel emphasized some key points of importance to IFAD, including:


·         The trend is that RCTs are becoming more common as the methodological standard expected by funding agencies;
·         The good news is that improved approaches to establish control groups are broadening the situations where RCT can be applied;
·         The bad news is that some funders are insisting on more use of RCTs than is either useful or feasible;
·         The challenge is to know when it makes sense to use rigorous, RCT-type approaches and what other types of approaches should be used based on what information is needed and what information can be realistically generated.

A key outcome of the subsequent discussion was that donors, including IFAD, invest a good deal of funding and effort in what are called ‘impact evaluations’ (RIMS level 3)  but which do not provide credible assessments of impacts.  At the same time, there is a great deal more that can be done to strengthen the basic output reporting and more qualitative assessments of how outputs are delivered and how the recipients respond to this. 

Regarding the first point on impact evaluation, the consensus in the room was moving towards what Chris Blattman, a Professor at Yale University advocates:  ‘Do R&D, not M&E’.  Essentially the idea is that impact assessment (level 3) should be focused on broader learning to determine what approaches generally work and under what conditions.  This requires a focused research program which develops carefully defined initiatives and employs rigorous, RCT-type evaluations of these initiatives conducted under different conditions in different regions (but not all projects!).  This  also requires an open attitude which does not focus on success or failure but rather on learning from the results whether they are what was expected or not (in fact, most of the great scientific advances in history were the unexpected, not predicted, results of experiments, but that is grist for another blog…) The objective is to generate something approaching generalizable conclusions regarding questions such as ‘ what is the best way to encourage increased use of fertilizer by smallholders to improve yields?’, or ‘what types of institutional conditions result in sustained empowerment of women in local, rural governance?’.   The result of this effort is an enhanced toolkit of ‘tested’ approaches and the conditions under which they should work which we can apply with some confidence of success.

The second point, regarding reporting, is that we need to focus more on monitoring to ensure that projects are doing the basics of transforming IFAD funding into goods and services for smallholder farmers, and, perhaps even more importantly, ensuring that these are of the right quality and delivered in a manner which meets the real needs of the target group.  Geoffrey Livingston, Sara Delaney and I wrote a paper about this for IFAD’s Smallholder Agriculture Conference in February of this year where we argued for much more attention to the challenges of spatial and temporal coordination of development activities, or more colloquially what we called ‘Right place, right time development’.  

So now for the modest proposal:  Let’s reorient our project M&E to  levels 1 and 2 where we can make a difference and use the information – specifically in monitoring the timely conversion of funding into goods and services for the target group through normal project reporting and due diligence.  Let’s then ensure that local universities or NGOs are hired to do annual ‘client satisfaction’ surveys regarding farmer’s views regarding the timeliness and quality of what they receive as well as how they are using the assistance provided.  As Geoffrey, Sara and I argued, simply addressing this part of the log frame or results chain would already likely lead to much better outcomes.

And what about impact studies?  Let’s ask those RCT-oriented members of the Board (and perhaps some interested foundations) to finance an internal IFAD fund for carrying out RCT and similar, high quality research on key issues for IFAD programs.  The IFAD fund could plan to provide finance and direct technical support for 2 to 4 RCT evaluations per year per Region which respond to priority issues for IFAD as a whole.  We can then share these results with partners, such as J-PAL, IFPRI, FAO, the World Bank and others in building up the collective rural development tool kit which program design could draw upon.  Of course we will continue to innovate and experiment beyond what is in the toolkit, but with more attention to designing these innovations in ways that facilitates evaluating if they work, and under what circumstances, we can strengthen the basis for the type of scaling up which is emphasized under IFAD 9.

And what will be the indicator for monitoring and evaluating this new M&E approach?  Simple - unlike Oakland Raider fans, we will ‘get over’ our frustrations with M&E and focus that energy on something more useful, like how we get effective agricultural finance working…. What do you think?

3 comments

  1. Steven

    It was good to read your blog. I am supporting project M&E in IFAD’s Asia Pacific Region (APR), and share many of your ideas.

    I agree it would be great if IFAD were to commission some Randomised Control Trials (RCTs) into the sort of changes that its typical interventions can bring about. However RCTs are not tools for measuring project impact.

    I agree that we should focus more on RIMS level 1 and 2 indicators. Level one are mainly indicators of project activities and outputs and should not present too many problems for progress reports. Many of the Level 2 indicators provide evidence of outcomes – the immediate result of project outputs, such as the number of farmers reporting increased yields. You have suggested annual `client satisfaction’ surveys to obtain feedback on the quality of support they are getting from the project. In APR we are suggesting that some projects try `annual outcome’ surveys – which are much the same, with indicators such as number of farmers reporting yield increases and size of these yield increases.

    A toolkit for outcome surveys has been developed and about six projects in India have carried out these surveys. Some projects have now done two annual surveys, and results are becoming more useful – as the methodology is improved and changes from year to year can be measured. In India project staff have been conducting these surveys rather than an external agency, which internalises the feedback from project participants as an important part of a learning process.

    Regarding impact surveys – although RCTs could help tell us about what interventions are likely to work best, I think we still need to measure the actual impact of projects as well as their outcomes. For example – does the outcome of increased crop production really result in reduced poverty? Using tools like semi-experimental designs and results chains, it should be possible to produce convincing evidence of project results and impact.

    There are more ideas for M&E on the IFADAsia portal (www.asia.ifad.org) – in particular see the discussion on making M&E more effective. The toolkit for annual outcome surveys is also available from this website, and includes guidelines for questionnaires and sampling, and a spreadsheet-based system for data entry and analysis.

    Edward Mallorie, Consultant supporting M&E for APR

     
  2. Steven

    It was good to read your blog. I am supporting project M&E in IFAD’s Asia Pacific Region (APR), and share many of your ideas.

    I agree it would be great if IFAD were to commission some Randomised Control Trials (RCTs) into the sort of changes that its typical interventions can bring about. However RCTs are not tools for measuring project impact.

    I agree that we should focus more on RIMS level 1 and 2 indicators. Level one are mainly indicators of project activities and outputs and should not present too many problems for progress reports. Many of the Level 2 indicators provide evidence of outcomes – the immediate result of project outputs, such as the number of farmers reporting increased yields. You have suggested annual `client satisfaction’ surveys to obtain feedback on the quality of support they are getting from the project.

    In APR we are suggesting that some projects try `annual outcome’ surveys – which are much the same, with indicators such as number of farmers reporting yield increases and size of these yield increases. A toolkit for outcome surveys has been developed and about six projects in India have carried out these surveys. Some projects have now done two annual surveys, and results are becoming more useful – as the methodology is improved and changes from year to year can be measured. In India project staff have been conducting these surveys rather than an external agency, which internalises the feedback from project participants as an important part of a learning process.

    Regarding impact surveys – although RCTs could help tell us about what interventions are likely to work best, I think we still need to measure the actual impact of projects as well as their outcomes. For example – does the outcome of increased crop production really result in reduced poverty? Using tools like semi-experimental designs and results chains, it should be possible to produce convincing evidence of project results and impact.

    There are more ideas for M&E on the IFADAsia portal (www.asia.ifad.org) – in particular see the discussion on making M&E more effective. The toolkit for annual outcome surveys is also available from this website, and includes guidelines for questionnaires and sampling, and a spreadsheet-based system for data entry and analysis.

    Edward Mallorie, Consultant supporting M&E for APR

     
  3. Shaheel said:
  4. Hello Mr. Steven,

    I do share some of your concerns in M&E. I do appreciate that you recognised that we need to put in more rigour in evaluating project and programme impacts. RCT is a sound methodology under experimental design. It would be interesting to know your views on other methods such as propensity score matching, double difference, regression discontinuity, and instrumental variable methods. We haven't explored any of these yet in IFAD project impact evaluation. It would be good to do more robust impact evaluation of some of our projects.

    In India our focus has been to imporve the results management system by tracking level 1 and 2 indicators. We have also done outcome surveys to track changes in our efforts towards establishing a Results Based Management system.