Rigorous impact evaluations are much more than just evaluations

by Constanza Di Nucci (SSD) and Ilaria Firmian (ECD)

  • Did you know that in the North of Mali, it’s estimated that access to irrigation increases household consumption by 27–30%? 1
  • Did you know that in Madagascar, eliminating transport costs doubles incomes of the most remote families because it creates non-farm earnings opportunities in towns? 2
  • Did you know that in Dominican Republic promoting financial literacy, unlike traditional principle-based courses, improves business practices. For those who may not be familiar – financial literacy are simple rules that help make better financial decisions, allows households to focus on the need to separate business and personal accounts; oneself a fixed salary, distinguishing between business and personal expenses, and easy-to-implement tools for reconciling accounts? 3
  • Did you know that in rural Bangladesh, only 69% of households accept free improved cook stoves? This suggests that there is more than what meets the eye – it is not all about money, it’s also about culture and social norms.4
These are all results from rigorous evaluations carried out to inform governments and development agencies about the impact of different projects and how to improve the design of future interventions based on measurable results.

In a context where we need to do more with less, it becomes crucial to make decisions based on evidence of what works and what does not. It is equally important to have programs which are more cost-effective and paramount that rural development interventions have to work better in a specific context, and at a reasonable cost. In this sense, rigorous impact evaluations can help us in the process of achieving higher results with limited resources.

To learn more about this, from 10-13 September 2012, the Environment and Climate Division and the Statistics & Studies for Development Division of IFAD jointly organised a learning event on rigorous impact evaluations, particularly for environmental and climate change interventions. This was done in collaboration with the International Initiative for Impact Evaluation (3iE). 3iE is one of the leading organizations working on high-quality impact evaluations and campaigns to inform better programme and policy design in developing countries. We had the benefit of hearing the expert views of Dr Howard White, Executive Director of 3iE, Dr Jyotsna Puri, Head of Evaluation, Deputy Executive Director of 3iE, and Dr Edoardo Masset, Research Fellow at the Institute of Development Studies.

The learning event started with a general presentation and discussion on experimental designs, including randomized control trials (RCTs). Colleagues presented interesting cases , such as the Bangladesh Integrated Nutrition Project (BINP). BINP was considered a success based on outcome monitoring, but turned out to be failure once it went through a theory-based impact evaluation, which clearly showed what was not working since the theory of change behind the intervention omitted key family dynamics that affected the outcomes.

Impact evaluations for climate change interventions
The discussion then addressed more specific environment and climate change related issues. This is clearly a new area for rigorous impact evaluations since currently the evidence base for climate change interventions is minimal.

One of the most interesting parts of the whole event was the session dedicated to “what we want to learn on smallholder adaptation”. Participants came out with a number of questions, ranging from cost-effectiveness of adaptation measure, to reasons for good practices not being scaled up; identification of incentives to take up climate smart good practices, role of autonomous versus planned adaptation intervention, role of social issues in adaptation and many more.

A common conclusion was the need to unpack the definition of “resilience”: if we want to increase climate resilience of poor smallholder farmers, we first need to know “how resilience looks like” in terms of specific and measurable indicators.

The learning event was dedicated to assess the suitability of rigorous impact evaluation methodologies for projects under the IFAD Smallholder Agriculture Programme (ASAP), including the conditions, limitations, and alternative methods.

ASAP is a multi-year and multi-donor financing window to channel climate and environmental finance to smallholder farmers through IFAD-supported programmes. ASAP will provide a new source of co-financing targeted specifically at scaling up and integrating climate change adaptation. We brainstormed on some of the preliminary ASAP project ideas under an RCT perspective, and although some of the reasoning is very similar to the one done for designing project logical frameworks, conducting an RCT requires to think much more through the theory of change and to the assumptions around it. But having a clear theory of change is not enough, because there is a large number of factors that need to be taken into account for a good design of RCT. These include:

  • the quality of the baseline,
  • sufficiently large sample size,
  • comparability between project groups and control groups,
  • possible spillovers,
  • contamination of treatment or control,
  • cultural factors,
  • communication with relevant stakeholders,
  • and many more.

Undertaking rigorous impact evaluations therefore represents a great opportunity for IFAD to help improve project operations, to successfully scale up interventions, and to be an innovative actor on the climate change scene. At the same time, RCTs are a powerful tool for knowledge management that can be used to influence strategic thinking not only at the programme level but also at a much higher global advocacy and policy level.

1/  “The Effect of Irrigation on Poverty Reduction, Asset Accumulation, and Informal Insurance: Evidence from Northern Mali” World Development, Volume 39, Issue 12, December 2011, Pages 2165–2175.
2/  Hanan G. Jacoby and Bart Minten. “On Measuring the Benefits of Lower Transport Costs” World Bank Policy Research Working Paper 4484. December 2007.
3/  Alejandro Drexler, Greg Fischer, and Antoinette Schoar “Keeping it Simple: Financial Literacy and Rules of Thumb” January 2011. 
4/  Grant Miller and A. Mushfiq Mobarak. “Intra-household externalities and low demand for a new technology: experimental evidence on improved cook stoves”. Stanford University December 2011.