Natural experiments (NEs) are an underexploited evaluation design. Our recent evaluation for the International Organisation for Migration (IOM) demonstrates that natural experiments provide valuable insights – in this case both about how returnees cope under pressure, and as a real-life ‘stress test’ of a programme’s effectiveness. In this blog we reflect on the experience, and share five lessons for anyone considering the use of natural experiments in the future.
What is a natural experiment?
Michael Loevinsohn, lead NE investigator, defines a natural experiment as “an observational study of a sharp, well-defined but unplanned change or extreme event. Natural experiments hinge on identifying an uncontrolled but opportune ‘intervention’, typically of a kind or on a scale that could not – ethically or feasibly – be implemented deliberately…”.
An important advantage of natural experiments is that they provide data from a real-world event – evidence of what happened, rather than hypotheses about what might happen. Natural experiments can this provide insights with a level of confidence that conventional evaluations might have difficulty matching.
Understanding the effects of Covid-linked shocks on migrant returnees in the Horn of Africa
In March 2020, IOM commissioned Itad to conduct an impact evaluation of the EU-IOM Joint Initiative for Migrant Protection and Reintegration in the Horn of Africa region. As part of the study, the evaluation team undertook a natural experiment of the effect of the Covid-19 pandemic on returnees in Ethiopia, Sudan and Somalia. The experiment aimed to assess the extent that individual components of the Joint Initiative’s assistance contributed to people’s resilience to Covid-linked shocks. By using fixed-effect multivariate regression analysis, we were able to identify the factors that helped or hindered resilience. This analysis provided valuable insights into returnee agency in the face of shocks, the actions they adopted in response, and what conditions and characteristics influenced their resilience. (You can read more about the evaluation and the benefits of natural experiments in our recent blog).
Five lessons from our experiences in the Horn of Africa
The following five lessons were summed up in our study report ‘Using natural experiments in crises: lessons for evaluation’ by Michael Loevinsohn and Tom Gillhespy with Chris Barnett, Leonora Evans-Gutierrez and Callum Taylor.
- Defining the study population
A challenge of certain evaluation approaches can be defining and identifying control groups that are sufficiently comparable to the experimental group. Depending on the context of the natural experiment, these challenges may still apply, but if considering a natural experiment in response to a large-scale shock (i.e. a natural disaster or conflict), everyone in the affected area will have been exposed to some degree. Understanding how variation in that exposure affects different groups/people is a main goal of the natural experiment. It is therefore important to clearly hypothesise the variation applicable for the specific study context. - Adapt to the realities of data collection
Planning data collection for an natural experiment can be challenging when responding to an uncertain, unplanned event. The absence of baselines presents difficulties for assessing the effects of a shock and associated impacts of a programme or intervention. For our natural experiment, we relied on retrospective data collection that was made possible because the start of the Covid-linked shock was a particularly memorable event – often due to the associated lockdowns and government restrictions. Remote data collection, while not optimal, helps to overcome budget and logistical constraints, though selection bias should be accounted for and addressed where possible. Wherever feasible, the programme’s monitoring and administrative data should be exploited in order to overcome challenges associated with in-person data collection and build a better understanding of the context and target population. - Consider natural experiments early when planning evaluations
While planning for a natural experiment in detail before the event is difficult, shock-prone contexts may be more likely to experience important or extreme events than others. In these contexts, practitioners and commissioners could anticipate the kind of likely event that may create the conditions for a natural experiment. Where possible, predicting and planning in advance what information will be useful regardless of the event will have implications for costs, as efficiencies can be achieved by integrating natural experiment approaches into existing monitoring and evaluation practices. - Make the fullest use of existing data to reduce costs
This is important to assist the integration of natural experiments into conventional evaluations and increase complementarity between the approaches. Any natural experiment will rely on three forms of data:
Exposure data, which indicate who is at risk: where, when, of what and to what degree;
Outcome data, which indicate what potentially happens as a result of exposure, that is, what the experiment is testing;
Contextual data, which situate exposure and outcomes historically, sociologically and environmentally, among others.
In the case of our natural experiment, we relied on existing programme and contextual data to understand exposure, collecting only outcome data ourselves.
- Raise the profile of natural experiments when commissioning evaluations
Natural experiments are underappreciated and there is a lack of awareness of what natural experiments are and how they can add value in the evaluation world. The recent recognition of natural experiments by the 2021 Nobel Prize in Economics may help to raise awareness of their value; however, until there are more and well-publicized applications of natural experiments in evaluation, they will remain underappreciated in this field. Evaluation commissioners and partners should consider developing flexible Terms of Reference which allow the possibility for inclusion of natural experiments and flexible use of funds to make the most of opportunities as they arise.
Planning for an unexpected event will always be a challenge, but in the contexts where migrant assistance is common, there is a degree of predictability that an ‘extreme event’ will occur. These five lessons suggest ways in which we can better prepare for the unpredictable.
We would love to hear if you have you been involved in a natural experiment or see an opportunity to apply this methodology. Please get in touch via email: chris.barnett@itad.com
Read the report ‘Using natural experiments in crises: lessons for evaluation’ by Michael Loevinsohn and Tom Gillhespy with Chris Barnett, Leonora Evans-Gutierrez and Callum Taylor.
The authors of this blog would like to thank colleagues at IOM (in particular, Davide Bruscoli) who supported this project, Statistics for Sustainable Development, and our in-country partners; JaRco in Ethiopia, Dansom in Somalia and Sayara in Sudan who conducted extensive fieldwork and without whom this work would not have been possible.
