Conflict, natural disasters and political turmoil often disrupt evaluation plans. But what if these challenges were instead used to stress-test the performance of humanitarian or development programmes?
We undertook an innovative natural experiment (NE) to assess how support from the International Organisation for Migration (IOM) contributed to returnees’ resilience to the ‘shock’ of the Covid-19 pandemic. The experiment was part of a wider evaluation of the EU-IOM Joint Initiative for Migrant Protection and Reintegration in the Horn of Africa region. This assessed how well its assistance helped migrants to voluntarily return to and reintegrate into their countries of origin.
In this blog, we outline our key findings as well as three benefits of using natural experiments in similar contexts prone to marked uncertainty and extreme events.
What is a natural experiment?
Michael Loevinsohn, lead NE investigator, defines a natural experiment as “an observational study of a sharp, well-defined but unplanned change or extreme event. Natural experiments hinge on identifying an uncontrolled but opportune ‘intervention’, typically of a kind or on a scale that could not – ethically or feasibly – be implemented deliberately…”.
Being immersed in real-world events, NEs provide evidence of what happened, rather than hypotheses about what might happen. NEs can thus offer insights with a level of confidence that conventional evaluations might have difficulty matching.
Understanding the ‘Covid-linked shock’
Our natural experiment examined the effect of the Covid-19 pandemic on almost 2,000 returnees in Ethiopia, Sudan and Somalia. The experiment aimed to assess the contribution of returnees’ actions and different components of the Joint Initiative’s assistance to people’s resilience to the ‘Covid-linked shock’. Whilst the main event informing the natural experiment was the Covid-19 pandemic and associated control measures, important co-occurring shocks– conflict, floods, locusts – exacerbated the effects of the pandemic. Collectively, these are referred to as the Covid-linked shock (CLS).
We examined the CLS’ impact on returnees across different well-being domains ( income, food security, health, education, housing and family and community acceptance) and explored how returnees had responded to its effects.
Mitigating and recovering from the CLS’ impacts
Using fixed-effect multivariate regression analysis and integrated qualitative research, we identified the factors that helped or hindered returnees to mitigate the CLS’ effects on their wellbeing and to recover from them – two key aspects of resilience. This analysis provided valuable insights into returnee agency in the face of shocks, the actions they adopted in response, and what conditions and characteristics influenced their resilience.
Key findings included:
- Livelihood adaptations
Covid-related lockdowns and transport restrictions limited access to work and markets, reduced availability of daily goods, and depressed returnees’ purchasing power. Food security declined everywhere, most severely in Amhara, Ethiopia. A common response to these shocks by returnees was to adjust their primary or secondary source of livelihood. In particular, increased engagement in agriculture – even in urban areas – was an effective strategy. This has implications for IOM programming. Returnees who engaged more in agriculture were able to mitigate the impacts of the Covid-linked shock and had higher rates of recovery in most well-being domains. Returnees saw other emerging livelihood opportunities but often weren’t able to seize them, lacking key skills or capital. Adaptive assistance from the EU-IOM Joint Initiative could have made a big difference to them. - Support from social networks
As well as making changes to their livelihoods, returnees changed their food sources and eating patterns to limit the deterioration of their wellbeing. Many returnees drew on support from family and social networks in the form of loans and short-term support as well as working more to increase food production. Returnees had much less capability to influence changes vis-a-vis health and education. Returnees who had been in the country the longest had more opportunities to develop livelihoods and social networks prior to the Covid-linked shock and therefore protect their wellbeing. - Added-value of the EU-IOM assistance
The assistance provided to returnees before their return to their countries was highly praised, with many returnees believing it had saved them from significant danger. Upon their return, many returnees reported having used the economic assistance to create microbusinesses that benefited their wellbeing. However, some criticised the scale of assistance and its appropriateness to their situation. Where assistance had been provided in a timely fashion, this helped returnees to mitigate the worst impacts of the Covid-linked shock, but less so on recovery. The provision of cash assistance was found to be most effective. A modest but timely and well-targeted cash intervention enabled the Ethiopian returnees who received it to mitigate and recover from some of the CLS’ impacts.
Browse the study reports on the IOM website
Three benefits of using natural experiments as evaluative tools
- Natural experiments provide evidence of an intervention being stress-tested
By observing how programmes perform under stress, and the extent to which those programmes help people and communities deal with that stress, we can learn more about people’s resilience and programme effectiveness. - Natural experiment can provide a broader perspective than programme-centric evaluations
Natural experiments are less programme-centric than a conventional impact evaluation. They take a more holistic view when seeking to understand how an event has influenced the lives of individuals and communities, rather than a narrow focus on only what the programme is providing. In this case we were able to look at effects on returnees’ well-being and assess the effectiveness of their responses, well beyond what the programme or most evaluations would ordinarily consider. - Natural experiments can be more suited to dynamic and shock-prone contexts
Natural experiments can respond to a changing implementation context and exploit shocks that might disrupt traditional evaluations. The latter that will typically be designed around fixed points in time determined by a programme or grant cycle rather than by significant real-world events. Not all extreme events that frame natural experiments will be ‘shocking’ but they will be memorable, which can aid respondents in recalling their experiences before and after the event or shock in question.
To conclude, natural experiments add value to evaluative processes as they allow us to learn more about people’s resilience in fragile and conflict-affected contexts – as well as how effective programme assistance is at supporting mitigation and recovery from the effects of shocks. By drawing on real-life events, we can realize several advantages over traditional designs such as RCTs and quasi-experiments: we can discuss events with those who have lived through them without fear of contaminating our control group; we are not bound by baseline/endline cycles; we can make use of retrospective and existing data, and we can exploit spatial and temporal variations in a programme’s implementation to rigorously test hypotheses.
Read the report ‘Using natural experiments in crises: lessons for evaluation’ by Michael Loevinsohn and Tom Gillhespy with Chris Barnett, Leonora Evans-Gutierrez and Callum Taylor.
The authors of this blog would like to thank colleagues at IOM (in particular, Davide Bruscoli) who supported this project, Statistics for Sustainable Development, and our in-country partners; JaRco in Ethiopia, Dansom in Somalia and Sayara in Sudan who conducted extensive fieldwork and without whom this work would not have been possible.
