Migration is a growing reality for millions of people across the globe. In 2021, there were about 281 million international migrants, representing 3.6 per cent of the global population.
Providing effective assistance to vulnerable and stranded migrants returning home voluntarily is a vital pillar of managing safe, dignified and legal migration. For this reason it should be high on the global policy agenda. However, there is scant evidence on the impact of programmes and policies aimed at assisting voluntary return and reintegration.
To help bridge this evidence gap, in 2020, the International Organization for Migration (IOM) engaged Itad and partners to work with them on an IMPACT Study to evaluate the impact of the ‘EU-IOM Joint Initiative for Migrant Protection and Reintegration in the Horn of Africa’. The Joint Initiative, funded by the European Union, was one of the largest ever EU investments in migration at €500 million across three regions of Africa. In the Horn of Africa, it sought to help more than 16,000 returnees in Ethiopia, Somalia and Sudan from 2017-23.
Real-world impacts
The evaluation demonstrated substantial real-world impacts.
Dr Katie Kuschminder is a leading scholar of migration and refugee reintegration, and member of the IMPACT Study team. She said:
“The IMPACT Study provides great value to the reintegration community as the first large-scale and comparative impact evaluation on reintegration. The focus on migrants’ protection in transit and their subsequent return and reintegration reflects the changing nature of migration and return outside the EU with increased south-south reintegration dynamics. This highly vulnerable group of returnees have significantly benefitted from the reintegration assistance as clearly shown by the Study’s results. The IMPACT Study sets a new standard for evaluating reintegration assistance and provides multiple insights for improving reintegration measurement and practice.”
The evaluation confirmed the value of assisted voluntary return and reintegration assistance. Findings from the evaluation were presented at an IOM regional workshop in Nairobi and the European Commission in Brussels, where they were well received. Early indications suggest the findings and recommendations will play a crucial role in informing future programming and evaluation decisions along with refining reintegration measurement.
Davide Bruscoli, Regional Information Management Officer for East and Horn of Africa of the International Organization for Migration, said:
“I am glad to say that the study achieved all its objectives and pushed the evaluability frontier to include a topical area (assisted voluntary return and reintegration) of development intervention that was previously left out. Throughout the evaluation process, the evaluation team, led by Andrew Pinney and Itad, demonstrated a high level of professionalism, expertise, and dedication. The study was conducted in close collaboration with IOM – even, I would say, in a spirit of partnership rather than simple service provision – with Itad and the evaluation team walking several ‘extra miles’.”
Reflecting on the potential impact of the evaluation, he added:
“At a time when funding for assisted voluntary return and reintegration initiatives is increasingly restricted to migratory routes going towards Europe, the IMPACT Study will also serve as a key advocacy tool for the large protection, return and sustainable reintegration of migrants from the Horn of Africa going towards Gulf Cooperation Countries and South Africa.”
Award-winning methodological innovations
The complexity and the unique evaluation needs of this programme led the team to develop innovative strategies for three common impact evaluation challenges: finding a valid ‘counterfactual’ group; matching processes for return and reintegration and collecting data in fragile contexts.
The team’s efforts were recognised with a UK Evaluation Society Prize for ‘Innovation in Methodologies’.
Dr Laura Hayward, Deputy Head of Evaluation at Ipsos UK who sponsored the Prize, said:
“The judging panel were impressed with the project team’s innovative approach to addressing key methodological challenges frequently faced in evaluation (the absence of a clear counterfactual, no access to a true baseline, and challenges around access and sampling, particularly where research participants are from vulnerable groups).”
The IMPACT Study evaluation team developed and deployed three methodological innovations which captured the attention of the judges:
Innovation one: calibrated impact evaluation
A key challenge was the absence of a standard counterfactual group to measure programme impact. Counterfactuals are problematic in reintegration evaluation as non-migrants, by definition, differ to returnees; and, as IOM provides support to almost all returnees, there was no comparable control group. To address this, the evaluation team adopted a ‘calibration’ group approach, comparing returnees with demographically similar non-migrant residents in their communities.
Innovation two: matched snowball sampling
To identify suitable non-migrants, the team devised a snowball sampling strategy, the first known use in the reintegration evaluation setting. Surveyed returnees were recontacted and asked to identify suitable non-migrants of the same sex in their communities, with similar age, education level and length of residency in the community. Once these nominated non-migrants were verified as suitable matches, they responded to the same questionnaire as the returnees. Returnees and non-migrants were paid a small incentive on the successful completion of the process. This approach efficiently matched one returnee with a similar non-migrant.
Innovation three: redefining baseline data collection
Collecting timeseries data is challenging as returnees often relocate within their country of origin, and this is often exacerbated by fragile and insecure contexts. In this case the challenges were compounded by the COVID-19 pandemic, which dramatically reduced the numbers of returnees arriving, and significant political instability in Ethiopia, Somalia, and Sudan. To overcome these challenges, the team conducted retrospective baseline data collection at the same time as the endline. The empirical evidence indicated that this was a reliable, practical, and cost-effective option for obtaining timeseries data.
Unique use of a natural experiment
In addition to these three methodological innovations, the evaluation team further demonstrated their ability to turn challenges into opportunities through deploying a natural experiment component focusing on the impact of the pandemic on the returnees’ wellbeing. Natural experiments are rarely done in a global development context but can offer insights with a level of confidence that is hard to match in conventional evaluations. Importantly, they provide data from a real-world event – evidence of what happened, rather than hypotheses about what might happen. In this case, analysis from the natural experiment boosted understandings of returnee agency in the face of shocks, the actions they adopted in response, and what conditions and characteristics influenced their resilience.
Dr Chris Barnett of Itad, who received the UK Evaluation Society award on behalf of the broader IMPACT Study team, dedicated it to the hundreds of returnees in Ethiopia, Somalia and Sudan who gave their time for the study and shared their stories in the hope that this would ultimately help improve the condition of fellow migrants in the future.
In accepting the award, Chris acknowledged the contribution of colleagues at IOM (in particular, Davide Bruscoli, who conceived and supervised the IMPACT Study); Statistics for Sustainable Development (especially Dr Andrew Pinney, Technical Lead of the evaluation team, Alex Thompson, Analyst, and Dave Mills, Lead Data Engineer); Applied Ecology Research (Dr Michael Loevinsohn); Dr Katie Kuschminder; Leonora Evans-Gutierrez and Callum Taylor of Itad; and Itad’s in-country partners: JaRco Consulting in Ethiopia, Dansom and the Somali Research and Development Institute in Somalia, and Sayara International in Sudan, who conducted extensive fieldwork and without whom this work would not have been possible.
The IMPACT Study was possible thanks to a generous contribution of the European Union and the active support of DG-INTPA (in particular, Inès Balança and Victoria Garcia-Guillen).
Related resources
- Read more about the IMPACT Study and access the reports
- Learn about the EU-IOM Joint Initiative for Migrant Protection and Reintegration in the Horn of Africa’
- E-learning self-paced course on Impact Evaluations for Return and Reintegration Programmes (developed by Stats4SD for IOM, with funding from the European Union).
- Read our reflections on the benefits on natural experiments
