The Innovation in Methodologies prize recognises individuals or teams that have introduced effective novel thinking with respect to methods, to the evaluation profession. It was one of three prizes awarded in 2023 by the UK Evaluation Society to celebrate the accomplishments of members of the evaluation profession, and to support their development. The prize is sponsored by Ipsos UK.
The award-winning IMPACT study was an impact evaluation of the ‘EU-IOM Joint Initiative for Migrant Protection and Reintegration in the Horn of Africa’, a flagship programme funded by the European Union and implemented by the International Organization for Migration (IOM) to save lives, protect and assist migrants along key migration routes in Africa. The complexity and the unique evaluation needs of this programme led the team to develop innovative strategies for three common impact evaluation challenges: finding a valid ‘counterfactual’ group; matching processes for return and reintegration and collecting data in fragile contexts.
Dr Laura Hayward, Deputy Head of Evaluation, Ipsos UK, said:
“Ipsos congratulates the evaluation team for their achievement in winning the Ipsos Innovation in Evaluation prize. The judging panel were impressed with the project team’s innovative approach to addressing key methodological challenges frequently faced in evaluation (the absence of a clear counterfactual, no access to a true baseline, and challenges around access and sampling, particularly where research participants are from vulnerable groups). Ipsos is delighted to have sponsored the Innovation in Evaluation Methodologies prize for the third year in a row. Innovation is a critical part of our offer here at Ipsos and we want to encourage the evaluation community to think bigger and smarter around how we apply evaluation to rise to global challenges.”
On receiving the award, Itad Consultant Callum Taylor, said:
“This project demonstrated the evaluation team’s ability to develop innovative methodologies that meet the requirements of clients, respond to challenging circumstances, and push the boundaries of what evaluations can achieve. Faced with complex systems and global challenges, we are delighted to have contributed to novel evaluation methodologies that now have proven value and can be applied across a range of contexts. This award is testament to Itad and our partners’ commitment to innovation and quality, and our resilience in the face of challenging evaluation contexts.”
Three methodological innovations for common challenge
The IMPACT Study evaluation team developed and deployed three methodological innovations which captured the attention of the judges:
Innovation 1: calibrated impact evaluation
A key challenge was the absence of a standard counterfactual group to measure programme impact. Counterfactuals are problematic in reintegration evaluation as non-migrants, by definition, differ to returnees; and, as IOM provides support to almost all returnees, there was no comparable control group. To address this, the evaluation team adopted a ‘calibration’ group approach, comparing returnees with demographically similar non-migrant residents in their communities.
Innovation 2: matched snowball sampling
To identify suitable non-migrants, the team devised a snowball sampling strategy, the first known use in the reintegration evaluation setting. Surveyed returnees were recontacted and asked to identify suitable non-migrants of the same sex in their communities, with similar age, education level and length of residency in the community. Once these nominated non-migrants were verified as suitable matches, they responded to the same questionnaire as the returnees. Returnees and non-migrants were paid a small incentive on the successful completion of the process. This approach efficiently matched one returnee with a similar non-migrant.
Innovation 3: redefining baseline data collection
Collecting timeseries data is challenging as returnees often relocate within their country of origin, and this is often exacerbated by fragile and insecure contexts. In this case the challenges were compounded by the COVID-19 pandemic, which dramatically reduced the numbers of returnees arriving, and significant political instability in Ethiopia, Somalia, and Sudan. To overcome these challenges, the team conducted retrospective baseline data collection at the same time as the endline. The empirical evidence indicated that this was a reliable, practical, and cost-effective option for obtaining timeseries data.
Alongside these methodological innovations, the evaluation team further demonstrated their ability to turn challenges into opportunities through deploying a natural experiment component focusing on the impact of the pandemic on the returnees’ wellbeing.
Methodological and real-world impacts
The evaluation has yielded substantial methodological and real-world impacts. It has shown that retrospective data can provide a practical and cost-effective solution for studying returnees; and demonstrated that the IMPACT Study’s successful use of calibration group of matched non-migrants can form a useful reference cohort, elevating standards for programme evaluation in similar contexts. The additional analytical work validating the existing IOM reintegration measurement framework provided evidence-based advocacy for improving this framework for future programmes, a process that is currently ongoing within IOM.
Dr Katie Kuschminder is a leading scholar of migration and refugee reintegration, and member of the IMPACT Study team. She said:
“The IMPACT Study provides great value to the reintegration community as the first large-scale and comparative impact evaluation on reintegration. The focus on migrants’ protection in transit and their subsequent return and reintegration reflects the changing nature of migration and return outside the EU with increased south-south reintegration dynamics. This highly vulnerable group of returnees have significantly benefitted from the reintegration assistance as clearly shown by the Study’s results. The IMPACT Study sets a new standard for evaluating reintegration assistance and provides multiple insights for improving reintegration measurement and practice.”
The evaluation has also confirmed the value of assisted voluntary return and reintegration assistance. Findings from the evaluation were presented at an IOM regional workshop in Nairobi and the European Commission in Brussels, where they were well received. Early indications suggest the findings and recommendations will play a crucial role in informing future programming and evaluation decisions along with refining reintegration measurement.
Davide Bruscoli, Regional Information Management Officer for East and Horn of Africa of the International Organization for Migration, said:
“I am glad to say that the study achieved all its objectives and pushed the evaluability frontier to include a topical area (assisted voluntary return and reintegration) of development intervention that was previously left out. Throughout the evaluation process, the evaluation team, led by Andrew Pinney and Itad, demonstrated a high level of professionalism, expertise, and dedication. The study was conducted in close collaboration with IOM – even, I would say, in a spirit of partnership rather than simple service provision – with Itad and the evaluation team walking several ‘extra miles’.”
Reflecting on the potential impact of the evaluation, he added:
“At a time when funding for assisted voluntary return and reintegration initiatives is increasingly restricted to migratory routes going towards Europe, the IMPACT Study will also serve as a key advocacy tool for the large protection, return and sustainable reintegration of migrants from the Horn of Africa going towards Gulf Cooperation Countries and South Africa.”
Dr Chris Barnett of Itad, who received the award on behalf of the broader IMPACT Study team, wishes to dedicate it to the hundreds of returnees in Ethiopia, Somalia and Sudan who gave their time for this study and shared their stories in the hope that this would ultimately help improve the condition of fellow migrants in the future.
In accepting the award, Chris acknowledged the contribution of colleagues at IOM (in particular, Davide Bruscoli, who conceived and supervised the IMPACT Study); Statistics for Sustainable Development (especially Dr Andrew Pinney, Technical Lead of the evaluation team, Alex Thompson, Analyst, and Dave Mills, Lead Data Engineer); Applied Ecology Research (Dr Michael Loevinsohn); Dr Katie Kuschminder; Leonora Evans-Gutierrez and Callum Taylor of Itad; and Itad’s in-country partners: JaRco Consulting in Ethiopia, Dansom and the Somali Research and Development Institute in Somalia, and Sayara International in Sudan, who conducted extensive fieldwork and without whom this work would not have been possible.
The IMPACT Study was possible thanks to a generous contribution of the European Union and the active support of DG-INTPA (in particular, Inès Balança and Victoria Garcia-Guillen).
Related resources
- Read more about the IMPACT Study and access the reports
- Learn about the EU-IOM Joint Initiative for Migrant Protection and Reintegration in the Horn of Africa’
- E-learning self-paced course on Impact Evaluations for Return and Reintegration Programmes (developed by Stats4SD for IOM, with funding from the European Union).
- Read our reflections on the benefits on natural experiments
