The role of evidence in policymaking
Evidence is crucial to successful policy-making. It helps governments identify the most significant challenges in our societies, as well as what works and doesn’t work to tackle them. But despite a growing global body of research available to inform decisions, policy-makers do not always have the knowledge, skills, incentives and systems to use research evidence effectively.
Building capacity to use research evidence
To help bridge this gap in low- and middle-income countries, the UK Government funded a major programme called Building Capacity to Use Research Evidence (BCURE). The £15.7 million programme ran from 2013-17 across 12 countries in Africa and Asia.
Running in parallel with BCURE from 2014-17, Itad conducted an independent evaluation of the programme in partnership with Stellenbosch University. (Read the full report and the executive summary on our website).
The evaluation applied an innovative realist methodology and had significant impact in shaping thinking, resourcing and programming around building capacity for evidence-informed policy.
Lessons on how and why capacity building works, for whom, in what contexts
Realist evaluation seeks to respond to the complex environments in which many evaluations are conducted, moving beyond the blunt question, ‘does it work?’ and instead asking, ‘what works in which circumstances and for whom?’.
The realist approach allowed the BCURE evaluation to explore the different ways partners attempted to build capacity, investigate whether and how far they worked, and unpick the ‘how and why’ behind success stories (and failures) in different contexts. This resulted in a set of well-evidenced insights on the underlying ‘causal mechanisms’ behind successful strategies, and lessons on how future projects could harness these mechanisms to build capacity for evidence use. This infographic summarises the headline messages from the evaluation, and the final set of causal mechanisms that emerged from three years of rigorous iteration and testing.
The evaluation recommendations directly informed the development of a £17 million follow-up programme launched by the UK Government’s Department for International Development (DFID) in 2018, Strengthening the Use of Evidence for Development Impact (SEDI). In particular, SEDI incorporated political economy analysis and flexible management arrangements, both of which were missing from the BCURE programme. This responded to insights from the evaluation on the importance of identifying entry points to evidence-informed policymaking given constantly shifting political windows of opportunity; and the value of working flexibly and collaboratively with government partners to co-produce policies or tools, rather than providing fixed interventions like stand-alone training.
In their Management Response to the evaluation, DFID said:
“The final independent evaluation report sets out very clear lessons and recommendations for DFID to take forward […] We agree with all six recommendations, which have collectively fed into five key principles for our future evidence-informed programming in DFID.’”
Lessons on the ‘nuts and bolts’ of realist evaluation
The BCURE evaluation was Itad’s first foray into realist evaluation, at a time when there was limited experience of applying the approach in the international development sector.
By extensively documenting our experiences, challenges and lessons we helped other evaluators learn when and how to apply the approach. In 2016, a CDI Practice Paper reflecting on lessons from the first year of the BCURE evaluation was among the first articles on how to navigate the complexities of realist evaluation and operationalise it outside of an academic setting, and was downloaded more than 4000 times. In 2018, Itad established a Realist Evaluation Learning Group to share and capture learning from across our growing number of realist evaluations with the wider world, including through a series of blogs. In 2020, we were invited to contribute to a special edition of New Directions for Evaluation on how to use mechanisms of realist evaluation. Lessons from the completed evaluation were also included in a second Centre for Development Practice Paper, which discussed the value of realist evaluation in complex programmes. Collectively, these articles have made a significant contribution to evaluation knowledge, with insights adopted by other evaluators working in a wide range of fields, from healthcare to housing to climate change.
Highlighting the value of a realist approach in complex contexts
The BCURE evaluation contributed valuable lessons on how to build capacity for evidence-informed policy making, which remain highly relevant today. More than this, it highlights the value of evaluation approaches that can grapple with the complex realities of policy making, to generate nuanced insights that can guide decisions about rolling out or scaling up interventions while taking into account real life constraints.
This impact story was developed by Viktoria Beran of Itad’s Research Analyst Programme, with contributions from Melanie Punton.
