Itad and 3ie recently carried out the baseline evaluation of the COVAX Facility and Advanced Market Commitment (AMC), commissioned by Gavi, the Vaccine Alliance. These two instruments aimed to guarantee global distribution and access to COVID-19 vaccines, regardless of countries’ income level. (Read our evaluation report for details on the initiative’s successes and challenges).
Uncertain and rapidly changing context
The evaluation was undertaken in an extremely febrile context where a wide range of different interpretations, and stories of what happened were – and continue to be – voiced. As such, the evaluation team and the commissioners of the evaluation (Gavi’s evaluation department) had to ensure that that the evaluation was robust, rigorous and critically, independent.
In addition, the evaluation was implemented in 2022: a time of significant shifts in the shape and implementation of the COVAX Facility and AMC. At a global level, the epidemiological picture of COVID-19 was changing at this time: demand for COVID-19 vaccines in many countries was dramatically shrinking, the ‘acute’ phase of the pandemic was seen to have passed, and focus (at least in richer countries) was on booster shots.
With an abundance of vaccine supply on their hands – something that had been lacking for most of 2021 – COVAX and broader vaccination targets shifted in response, away from the ambitious 70% of country populations globally to be vaccinated, and more towards the unique needs of countries with the least vaccinated populations.
To align with these global shifts, the strategy and operational components of the COVAX Facility and AMC were adjusted, including for example delivery support at country level to get vaccines into arms.
Zooming out, global health narratives started to change from “what can we do?” to “what can we do next time?” – the focus became future pandemic preparedness.
Generating timely, credible and relevant evaluation outputs
This fast-changing environment presented a challenge to position the evaluation outputs to be as timely, meaningful and relevant as possible. We wanted to:
- Capture and reflect the COVAX ‘story’ firmly anchored in the unique context in which decisions were made: what were the parameters in which COVAX was operating, and what was beyond and within its control?
- Elevate evaluation findings/learnings to ensure their relevance beyond a specific set of circumstances in 2020-21, and ability to feed into conversations on preparedness for the next pandemic.
- Communicate essential learnings from the evaluation in a succinct, compelling and timely way, against a shifting context where conversations between global health actors were taking place in real-time.
We share some of our learnings below.
- Anchoring in time and context. The evaluation was the first part of a multi-stage evaluation of the COVAX Facility and AMC, which would document initial design decisions, unpack implementation to date, and present early evidence on emerging results – and, critically, provide an impartial counterweight to media narratives focused on COVAX’s relative failures to date. Recognising that the implementation context was changing, it was vital for us to work with Gavi to communicate this purpose and scope with evaluation audiences, as well as being clear about the time period of focus (March 2020 – December 2021). More broadly, our theory-based evaluation design underpinned this approach through the development of ‘timestamped’ theories of change which captured the thinking at the time, to frame data analysis in the right context.
- Telling the story in a robust, independent way to cut through the noise. Understanding the febrile, sensitive and political context in which our stakeholders were operating, we wanted our evaluation to be methodologically robust to ‘cut through’ the conflicting and contrasting media accounts of what happened. We also had to come across as a credible actor in this space, and immerse ourselves in the technical detail. There were no shortcuts to developing an in-depth understanding of our key informants’ roles and tailoring our lines of enquiry during data collection: our ability to get under the skin of COVAX and use stakeholders’ time effectively involved extensive secondary data review and leveraging technical expertise around the different aspects of the subject matter.
- Understanding shifts in the conversation around COVAX. From the outset, Itad’s objectives with the baseline evaluation had been to provide learnings for course correction, Gavi’s strategic considerations, and for future pandemic preparedness. Our assumption at the start of the evaluation was that the majority of our findings and learnings would constellate around course correction, so feeding in real-time insights and information was going to be a critical feature of the evaluation. However, it became clear to the team during the evaluation that – given the drop in demand for vaccines and end of the ‘acute’ phase of the pandemic – the global conversation was moving away from the immediate COVID-19 vaccine response, towards future pandemic preparedness. Through conversations with Gavi, and the ability of our team to ‘take the temperature’ on key global health actors’ appetites for learnings from COVAX, we were able to anchor our recommendations in the most relevant space: contributing to strategic thinking among multilaterals and governments on how best to respond to the next global pandemic.
Our experience of evaluating the COVAX Facility and AMC underscored how important it is for evaluations to provide insights that account for the context and understanding of pathways to change at the time of intervention. We know from experience that in emergency or crisis contexts where funding flows in rapidly, tracking impact and understanding policy choices can be challenging. We also know that COVID-19 will not be the last pandemic or major global health shock, so the need for robust, agile and context-aware evaluations is going to become ever-more critical.
Further evaluative enquiries around COVAX and other global COVID-19 response mechanisms are ongoing. The more evaluators can engage with these flagship global responses, the stronger and timelier critical insights will be. Moreover, our own learning around how to carry out robust, context-aware evaluations will continue – ensuring that quality evidence and learning underpins the next investments to build the world’s pandemic response capacity.
Related resources
- Read the evaluation report
- Read Gavi’s management response to the evaluation
- Learn more about our work and the Covid-19 pandemic
- Watch ‘Ensuring Access to Vaccines for Future Pandemics: Insights and Lessons Learned from COVAX’, a Centre for Global Development talk featuring Gavi, Africa Centres for Disease Control and Prevention, Coalition for Epidemic Preparedness Innovations, and Itad.
