We’ll be sharing learning from the evaluation alongside CRS colleagues at the Savings Led Working Group session on Members Day of the SEEP Annual Conference – so pop by if you would like to learn more.
What is an ex-post evaluation?
Ex-post evaluations are (by definition) done after the project has closed. There is no hard and fast rule on exactly when an ex-post evaluation should be done but as the aim of an ex-post evaluations is to assess the sustainability of results and impacts, usually some time will need to have passed to make this assessment.
A little bit about EFI
EFI was a Mastercard Foundation-funded program in Burkina Faso, Senegal, Zambia and Uganda whose core goal was to ensure that vulnerable households experienced greater financial inclusion. Within EFI, Private Service Providers (PSPs) formed and facilitated savings groups using CRS’ Savings and Internal Lending Communities (SILC) methodology, with the SILC groups responsible for paying the PSP a small fee for the services that they provide.
This payment is intended to improve sustainability by incentivising the groups’ facilitators to form and train new groups, as well as providing continued support to existing groups, beyond the end of the project.
A little bit about the evaluation
So, if the aim of the PSP model is sustainability, you need an evaluation that can test this! Evaluation at the end of project implementation can assess indications of results that might be sustained into the future. However, if you wait until some time has passed after activities have ended, then there is much clearer evidence on which activities and results are ongoing – and how likely these are to continue. Uganda was also a great test case for the evaluation because CRS hadn’t provided any follow-on support.
Our evaluation set out to assess the extent to which the EFI-trained PSPs and their SILC groups were still functioning 19 months after the programme ended and the extent to which the PSP model had contributed to the sustainability of activities and results.
What the ex-post evaluation found
We found a handful of findings that were only possible because it was an ex-post evaluation:
- There were 56% more reported groups among the sampled PSPs at the time of data collection than there were at the end of the project.
- Half of the PSP networks established within the sample are still functioning (to some extent).
- PSPs continued to receive remuneration for the work that they did, 19 months after project closure. However, there were inconsistencies in frequency and scale of remuneration, as well as variation in strategies to sensitize communities on the need to pay.
This only covers a fraction of the findings but we were able to conclude that the PSP model appeared to be highly sustainable. The evaluation also found that there were challenges to sustainability which could be addressed in future delivery of the PSP model. Significantly, the PSP model was designed with sustainability in mind – and this evaluation provides good evidence that PSPs were still operating 19 months after the end of the project.
What made the evaluation possible
We get it. It isn’t always easy to do ex-post evaluations. Evaluations are usually included in donor-implementer contracts, which end shortly after the project ends, leaving implementers without the resources to go back and evaluate 18 months later. This often results in a lack of funding and an absence of project staff. This is also combined with new projects starting up, obscuring opportunities for project-specific findings and learning as it’s not possible to attribute results to a specific project.
In many ways, we were lucky. Itad implements the Mastercard Foundations Savings Learning Lab, a six-year initiative that supports learning among the Foundation’s savings sector portfolio programmes – including EFI. EFI closed in the Learning Lab’s second year and with support from the Foundation and enthusiasm from CRS, we set aside some resource to continue this learning post-project. So, we had funding!
We also worked with incredibly motivated ex-EFI, CRS staff who made time to actively engage in the evaluation process and facilitate links to the PSP network, PSPs and SILC group members. So, we had the people!
And, no-one had implemented a similar PSP model in supported districts of Uganda since the end of EFI. So, we were also able to attribute!
Why we should strive to do more ex-post evaluations
Despite these challenges, and recognising it isn’t always easy, doesn’t mean it is not possible. And with projects like EFI where sustainability was central to its model, we would say it’s essential to assess whether the programme worked and how the model can be improved.
Unfortunately, practitioners and evaluators can shout all we like but the onus is on funders. We need funders to carve out dedicated resource for ex-post evaluations. This is even more important for programmes that have the development of replicable and sustainable models at their core. For some projects, this can be anticipated – and planned for – at project design stage. Other projects may show promise for learning on sustainability, unexpectedly, during implementation. Dedicated funding pots or call-down contracts for ex-post evaluations are just a couple of ways donors might be able to resource ex-post evaluations when there is a clear need for additional learning on the sustainability project results.
This learning should lead to better decision making, more effective use of donor funds and ultimately, more sustainable outcomes for beneficiaries.
