This week, Itad organised and hosted an Expert Roundtable as part of its ongoing assignment for the Organisation for Economic Co-operation and Development (OECD) looking at results-based payments in the area of governance.
Participants represented the Department for International Development (DFID), the Swedish International Development Cooperation Agency (SIDA), the Centre for Global Development USA, the Overseas Development Institute (ODI), Social Finance, the German Development Institute (DIE), Cordaid, Manchester University, and Institute of Health Policy & Management (BMG).
The overall aim of the study is to look at four areas:
- The types of governance reforms that are most commonly targeted by results-based payment approaches, and which are the most common modalities;
- How well results are delivered by these mechanisms, and the sets of circumstances that can most likely lead to behaviour change leading to long-term impact;
- What, if any, unintended consequences have occurred specific to the modality and approach;
- If it is possible to do a cost-effectiveness assessment, and if there is there evidence that similar results could have been achieved by other means.
Clearly, this is a very wide-ranging set of questions that collectively and individually contain a number of complex sub-questions. The purpose of the Roundtable was to build on the data collection and analysis process that has been undertaken so far, and to explore in depth these issues from both a conceptual and a practical viewpoint based on the participants’ varied and extensive experience. The higher level framing was to consider these issues within what the study is terming ‘the political economy of results-based payments’, and what one participant described as ‘the diplomacy of development’.
An initial discussion was undertaken around the conceptualisation and nature of public sector reform (PSR), this being the specific focus within governance that the study has been asked to look at. The definition for PSR, as set out the TOR and adopted in the early stages of the study as a framing is that it ‘…consists of deliberate changes to the structures and processes of public sector organisations with the objective of getting them to run better. Structural change may include merging or splitting public sector organisations while process change may include redesigning systems, setting quality standards and focusing on capacity building.’
Whilst this definition seems broad, participants at the Roundtable unanimously agreed that the reality of public sector reform is that it is not contained simply to public sector organisations and bureaucracies, but involves a wider and more complex set of inputs and interactions from different stakeholders. This then links to a contextually-driven way of looking at results-based payments, seeing them as linked to and mutually influencing a range of other programmes and mechanisms, which has ramifications for what theory of change – discussed more below – really underpins the use of these mechanisms, and the change that is expected to be delivered by results-based payments alone, and as part of complex systems.
Another example of the more complex and messy reality is that public sector reform can happen in programmes that are not only not focused only on public sector reform, but are focused much more on service delivery rather than governance. A good example of this is the ongoing programme of work that Cordaid has been doing to roll out results-based financing in the health sector across Africa. A key part of this process is the participation of communities in the data quality assurance, and from this holding service providers and authorities to account with the view to improving systems and processes – and as such moving much more closely into the PSR space.
In the afternoon, a very interesting discussion took place on the theory, or theories, of change and the associated assumptions that underpin results-based payments. A useful framing and start point for this is the recent article from CGD which, whilst noting that this area is composed of many theories of change, mooted four potential theories of change for consideration:
(1) that the offer of a financial incentive will lead to some behavior change by the recipient; (2) that the performance funding makes results visible in a way that improves management; (3) that the focus on results will improve accountability to constituents or beneficiaries; or (4) that the agreement gives recipients more discretion and autonomy to innovate and adapt their activities.
These four suggestions were debated at some length by participants, who also suggested other potential additions to this, such as facilitating donor reform at the same time as recipient reform, the role of peer pressure and of building evidence, and of providing leverage to reformers within government. Findings such as in the recent evaluation of DFID’s results-based education programme in Rwanda were also considered, which involved a ‘deduced theory of change’ to test the programme against, finding it falling short. Whether these are really theories of change, or objectives, or something else, and how they can be useful to policy-makers and programme managers will be an issue contained in the final paper for the study.
To end the day, the discussion moved to consider how well results-based payments align with key current sectoral debates as represented by the Doing Development Differently (DDD) manifesto, itself influenced by the work of Matt Andrews’ work on problem-driven iterative adaptation. On the face of it, there seems little that links results-based payments to these areas: indeed, it could be argued that they are at odds. One proposes a set of pre-determined results that have to be met for payments to be made, the other proposes a grounded, locally-led approach where investment in small bets allows for an uncertain path towards an end result.
However, the discussion on this revealed that, depending on the results-based mechanism in question, there may in fact be a lot of complementarities between results-based payments and these approaches, and, though limited at present, further evidence in this area could prove extremely useful for policy-makers and those looking to design and implement more flexible and adaptive programmes. Results-based payments at outcome levels could give donors confidence that their funds are well spend, while allowing programmes to adapt and iterate. Or, in the words of one participants, “results-based payments can make DDD less scary, and DDD can make results-based payments less scary”.
The thorny issue of value for money (VFM) also was considered. As with most discussions of VFM, challenges arose from the variety of possible analytical and conceptual approaches to this. The study has found VFM addressed in a range of different ways relating to results based payments, and differing conclusions as to if this is being delivered. Two elements of this were posed at the roundtable: if payments are withheld in a results-based payment programme, then at a basic level that money has been saved for whatever countries’ taxpayers ultimately supported, though of course this needs to be looked at across the whole duration of the programme, and possibly beyond it to take account of longer term benefits and costs, as far as possible. Secondly, if resources are spent delivering the results in question, these are resources that then weren’t spent elsewhere, so how far can VFM take into consideration this opportunity cost?
Itad will shortly be drafting the final products for this study, which will be submitted to OECD for review within the next weeks, when decisions will be made on publishing and publicising.
The meeting was held under Chatham House rules in order to facilitate a free and open discussion.
Jake Allen, with Florian Schatz, October 2015
