Itad is working in collaboration with IMC on Ideas to Impact, a DFID-funded programme supporting research and development in climate technologies for developing countries. Itad is leading the design and implementation of Ideas to Impact’s monitoring & evaluation. In this blog, the team share what they’ve found different about monitoring and evaluating innovation prizes for development.
Do innovation prizes work in development? What is their value and use, especially compared to other forms of funding? What pitfalls and dilemmas might prize designers, donors and implementers face when using innovation prizes for social change? As the team leading on evaluation and learning for Ideas to Impact, Itad is pursuing all of these questions but we are also keen to share our learning on monitoring and evaluation (M&E) of innovation prizes for development.
This short series of blog posts draws on discussions the Evaluation Team had once three of the five prizes had been launched and we had finalized our evaluation framework. Our observations come from the M&E team’s experience in designing an M&E framework for Ideas to Impact, how we approached it and why. As the programme continues, we will share what we have learned from implementing the framework and about the value of innovation prizes for solving development problems.
Why are prizes for development interesting for M&E practitioners?
Innovation prizes present a compelling intellectual challenge to those tasked with evaluating their development impact. At the start of Ideas to Impact we wondered if it would be possible to develop an M&E system that works with the private sector aspects of innovation prizes while making people accountable for the consequence of the prize. How would we assess the value of an innovation prize for development and compare it to other funding mechanisms? How would we measure the effect inducement prizes have on development stakeholders’ behaviours?
The scale of some development problems, such as climate change, seem to demand a new way of thinking and a wider pool of problem-solvers but innovation prizes in this context bring a lot of unknowns. Innovation prizes are an untested but increasingly debated option and we were interested in learning more about them. As internal (rather than external) evaluators for a prize, you get to be on the inside, asking questions and pushing design discussions, while having space to step back and connect the prize to wider development outcomes.
Using innovation prizes in development is different to their more conventional uses because they bring with them a greater complexity. Typically, in other contexts, innovation prizes have a short results chain; the innovation prize platform helps you to identify a good idea and the commissioning organisation may evaluate whether the idea works or not. But the evaluation of Ideas to Impact is looking at whether generating ideas in this way is a good way of finding effective solutions to development problems. One of our emerging hypotheses is that a single prize can’t cut through all the complexity and multiple stages in a results chain.
What new evaluation challenges do prizes present compared to traditional evaluations?
Planning the M&E for innovation prizes has been more complicated than we anticipated. The design of innovation prizes to contribute to international development often requires multiple stages and different prize types. Therefore, it is often not just the case of having one prize to deal with one problem, but using a set of prizes designed to catalyse a number of changes within a single results chain. We responded to this challenge by developing an overarching evaluation framework that enables us to draw conclusions across the prizes, while tailoring the evaluations at an individual prize level to examine effectiveness within a specific context.
Prizing for development outcomes brings a lot of risks: from the unpredictability of the solutions submitted, to the unintended consequences that might come of incentivizing behaviour among certain groups of people. Identifying and measuring the change targeted by a prize award is challenging. Meaningful prize evaluation in the context of development needs to go beyond prize award to measuring higher order outcomes and mapping out potentially long and complex results chains. We believe that clearly defining and returning to target outcomes can help here.
Developing a Theory of Change helps you to rehearse best and worst case scenarios and refine the prize based on these insights. But prize design evolves in line with local contexts right up to launch and even beyond as more is learned from implementation. Theories of Change and evaluation plans are therefore also subject to change. Coupled with this, prizes for development are influenced by unexpected events and fluctuations in their external environment; to date the prize teams have had to react on several occasions to shifting government priorities and natural disasters while designing and launching their prizes. The M&E team and its plans have to be able to respond to these changes quickly.
In our next post we explore how we are having to work differently as evaluators of innovation prizes for development and which of the traditional evaluation tools and approaches we are finding most useful.
Cheryl Brown, August 2016
This blog was originally posted on the Ideas to Impact website.
