At Itad, we see choosing an evaluation approach as an art as much as a science: there’s always a lot to consider. As a result, our evaluation designs are often hybrids, drawing on a range of different approaches in order to tick all the boxes.
In fact, deciding which approach to use means considering the interplay between the evaluation questions, the approaches that might be feasible given the nature of the programme and its context, the various other interests and preferences of the commissioner, and of course what is possible given the resources available (see Elliot Stern’s and Barbara Befani’s evaluation design triangles). Yet, how do these considerations influence realist evaluations?
This second blog from the Itad Realist Evaluation Learning Group asks: how far can realist evaluation be modified, watered down, or ‘mixed and matched’ with other evaluation approaches? Or should we aim to be ‘purists’ in order to retain the integrity of the approach? We came up with four insights – what do you think? Let us know in the comments.
-
Be careful about saying you’re doing a ‘realist-inspired’ evaluation…
We all agreed that realist evaluation is more than an approach: it’s a way of thinking. More precisely, it’s a specific way of thinking about causality and how to test causal relationships, based on a realist understanding of reality and how we can know about it (see Gill Westhorp’s great introductory paper for a primer). Applying it means ‘thinking like a realist’ at all stages of an evaluation, from design, through to sampling and data collection, and then undertaking analysis in a very specific way. We felt that approaches that just adopt ‘elements’ of realist evaluation are likely to miss this, especially when implemented by evaluators who haven’t applied the approach before. There’s a risk of falling into common traps like conflating mechanisms with interventions, or failing to grasp the nuances of context-mechanism-outcome configurations and how to develop and test them. There’s a real possibility that a ‘realist inspired’ evaluation has all the jargon of a realist evaluation but none of the benefits we highlighted in our first blog.
-
…but realist ways of thinking can be very helpful when used informally
On the other hand, we’ve found that the realist evaluation ‘way of thinking’ can be beneficial even when the application of the approach isn’t perfect. For example, in the BRACED evaluation the Itad team introduced realist ideas to project implementing partners, who have been reporting their results in terms of contexts, mechanisms and outcomes – and although not all of them became expert realist evaluators overnight, many partners said this encouraged them to really think through how and why they expected their programmes to work. Some members of our learning group also feel that we take the realist way of thinking into other evaluations, now that we’ve internalised the approach. The practice of digging beneath the surface to find underlying causal mechanisms (and the contexts that enable or constrain them) can be especially helpful when getting to grips with what a programme is hoping to do, developing theories of change, or conducting interviews.
-
Other approaches might be needed to answer questions about contribution
In both the BRACED and BCURE evaluations, commissioners wanted to know whether the programmes had made a meaningful contribution to observed outcomes, and how important that contribution was in a context where many other factors are influencing change. We found this question was difficult to answer using realist evaluation alone. Realist evaluation will give you a tested theory in the form of a set of confirmed CMO configurations which explain what worked, for whom, in what circumstances. But these don’t necessarily add up to a clear picture of how important the programme’s contribution was to achieving any given outcome, in comparison to other factors. (*We’ve put a more techy breakdown of how we understand realist evaluation approach to contribution at the bottom of this blog, if you’re interested!) However, contribution analysis is specifically designed to answer the question ‘how far did my programme contribute to change’. Can the two approaches be combined?
One important challenge to this is that realist evaluation and contribution analysis seek to demonstrate causality in quite different ways, even though they are both use a generative model of causation. Contribution analysis works through establishing a plausible contribution claim – looking to increase our confidence that a programme contributed to change through weighing up the relative importance of the programme vs other factors, in a fairly pragmatic way. Realist evaluation establishes causality through a more formal process of developing and testing more nuanced, granular theories (CMOs), about how and why change happened. How can you combine these two approaches in light of these differences?
-
It might be better to talk about ‘sequencing’ rather than ‘combining’ approaches.
In order to avoid muddying the theoretical waters, we felt it makes more sense to talk about ‘sequencing’ approaches rather than ‘combining’ or ‘merging’ them. This is the approach we took in BCURE and BRACED – applying a ‘contribution analysis’ lens and a ‘realist lens’ separately in a three step analytical process, that involved focussing in on a specific outcome and then sequentially ‘digging deeper’…
- First, establishing that an outcome did (or didn’t) happen.
- Then looking for evidence that the programme did (or didn’t) contribute, and the influence of other (non-programme) factors: the contribution analysis lens, in a rather top line way.
- Finally, digging deeper using the realist lens, to unpack how and why the programme did (or didn’t) contribute.
This is described further in the BCURE final evaluation report and annexes. While this solution may not be a pure application of either realist evaluation or contribution analysis, we found that it helped respond to the pragmatic needs of evaluation commissioners and users by providing a clear contribution judgement – while still generating the detailed, granular ‘how and why’ explanations that realist evaluation excels at.
These four insights are just scratching the surface. There are many other interesting conversations to be had about how realist evaluation can be combined with other theory based approaches including process tracing and QCA – and that’s not even getting into the heated debate currently ongoing in the realist evaluation community about whether it makes sense to do ‘realist RCTs.’ We’re looking forward to continuing these techy conversations over the coming months!
Realist Evaluation Learning Group, August 2018
* More detailed musings about how realist evaluation deals with contribution:
Realist evaluation establishes causality through developing and testing granular theories (CMOs) about how and why change happened. While realist evaluation doesn’t explicitly talk about ‘programme contribution’, the programme is part of some theories, and not part of others. Through testing the theories, the evaluation aims to confirm that an outcome in x context emerged through y mechanism – and how the resources introduced by the programme gave rise to this (or how factors other than the programme gave rise to mechanisms that led to the outcomes). Therefore you have attributed the change to a particular causal explanation which may or may not include the programme. In this sense realist evaluation is more binary than contribution analysis – the programme is either part of a tested, confirmed CMO or it is not (in x context).
