On November 4th and 5th I attended the conference ‘State of the Art Realist Methodologies’ in Leeds, hosted by the University of Leeds. Here are some of the highlights for me.
*Disclaimer: This post dives straight into Realist Evaluation concepts – my apologies to the uninitiated.
1. I learned that we are not alone in our struggle to explain Realist Evaluation to others.
When explaining Realist Evaluation (RE) to someone not familiar with it, we often struggle to find a balance in simply describing an approach that aims to find out ‘what works, for whom, in what context’ and launching into a far-too-detailed, and ultimately confusing, explanation of Context-Mechanism-Outcome (CMO) configurations and other such RE concepts. This is not a great surprise, as RE practitioners are good at confusing each other too. I have been working on Realist Evaluations for almost a year now, and I found myself bamboozled and bewildered by some of the concepts and jargon used by others at this conference. However, I learned I need not worry, as the realist community asserts itself as one open to disputing these concepts and terms. But how does this help non-practitioners understand RE? While we will continue to work to generate clear explanations of our own, at the conference it was revealed that the RAMESES II project team are working on their own explanations of RE for laypeople. We eagerly await them!
2. I learned about ‘Layered theory’ – a good fit for a Realist Evaluation of complex market systems interventions.
Gill Westhorp gave a great presentation on ‘layered theory’ in Realist Evaluation, based on her published work. I was already familiar with the concept of ‘ladders of abstraction’ in Realist Evaluation – moving from granular CMOs at the bottom, up to more abstract programme theories; but Gill then introduced the idea of developing programme theories for levels of systems: where a low level system might be an individual, the next level a family, and then a community. In market development we are used to thinking in systems, so I see a nice fit with Gill’s theory – we can develop programme theories for the level of the individual market actor, the level of single market transactions between two actors, and the level of an entire market system. I think we may already do this, but Gill’s ideas can pave a way for us to be more purposeful and systematic in our approach.
3. I learned some new definitions of/ways of conceptualising a ‘Mechanism’…
We spent a large part of Day 1 of the conference discussing ‘Mechanisms’. Some of the main takeaways for me were 1) the possibility to define Mechanisms differently depending on what level of system you are looking at e.g. it looks like the reasoning of an individual at the level of the individual, and can look like a force or something more abstract at the higher levels of systems. This gem is thanks to Gill Westhorp again. The other idea that I like (but think will make life a whole lot more complicated), is the notion that a Mechanism works on a continuum (like a dimmer switch) rather than in a binary fashion (like an on/off switch). This idea came courtesy of Dalkin and Lhussier. It also leaves me with more questions, like, how do you measure the ‘degree’ to which a Mechanism is working rather than its mere presence or absence?
4. …And another definition (for ‘Context’).
Ray Pawson also talked us through the Realist definition of ‘Context’—‘Context does not equal locality!’ When we hear the word ‘Context,’ it’s so easy to think that this means every single thing that surrounds the intervention; but Pawson says this is not so, it is important (and a relief) to focus only on those parts of the context that are relevant to the intervention and how it works.
5. I learned about some of the problems with Evidence-based Policy making / why The Wire really is the best television show ever.
In the session ‘Policy: Lessons from Realist Research’, I learned that Season three of the HBO show The Wire is based on a real-life incident in London in 2000, when Met Police Chief Brian Paddick decriminalised marijuana in Brixton so that his police force could concentrate on bigger, more serious crimes. Despite initial evidence that his plan seemed to yield positive results, Scotland Yard was not impressed and Paddick was hounded out of his job. Art imitates life: in The Wire, Major Colvin’s innovative establishment of a free-zone for drug-dealers in west Baltimore, dubbed ‘Hamsterdam’, seems to yield positive results for the surrounding communities – but (spoiler alert!) he is eventually removed from office in disgrace and the experiment is squashed. Why are policy makers so disinclined to use evidence? The conversation in this session focused on what we can do to see if our work as evaluators is in fact having any effect on policy. One person suggested we can build our own Theories of Change for how we think our work will influence policy-makers, and another said we can conduct our own (realist) self-evaluations to assess how our works may have influenced which policy makers, in what context. With the right approach to sharing our findings – the right CMO configurations—we hope policy makers will use evidence to establish programmes with demonstrated positive effects.
**
In truth, I learned more than five things at the conference, though there are plenty of aspects of Realist Evaluation that still confuse me, or seem too academic. That being said, I came away from the experience with a new confidence to carry forward my work in RE, knowing that it is an approach that is flexible and comes from a community that eschews dogma, and embraces knowledge sharing and debate.
Jessica Rust-Smith, November 2015
