I love working as a Monitoring and Results Measurement (MRM) specialist. Firstly, because it’s intellectually stimulating and you really contribute to the effectiveness of the project. Secondly – and this is a secret among MRM managers – because you can criticise other people’s work, while few people are in a position to criticise you. When a project manager comes to you with a new project proposal, you can dismiss it as simplistic and illogical. When preliminary results for an intervention come in, you can raise lofty concerns that they are not good enough. Hard-pressed managers, however, seldom have the time or inclination to trudge through realms of data to verify your methodology or criticise your analysis. So MRM specialists face relatively little scrutiny – unless, of course, they have a DCED Standard Audit.
The idea of the audit is simple. The DCED Standard is a framework for measuring results in private sector development, followed by Enterprise Partners and approximately 150 other programmes across the world. The DCED Standard describes seven elements for a good MRM system, each of which is broken down into sub-elements (called ‘control points’) and indicators for successful performance (called ‘compliance criteria’). During an audit, two experienced external consultants review your MRM system against these compliance criteria. They go into great depth – checking results chains to ensure programme logic, reviewing the formulas of your projections, assessing the strength of methodologies and the quality of your analysis. The auditors don’t collect any primary data – they just verify that your data collection plans and methods followed good practice.
Our audit was successful, with a score of 96%. While being audited, I learned four important lessons:
A DCED Audit requires MRM to be mainstreamed across the organisation
As MRM Lead, I had expected to be heavily involved in the audit; sitting in every meeting, answering questions, providing data. When the audit happened, I was surprised to find that I was almost completely ignored. The auditors met with my team, the intervention managers, and senior management, but I was only needed for a brief discussion about the overall aggregation system. At first, this seemed strange, but I realised that the aim of the audit was to understand how well MRM principles are understood across the team – not whether the MRM Lead can answer questions.
You don’t have to be perfect
The auditors picked up on a number of minor mistakes in our system, such as errors in results chain logic, or inaccurate projections. Despite these mistakes, we still received a high score. The auditors do not look for a flawless results measurement system, but one that is ‘good enough’, with no major errors and an acceptably low level of minor errors. As a consequence, a programme that is effectively using the DCED Standard doesn’t need to put huge amounts of effort into preparation for the audit. Documents need to be checked and systematised, but will not need major revisions.
A successful audit increases levels of trust in the MRM system
Like many large programmes, our results are heavily scrutinised and questioned. The DCED Audit allowed us to present results with more confidence, and external reviewers understood that they had already been robustly checked. Our latest Annual Review team praised the MRM system for being ‘comprehensive’, ‘robust’, and ‘committed to transparency and openness’. The audit undoubtedly contributed to this result.
Whether successful or not, the audit helps internal MRM systems
The audit increased the pressure on our teams to improve their MRM, and helped them to understand areas where improvement was needed. For years, I have been telling intervention staff about the importance of close integration with the MRM team, understanding the MRM data, and using results for management. My training and advocacy has had some results – but the possibility of failing an audit was a very effective motivator! For a brief period, all staff were familiar with chapter and verse of the DCED Standard, passionately involved in their results chain development – and some of them even read the MRM Manual.
Moreover, the questions and challenge posed by the auditors highlighted gaps in our systems. Two teams independently decided to hold MRM workshops to address these gaps and promote closer working relationships between the MRM focal and intervention focal. The greater awareness of and interest in MRM issues has still continued, and made a lasting difference to the project.
Written by MRM Lead, Adam Kessler. Itad are responsible for MRM within Enterprise Partners’ project.
This blog was originally published on the DECD website.
