When Theory Meets Practice in Monitoring and Evaluation: A Personal Story from Albania
We first met a year and a half ago. Admir was a candidate for the Monitoring and Evaluation (M&E) Manager position of a youth employment project, RisiAlbania. Zenebe was a member of the recruitment panel, participating from the Head Office of Helvetas Swiss Intercooperation.
Admir got the job. Yet this was just the start…Development practitioners work in complex, rapidly changing environments. Practical and right-sized M&E systems are therefore important to enhance the quality of projects, for example regarding whether an intervention is contributing to a project’s goal.
In this blog post, we look back at our expectations – both from Admir and the project – and review what has happened since a year and a half ago in designing, implementing and improving the M&E system of RisiAlbania as simply good management.
Starting off on the right foot
The job description outlining the roles of the M&E Manager was clear. Admir was confident that he would fulfil his roles and support the project in thinking through what it plans to do and what it seeks to achieve. There was clarity from the start about asking how and to what extent the project has achieved an impact on young people and their employment opportunities. For this, the project needs to rigorously assess the effectiveness of its interventions and continuously adapt its strategies.
The need for a full-time M&E Manager had been pointed out by the external review of the project, which also stressed the importance of actual participation of other members of the project. Specifically, there was a need for the project for up-to-date information that would allow adapting its strategy in reasonable time. The project operates with small, technical teams and at the time M&E budget was relatively limited. During the first phase, M&E activities absorbed approximately 1% of the total budget. Using his experience in M&E, Admir was eager to find innovative solutions that adapt the M&E system to fit the size and capacities of the project.
Recruiting a full-time M&E Manager would, it was expected, address the challenge of devising a means of utilising time and financial resources effectively. The project initially underestimated the need for a full-time Manager, as well as the development of broad sector strategies that further stretched relevant resources that could have been put towards a practical M&E system.
What has improved?
We met once again in December 2017 in Tirana, Albania. We started doing a quick inventory of what went well.
The M&E system of the project has improved a lot. As a case in point results chains – a visual tool to show what the project is doing, and why – are really being used as a planning tool. Gaps in information that existed in the past are addressed through timely gathering and analysis of relevant data and by better referring back to the results chains as blueprints for work plans. This is highly important as decisions taken are informed and based on evidence.
Here is an example of the agribusiness intervention of the project. The service market did not accept trained people as freelancers because the process of selection for the training of such freelances was not done properly. The M&E system did not contribute to planning and monitoring partners of the project from the start, not just at the end through partner interviews. The project did not have enough human resources to do this with all the partners; the project could have used subcontractors to help in implementation. Thanks to the improved M&E system, the project is able to make evidence-based decisions to change the intervention and to share this knowledge externally with business partners or other projects through meetings, conferences and training. The last one, where Admir had the chance to be part of the trainers, was done in December 2017 in Albania with the presence of other development projects implemented in the Western Balkans.
In addition, even though there have been staff changes, new members of the project clearly understand and they are committed to play in most aspects of monitoring – from developing results chains to data collection. Admir, together with the Project Management, has been leading in establishing an M&E culture of honest enquiry among staff to find out what works and what does not, and why. This goes together with investing sufficiently and continuously supporting staff capacity through training (or coaching) and regular discussions and reflections, as well as strengthening collaboration/knowledge sharing and learning with other projects.
The M&E system is also supporting the project to articulate business models – as part of an innovation and/or adaptation/solution, which either foster a brand-new idea or rethink the existing practices that will spur the business growth. The project staff has recognized that the purpose of the M&E system is not just to produce more data/information; it essentially informs action by identifying business opportunities through feedback mechanisms. Such articulation and reflection was missing for some of the interventions of the project in the past.
Reality still continues to bite…
We do not claim that the whole process has been seamless. For example, the most difficult task has been the attribution challenge – that is, establishing the causal links between the project intervention and impacts achieved. First, data has been scarce and less reliable. Second, identifying and using feasible methods of collecting data is still a challenge. The project continues to learn and improve this, intervention by intervention, together with other project staff by testing different tools such as before-and-after-comparison-with-opinion, or comparison groups’ method, and by tailoring the tools to the needs of the project. Admir has been scheduling personalised meetings and finding out with the team the right-sized solution for each intervention.
Like many other projects, representing or visualizing data to different audiences such as donors and local level stakeholders also continues to be a challenge. Development practitioners are under increasing pressure to demonstrate that their work is relevant and it contributes to sustainably addressing poverty and unemployment. For this, they report and tell stories to a range of stakeholders. Clear thinking is the basis of telling good stories. However, development practitioners, like us, often get trapped in “clutter” — the state of having unimportant details and less relevant jargons in the storylines. Generating reliable and relevant data through the M&E system and translating the data into good stories continues to be an important agenda for the project.
Additional sources
Athors
