Going Beyond Numbers
Katrin Ochsenbein from SDC Kosovo shares her thoughts on impact, failure and learning.
There are as many definitions of impact as there are development practitioners. Projects usually report their progress around quantitative indicators, such as number of job placements achieved in the case of youth unemployment. But impact goes beyond numbers, and achieving it usually requires trying out innovative approaches that sometimes fail.
Niklaus Waldvogel discusses the many dimensions of impact and strategies to cope with and learn from failure with Kosovo-based Katrin Ochsenbein, the Regional Advisor Employment and Income for the Western Balkans at the Swiss Agency for Development and Cooperation (SDC).
Niklaus Waldvogel: What is SDC’s understanding of impact, and how do you define positive impact?
Katrin Ochsenbein: Following the Market Systems Development (MSD) logic and SDC’s understanding, a project usually only has direct control over the outputs of its interventions. Systemic changes following these outputs lead to what we call 'outcomes' and they should contribute to the desired impact. This impact is usually formulated at a very high level, for example poverty reduction. A 'good' impact can be described as sustainable improvements for the target group compared to a baseline.
NW: These systemic changes you mention – crucial to the MSD approach – are a difficult concept to grasp. How does SDC assess the potential for systemic change before a mandate is given out?
KO: Talking to different people, I realize that systemic change often gets confused with systems change. In my view, a change in the overall system is the consequence of many systemic changes. To properly design projects with a high likelihood of achieving systemic change, SDC uses theories of change where we lay out the rationale and the assumptions behind an expected contribution towards poverty reduction for a target group. This tool is central for our results-based management because it forces us to verify hypotheses not only at planning stage but also throughout the implementation of a project.
NW: In your opinion, can we measure impact only through quantitative indicators linked to goal-level targets or also through qualitative changes stimulated by projects? And do you think our current measurement methods allow projects to capture unexpected outcomes?
KO: In Private Sector Development (PSD) projects, standard measurements such as Full-Time Equivalents (FTE) are can be used as quantitative indicators. But it's crucial to distinguish measurement tools (e.g. indicators) from impact, as impact can be measured quantitatively and/or qualitatively. In MSD, the most important results are delivered by other actors in the system, with little direct control of the projects. Examples are entrepreneurs changing their behavior or officials adapting regulations. To understand and prove the project’s contribution or attribution and be able to claim these results is a difficult task and requires a robust monitoring system. The same is true for example for governance projects.
When appropriate, unexpected outcomes should be captured and incorporated into management decisions. Projects should hold regular reviews in their life cycle and discuss and potentially use unexpected results to adapt the interventions.
Impact and failure in practice
NW: Sharing good practices of projects is very helpful as a reference for future project design. Some projects are highly successful while others struggle in generating impacts, due to both internal and external factors. Can you mention examples of good impact in development cooperation? What are key elements of their success?
KO: An impactful activity comprisis the Job Clubs that have been promoted in Bosnia and Herzegovina as part of the reform of Employment Services. They represent a way to actively involve young people in the job searching process and somehow reactivate them out of long-term employment. The World Bank is currently scaling up the reform in all public employment offices in the country. Another example is the Youth Guarantee Scheme in North Macedonia, which is implemented by the EU. It generated very good results in terms of outreach and raised the interest of the government that wants to promote the scheme throughout the country.
These examples show that projects applying a systems approach can lead to a very good impact. I think the reason for success in the two examples is the creation of relevant links and the subsequent leverage by partners and the government. They all started with a pilot and lead to a systemic change through scaling up. All the stakeholders were convinced of the benefits and there were clear and shared visions of the objectives.
NW: Why do some projects fail to achieve impact? Is failure considered normal for SDC, considering the complex context development projects are working in?
KO: Of course, failure is normal. As you say, we are dealing with a lot of uncertainty in development cooperation. There are many external factors that are out of the direct control of the projects. It's possible that the landscape of potential partners changes midway through a project, including institutional partners. There can also be implementation failures: a project can have insufficient targeting or hire the wrong personnel. To tackle these, risk management is crucial - by that I mean a constant reassessment of the underlying assumptions of the theory of change throughout project implementation.
NW: One of the key challenges we face as implementers is the tension between short-term and long-term results. Can such a tension be managed or reconciled?
KO: In my opinion, there isn’t such a strong tension between the two. Yes, projects need to show results within their time-frame and simultaneously work towards systemic changes that may take more time to happen. But quick wins or short-term results can be used to build towards achieving longer-term outcomes. It all comes down to choosing the right intervention strategies to achieve results in the intended time-frame. However, I acknowledge that for intervention managers, who must make sure that short-term results are reached, it isn't always easy to keep in mind why they need to design systemic interventions.
On our side, SDC offices in the Balkan region sometimes struggle to engage in discussions at the outcome instead of the activity level with implementers of MSD projects and to understand, where they stand regarding the implementation process. It would be helpful to have a bigger focus on outcomes and discuss how they're going to be reached. I think this is an area where the communication between us and the projects could improve.
Managing and communicating failure
NW: The development sector struggles to communicate and learn from failures. This is partly due to a reliance on donations and politically driven aid budgets, which means there is a tendency to focus on success stories and “sweep failures under the carpet”. Learning is taken as an add-on and not as central element of development work. Even if learning happens, most projects’ knowledge or experience is tacit and only available to a narrow circle of colleagues. Nevertheless, experimenting and making mistakes are integral parts of our complex work. How can projects become better in achieving impact through learning from failure?
KO: It is essential that failures and difficult experiences are communicated, among peers but also towards the donor. Obviously, the wording of the messages must be adapted according to the counterpart of the discussion. It is also important to not just say that something hasn’t worked, but to explain why and how the situation can be improved, what is the learning effect. For SDC it is not a problem when not everything goes as planned, but we want to know why. Moments like strategic reviews are perfect for these discussions. And of course, it will be difficult for a donor if a project just reports failure after failure and doesn’t show that it is able to learn and adapt.
NW: Do you think that something should change in future mandates or in a typical project setup to leave more room for experimentation and learning from failures?
KO: The opportunity funds that we have throughout the Western Balkans region have been a helpful instrument in first phases of projects that apply market systems development approach, especially to test partnerships, innovations and to pilot activities. Backstopping can help transferring learnings from other settings. Foreseeing learning activities transparently within communication and MRM budget is also helpful. Finally, it is also about maintaining a general culture of learning within the projects, following adaptive management practices. Learning can’t be done by just one person, it needs time and space. This should be reflected in the job descriptions of project staff as a continuous task.
NW: How can donor agencies contribute towards more openness in the development sector regarding failures, and help developing a culture of learning from them?
KO: Donor agencies such as SDC manage public funds and must follow the principles of accountability, transparency and an efficient use of resources. Their incentives to actively communicate failures is therefore rather low. And yet, I have the impression that Switzerland is more open than others, for example the report to parliament includes examples of projects that did not turn out as intended. Nowadays we talk about aid effectiveness, Sustainable Development Goals (SDG’s) and long-term processes, including more adaptable donor agencies that don’t just stick to the same things all the time.
NW: So, how should a knowledge management and learning system that is integral to project work look like? What kind of investments are needed?
KO: It is useful to embed learning deeply within a project’s monitoring functions, but of course there is a trade-off between scientific rigor and pragmatism. The important question concerns the absorption capacity. There’s so much material already out there, webinars and other products, neither SDC nor project staff can read and follow all of it.
Websites like DCED and BEAM Exchange are very useful as repositories, but in my experience people are still more likely to check out information personally recommended to them. For me as regional SDC advisor, exchanges with colleagues in the region provide excellent opportunities to learn from each other, despite all the differences. I guess, Helvetas realizes the same within its regional working groups in Eastern Europe.
This article appeared in the December 2019 issue of Helvetas Mosaic.
Subscribe to Helvetas Mosaic
Helvetas Mosaic is a quarterly published by Helvetas Eastern European team for our email subscribers and website visitors. Our articles explore new trends and fresh ideas of international development work in Southeast Europe.
