We Tried to Capture Rich but Hard-To-Measure & Unanticipated Changes. And This Is What We Learnt

Three months ago, we started a regional initiative to capture and describe some of the effects of three youth employment projects of the Swiss Agency for Development and Cooperation (SDC). The RisiAlbania, EYE Kosovo and MarketMakers Bosnia & Herzegovina projects are implemented by Helvetas and its consortium partners in Eastern Europe (Partners Albania, Management Development Associates in Kosovo and KOLEKTIV in Bosnia & Herzegovina).  

The objective was twofold: to learn about and improve the projects’ contributions to more and better changes in the selected sectors, and to document and share the learning with other stakeholders. In order to keep the study manageable, we focused on the Information and Communications Technology (ICT) and Business Process Outsourcing (BPO) sectors.

Assessing and knowing about changes (including impacts) in development cooperation remains complex and nuanced. We therefore recognize the importance of understanding concretely what kind of changes happen.

A simple but profound question is: how do we get to know about changes in systems that happen but are not identified or expected by projects?

Tracing changes through Outcome Harvesting

We used Outcome Harvesting, a method that enables evaluators or projects to identify, formulate, verify, and make sense of outcomes of their initiatives. The method was inspired by the definition and is grounded in the same principles and practice of an outcome in Outcome Mapping: a change in the behavior, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution (that are within the sphere of your influence).  

The method doesn’t measure progress towards predetermined outcomes or objectives, but rather collects evidence of what has been achieved, and works backward to determine whether and how the projects contributed to the change.

Outcomes were harvested in a two-day workshop with each project in October 2019. Besides the project teams, representatives of the consortium partners, co-facilitators and partly also sector partners participated in the workshops. A small number of key informant interviews to verify the outcomes harvested by the project team and to potentially find additional outcomes.

Based on the outcomes, the teams and consultant strung together a coherent narrative, that gives an easily accessible but comprehensive picture of what the respective project has achieved and how it has contributed to these changes.

A second two-day workshop was held with each team in November to complete the descriptions of the Outcomes, refine some of the categories to tag outcomes, and discuss the criteria to assess whether a change is systemic or not. Furthermore, the work of the teams during the second workshop led to a better understanding on how to rate the projects’ contribution in a more objective way.

For the substantiation of the outcomes, we didn’t choose to validate a selected group of individual outcomes and contribution statements by external sources, which is usually the case in outcome harvesting. Instead, we preferred to work with a selected group of experts in each country providing feedback on the credibility of the entire narrative that was developed for each project. Our idea was to minimize the likelihood that the team could in the worst case intentionally overstate the effect of the project but might in any case introduce an unintended bias in favor of the project.

As is common in Outcome Harvesting method, the prime informants of the study were the project teams themselves. This is one of the major limitations of our attempt. From an evaluation perspective, this can be problematic.  

Changes occur at different levels and in different time frames

Changes in the form of “impacts” are often defined as the subsequent effects on the target population or group, be it in the form of income, job, resilience or empowerment. For this to happen, changes at different levels need to occur.

However, what we learnt (and confirmed) from capturing and describing changes in the three youth employment projects is that changes don’t happen in a linear way. This is the one of the downsides of existing “impact logic” thinking in most projects, including those that apply a market systems development approach.

As Michael Kleinman forcefully argues, “our logframes are fictions – necessary fictions, but fictions nonetheless.” They ignore that development cooperation is becoming more complex not because of its size, cost and duration. It’s mainly because the systems in which development work takes place are complex adaptive systems.

Outcome Harvesting allowed the teams to collect and organize all changes they could observe, whether they were part of their plan or unexpected. Like this, the teams found that they have generated several unexpected (positive) changes. These changes had not been identified/recognized and documented before. One key lesson from this is the need for the readiness of staff to embrace learning and the requirement for skills and an inquisitive nature to detect patterns and understand power and influence.  

All the three youth employment projects are roughly half-way through their second phases. The focus of the study was on the effects of interventions from this second phase. Yet some changes that have only become visible now were contributed to by interventions from the first phase. An example is the job portal (duapune.com) in Albania which upgraded the features of its website (like filtering of candidates) and introduced payable online recruitment services. It took time for the services to become profitable and become financially sustainable. At the same time, some outcomes that are targeted for the second phase might not yet be visible as they take more time to materialize.

The way forward

After consolidating all outcomes and further revising the outcome narrative, we will organize a sense-making workshop, in which all three teams can work with the results of the study and discuss how these results can improve their understanding of what works and what does not in the project context.

Topics that we’ll touch upon are how the actual outcomes differ from the original plan, how a counterfactual narrative could look like, what the effect of the outcomes on gender and social inclusion is and how we can better understand the systemic effects of the projects as well as their contributions to the effects. We’ll also look into the future and try to understand what this all means for the future strategies of the projects.

An interesting discussion will be around the question of whether outcome harvesting can be integrated as a complementary method into the monitoring and results measurement efforts of the individual projects.

Additional sources

• Fresh Insights into Impact & Learning from Failures
• Learning and Adaptation for Better Development Outcomes
• Three Considerations when Measuring “Success” in Development Cooperation: A Conversation with Marcus Jenal
• How Chasing Fixed and Large Outreach Targets Hinders Scalable and Sustainable Impacts

Cover picture: Helvetas