Covid-19’s global march has pushed development work to respond swiftly and nimbly to new and longstanding problems. When the crisis finally begins to ebb, we will inevitably ask ourselves: “What worked? Why? Why not?” In this, our third piece in our series on learning and development work, we argue that, before we can answer such questions, it is necessary to have answered a prior one: How do we express the desired impact of our interventions through hypotheses?  For this, we analyze two potentially useful tools:  Experience Maps and Causal Diagrams.

Despite the ubiquity of impact assessments in the development world, the traditional metrics we use to evaluate our interventions sometimes contain a fatal defect: they do not measure the success or failure of an intervention in terms of an expected effect.

This could be a matter of funding. When funds are not designated for developing a framework, defining indicators, and collecting, processing and analyzing data, it is reasonable to expect that organizations will select indicators that can be tracked and reported without additional investment.  Therefore, monitoring and assessment indicators often merely quantify and report the intervention process itself (ie, x number of devices were distributed, y number of workshops took place, etc.) using data that will be collected as a matter of course. However, we do not always evaluate the impact the intervention has on development, or even monitor indicators to evaluate whether there is a causal process that in theory could lead to the desired impact.

Alternatively, when a funder requests an impact assessment, we generally do have funds available for the task. However, this “external” requirement—often carried out by external consultants—is unlikely to be adopted and internalized by organizations as a tool for learning and innovation. In the worst of cases, the impact assessment can feel more like a box to check than an important learning resource.

This is a problem for sustainable development practitioners because it implies carrying out actions without an explicit learning strategy. Acting without applying lessons-learned to future action is like navigating without a compass. With this in mind, and in order to foster a focus on learning, the UNDP has in recent years incorporated new assessment devices, such as the Theory of Change methodology.

How do we articulate our expected impact?

There are numerous tools for generating hypotheses. The “Theory of Change” methodology should be carried out with the participation of all the stakeholders of proposed intervention: first, in an analysis of cause and effect; second, in and ends/means analysis; and, finally, in the selection alternatives for solving problems in a manner that is coherent with participants’ experiences. Nevertheless, applying this method requires funding for the design phase of a project, which is not always available. This dilemma can give rise to “theories of change” that are created post hoc, privilege funders’ key concepts and priorities, or are too complex to aid the design of parsimonious and valid assessments of an intervention’s expected effect. 

In order to complement the Theory of Change and to generate hypotheses that are clear, concise and testable—even when time and funding for project design is limited—the #AccLabPy proposes two tools: the first is the Experience Map. This tool helps us identify the key elements and milestones of an intervention that has already taken place. The process of radically simplifying the moving parts of an intervention provides us with the raw material for the second tool: “If/Then” statement and Causal Diagrams.

The causal diagram harnesses one of the implicit strengths of applied development work.  Carrying out an intervention, observing its trajectory and evaluating its results gives us a privileged view into the causal mechanisms operating in a given social field. Transforming the messy reality of an intervention into a causal diagram exploits this wealth of observations and lessons-learned, and forces us to make them explicit.

The process is simple. It involves identifying the dependent and independent variables of the social process in question—what is the desired result and what is the treatment proposed by the intervention—and making an educated guess about the sequence of events or effects (the intermediate variables), that link one to the other. The model also allows for interactive or conditioning variables—factors that amplify the effect of a given variable or its antecedents.

Learning to learn better

The #AccLabPy tested the utility of these devices in two internal workshops as well as two that were carried out with non-UNDP participants. In the first workshop, participants remarked that creating very clear, simple sequences of causally-linked events is not only useful for developing hypotheses, it is also an important storytelling tool that aids efforts to communicate an intervention experience. A causal chain encapsulates the principal milestones as well as the narrative and causal flow of any intervention.

In the second workshop, participants were able to directly compare theory of change methodologies with causal diagrams as tools for explaining the impact of social interventions. Various participants commented that, when using the causal diagrams, they were able to produce a more effective description of the expected impact because the nature of the tools forced them to pare the description down to its fundamental components.

We were also able to experiment with these tools in research workshops with participants from two different social innovation contests: the 2020 edition of the Small Grants Programme (Programa de Pequeñas Donaciones, PPD), which supports woman-led, rural enterprises; and the 2020 edition of the I-Lab, titled “Seeds of Wellbeing” (Semillas de Bienestar), which sought to promote innovative solutions in the area of food security.

In both workshops, the participants designed Experience Maps, Causal Diagrams and an even more simplified version of hypothesis formulation using If/Then models. The objective of these exercises was to clarify and refine the hypotheses related to their contest proposals.

In the case of the PPD, the participants found themselves agreeing with their counterparts in the UNDP’s internal workshops in that the causal diagrams helped them develop the storytelling aspects of their proposals. After the workshop, due to the positive reaction to the tools, the program decided to include the hypothesis development tools in their official application. In the following weeks, the PPD projects will receive a followup assessment meant to deepen their understanding of the lessons generated using the tools.

I-Lab participants noted that the causal diagrams occupy a sweet-spot with regard to the simplification of reality. The extreme simplification of the IF/THEN exercise allows for the unambiguous expression of the proposed treatment and expected impact. However, the causal diagram, in addition to generating more clarity, helped participants rethink their projects’ value propositions.

The final leap--from hypotheses to experimentation

As noted by the participants of the various workshops organized by the #AccLabPy, hypotheses are radical simplifications of reality. Causal Diagrams occupy a sort of Goldilocks zone between complexity and simplicity, providing just the right amount of detail needed to focus an intervention or value proposition. The causal diagrams help predict results and to discard failed predictions. As a result, they are useful guides for the design, implementation, and assessment of our interventions.

In the following (and final) blog of the series, we will delve into this last point. We will focus on how to make the transition from a well designed hypothesis to an impact/process assessment protocol using, for example, paired indicators and monitoring sources. If you have any questions, or wish to share other tools and methods for the development of impact hypotheses, leave a comment and we will make sure to link to these resources in  #AccLabPy’s final entry into this inaugural blog series on learning and development work.

Lea este blog en español aquí.