Appendix 6.1: How to evaluate complex and context-specific development programs
The international development arena is characterized by programs that can no longer be easily exemplified by the standard linear relation, input- actions-output-outcomes (Duflo, 2004).
As part of both the increasingly crucial global-local dialectic and the global rethinking of the development paradigm, traditional typologies and classifications of development programs have started to lose ground, with the concomitant emergence and diffusion of the consolidated and interrelated characteristics of complexity and context-dependency.Firstly, the complex architecture of development programs is due to their multidimensional focus and concomitant multiple activities and tools, the prevalence of “soft” activities with strong focus on processes (rather than simply on outcomes) and long-term impacts, and the evolving institutional dynamics at different levels (Pawson, 2005; Stern et al., 2012). In addition, a multiplicity of policymakers and stakeholders sustain, lead and affect overlapping programs with different political agendas, timeframes and modalities, within a multilevel system of dialogue, coordination, learning and governance.
Secondly, the place dependency of development programs are inevitably linked to the varied characteristics of the geographical, economic, social, institutional and cultural context of action, thus yielding longer-term evolutionary trajectories. Moreover, horizontal and vertical integration processes characterizing development programs are strictly related to the peculiarity of the complex network of local and extra-local relations in which each intervention is embedded.
Examples of similar programs, henceforth defined as complex context-based development programs, can be identified both in the international cooperation arena20 and in national strategies,21 as well as among the initiatives implemented by Local Economic Development Agencies (see Chapter 5) or by territorial communities themselves.22
Overall, these characteristics crucially affect program assessments and impact evaluation procedures (Hertting and Vedung, 2012).
In these cases, causality is often less clearly ordered and feedback loops can generate vicious or virtuous circles and synergies within dynamic processes of multi-causality, including those involving intangible aspects. In addition, the process and procedural aspects characterizing these programs appear as important as the outcomes at individual and community level. It is therefore complicated to isolate and disentangle attribution of the (net) impact to a single actor among many (civil society, public, private, local, national, international), as well as to build up a counterfactual at the meso level, given that these programs are strongly context dependent and focus on “soft” activities.The complexity, the time horizon and the multi-faceted elements of development programs not only entail the need to search for new procedures, designs and methods for impact evaluation23 but also call for a new perspective in building up a theory of change. In this regard, the CA can enrich the informational base for multidimensional socio-economic assessments (Alkire, 2002), focusing evaluations on what people are able to do and to be (final outcomes) and have reason to value, including immaterial aspects of their life.24 This perspective would give salience to the individual's and communities' experience, values and participation (Mayoux and Chambers, 2005), taking into account multiple and often alternative causal strands (Stern et al., 2012). The impact of development programs is therefore evaluated in terms of the enhancement of enabling environments for human flourishing, taking into account the elements of the STEHD framework and the main questions for its procedural application.
Therefore, the STEHD framework could represent a relevant reference point to elaborate a tailored theory of change for the impact evaluation of complex and context-based development programs, especially if framed within an evolutionary SHD perspective at the local level (see Figure 2.3 in Chapter 2).
As a result, the success or failure of development programs is conceptualized in terms of achieved functionings and opportunities set enlargements, both at the individual/household and community/territorial level. In other words, a complex evaluation needs to focus on the characteristics and functionings of the local systems where individuals live and interact, in order to capture the complexity and place dependency of programs' mechanisms and outcomes.Within such a perspective the selection of the evaluation domains is considered the result of participatory deliberation processes on what should be included and why. This avoids the pitfall of ‘insisting on one pre-determined canonical list of capabilities, chosen by theorists without any general social discussion or public reasoning' (Sen, 2005, p. 158). This argument reinforces the idea that evaluations should not ignore their own political dimension, recognizing that each process aimed at linking the analysis of reality to decision-making cannot be neutral (Harriss, 2007).
Nonetheless, this approach is not sufficient by itself, and requires appropriate operational procedures and methods to realize comprehensive evaluations (Chiappero-Martinetti and Roche, 2009; Biggeri and Libanora, 2011). The combination of quantitative and qualitative methods is widely debated in the literature (Bardhan, 1989; Bourguignon, 2003; Chambers, 2003; Kanbur, 2003; Bardhan and Ray, 2006; Cosgel, 2006), and this approach directly encourages the use of mixed methods25 in evaluation.
It is important to stress the remarkable potentiality of mixed methods to explaining both outcomes and processes, especially when (i) programs are not decomposable into single linear causal chains; (ii) evaluations are focused on community-level dynamics and institutional change; (iii) stakeholders' participation is considered to have an intrinsic value for evaluations' success; and (iv) evaluations involves immaterial dimensions (e.g. self-esteem, reduction of social stigma) (Rao and Woolcock, 2004; Osmani, 2009; Stern et al., 2012).
To conclude, a sound theoretical approach together with mixed methods are both required (Barahona and Levy, 2003; Stern et al., 2012), and their consistent combined utilization can represent crucial a value added for comprehensive evaluations within an SHD perspective.