Appendix 4.1: Structured Focus Group Discussions as a tool to analyse complex development programs
Several methods can be applied to make the STEHD framework operational. In this appendix, the Structured Focus Group Discussion or Structured Participatory Group Interview is presented (see Biggeri and Ferrannini, 2014).
This tool can be used to collect information and to assess the level of engagement and contribution of different actors in complex development initiatives on various dimensions of the STEHD framework (i.e. the functionings of a local system or the capabilities of individuals with certain characteristics).Although the participatory feature of this tool leaves open the possibility to collectively discuss in details the critical elements of an LDS, it is also “structured” as the steps are clearly defined (sequence of actions and sequence of questions) and aid comparability. Therefore, SFGDs can complement quantitative data, or in its absence can constitute a valuable qualitative method by itself, for impact evaluations.
This tool includes a wide array of informed actors (community stakeholders, national stakeholders, international stakeholders and even individual beneficiaries), invited to collectively assess the impact of a development program on various territorial achieved functionings or individual capabilities.20 During its elaboration phase, the tool is reasonably flexible and adaptable to the features of development programs and to the stakeholders involved. The training of local experts/facilitators, the standardization of the procedure and the heterogeneous selection of participants21 are essential in order to ensure rigorous and comparable evaluations, especially when control groups are included in the analysis (Biggeri and Ferrannini, 2014).
The selection of participants among informed actors ideally requires nested random sampling within each homogeneous group of stakeholders, which can be accomplished only if there has been a precise mapping of different stakeholders.22 To guarantee the participation of informed stakeholders and beneficiaries, the closest representation of local society and the finest balance of influence and power are important.23
The SFGD is related to a Matrix score, that is, a sequence of questions (rows * columns) with a corresponding score (see later Table A.4.1 for an example).
Here lies the main difference with traditional FGDs, which are usually intended to address one or two specific issues in depth. The score assigned to each question reflects the level of a given functioning/capability and the perception of temporal change (if any) within this particular dimension. The identification of a certain score24 and assessment for each dimension recorded in the table, as previously mentioned, is based on collective discussion and group answers rather than on individual answers, until an agreed evaluation is reached.25 In the table, each cell of the matrix represents a question. As in a questionnaire, these questions run in sequence and must not be ambiguous (see Biggeri and Ferrannini, 2014, for discussion of procedural details including the "impartial spectator”). At the end of the procedure, in order to increase the tool's internal validity, participants can be asked to freely comment on the fidelity of the representation achieved together, with respect to the level of opportunity in their communities in each dimension. In their debriefing, they are also asked to express an opinion regarding the exercise as a whole. The matrix results are then provided to participants26 for their own use and analysis, contributing to community empowerment. Overall, it is possible to derive useful insights on the counterfactual situation in the absence of the program, that is, what would have been the level of functionings of the local system without the development program?As an example, the structure of one SFGD concerning the assessment of the program Desarrollo para la construction de la paz en medio del conflicto implemented by UNDP/ART REDES in the Department of Narino (Colombia) are reported in Table A.4.1.27
This SFGD focused particularly on economic dimensions and involved different participants, drawn from political institutions,28 local universities, civil society organizations and producers' and workers' organizations.29
Participants were initially asked to evaluate the functionings of the LDS today and retrospectively, that is, five years previously (columns 2 and 3).
Then, they collectively assessed the contribution of different local, national and international actors (columns 4 to 10, one corresponding to the UNDP/ART REDES program) in terms of different dimensions. Finally, participants were asked to identify the "perceived counterfactual”, that is, how the LDS would have been able to function today in the absence of the program (column 11).The results that emerged from this SFGD were confirmed by three other SFGDs conducted in Pasto (Narino's main municipality) with different groups of stakeholders. In particular, the role of UNDP/ART REDES as a catalyst in the coordination of multilevel actors and elaboration of strategic development plans through participative processes was repeatedly emphasized.
The same procedure was administered in Morocco and Ecuador (see Agostinucci and Biggeri, 2012a), allowing these countries to avoid high- cost surveys, obtain collective assessments and empower local communities in their capacity to evaluate the programs' implementation and territorial capacities.
Table A.4.1 The matrix for the application of an SFGD in Narino, Colombia
| Dimension | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | |||||||
| Relevance of Dimension | Functioning of the System | Functioning of the System 5 Years Ago | Contribution of Civil Society | Contribution of Farmers and Producers Organisations | Contribution of National Government | Contribution of Regional Government | Contribution of UNDP ART program | Contribution of USAID | Contribution of EU bilateral donors | Perceived Functioning of the System without ART | ||||||||
| (1) Ability to elaborate a strategic and shared vision of local economic development | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (2) Access to economic opportunities for marginal groups | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (3)Joint involvement of public and private actors in local development programs | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (4) Capacity to coordinate the action of international donors (including decentralized cooperation), local actors and national government | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (5) Access to services for SMEs | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (6) Investment for local economic development | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (7) Capacity-building for workers and producers organization | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (8) Capacity-building for other economic actors | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (9) Promotion and empowerment of SMEs | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (10) Institutionalization of economic promotion practices (international cooperation) | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (11) Creation of opportunities of employment in the formal sector | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (12) Upgrade of informal economic activities | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (13) Access to markets (international) | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (14) Information and transparency | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||
| (15) Ability to react to conflicts and criminal economy | (1-10) | (1-10) | (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+,-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+■-); (1-10) | (yes, no); (+,-); (1-10) | (1-10) | |||||||