3M Participatory Evaluation: Introducing the Methodology

The Global Partnership seeks to improve children and youth in peacebuilding (CYP) practices and impact and strengthen the evidence base supporting CYP. As one step toward this end, the Global Partnership undertook a multiagency, multicountry, multi-donor (3M) evaluation.

1. Map who is doing what and where to support CYP.

2. Nurture durable partnerships increasing CYP quantity, quality, and impact.

3. With children and youth, assess the quality and impact of child and youth participation in peacebuilding and variables influencing CYP impact.

4. Build the capacity of children and youth to meaningfully participate in CYP evaluations.

5. Present key findings and recommendations to stakeholders to help increase the quantity, quality, and impact of CYP work.

3.1 Implementation Structure and Evaluation Methodology Development

The evaluation methodology supported a participatory evaluation process involving children, youth and adults as evaluators in Local Evaluation Teams (LETs). The evaluation was primarily qualitative. A multi-method approach was applied, including focus group discussions (FGDs) using participatory evaluation tools with different age groups, online mapping, interviews, drawing, stories, and anal­ysis of available secondary data. In particular, visual participatory evaluation tools including a timeline, a before and after body map, and other tools were adapted (Save the Children et al. 2014) and applied to help “capture complex social change processes and illuminate interactions between interventions and the context” (OECD 2012, p. 32). Visual participatory tools used during FGDs offered increased opportunities for participants of different ages, especially younger participants, to interact and share their views, experiences, and feelings concerning child and youth participation in peacebuilding (O’Kane 2008; Kitzenger 1994).

In each of the three countries, a Country Steering Team supported the initiation of the evaluation process in their respective countries, including hiring a fulltime country evaluator, considering the necessity of hiring a partnership coordinator, identifying 3 to 4 regions within each country where Local Evaluation Teams

(LETs) were to be formed, identifying LET coordinators, and organizing a 4-day capacity-building workshop to launch the process. Eleven Local Evaluation Teams were formed across the three countries, one in each of the following regions:

• Four regions of Colombia: Bogota-Soacha, Cali, Manizela-Armenia, and Montes de Maria

• Three districts in DRC: Bukavo, Goma, and Kitchanga

• Four districts in Nepal: Doti, Mahottari, Nawalparasi, and Rolpa

An evaluation protocol, ethical guidelines, participatory evaluation tools, forms, information leaflets, and other documents were developed by the global lead evaluators to support the in-country processes. During the evaluation all partici­pants were required to sign informed consent forms. In addition, informed consent for the participation of children under 18 years was also gained from their parents or guardians. All evaluators were required to sign the evaluators’ Code of Conduct and Behavior Protocol, which included adherence to child safeguarding policies. Fur­thermore, global lead evaluators supported facilitation of the country’s capacity­building workshop and participated in regular Skype calls with the country evalu­ator, to mentor and support the evaluation process, paying particular attention to quality and ethical evaluation practice (see McGill et al.

3.2 Evaluation Limitations

The evaluators recognize a number of limitations to the methodology including:

• Most initiatives evaluated lacked baseline data on CYP quality and impact. Furthermore, due to the complexity of undertaking evaluations of multiple initiatives to examine their cumulative impact, it was not feasible to gather data from control groups. In the absence of baseline data and control groups, the importance of triangulating data from different sources and evaluation methods was crucial to ensure valid and robust findings.

• Limited conflict analysis and examination of CYP’s theory of change due to time constraints and choice of priority evaluation tools. To respond to this, a conflict analysis problem tree tool was used by LET members at the initial capacity­building workshop in Colombia and Nepal and at the Analysis Workshop in DRC.

• A user updated online platform was intended to broadly map and analyze the location, type, quantity, and quality of CYP initiatives across each country. As described below, a web platform was developed for such purposes, but technical challenges led to delays that prevented its effective use during this evaluation (see www.GPCYP.com/map).

• Time and financial constraints greatly limited the amount of data collected, the number of initiatives evaluated, the number of countries in which the evaluation was conducted, and more quantitative data analysis using research software.

3.3 Participants Defined

Different cultures and organizations define groups of young people differently. In this evaluation child and youth peacebuilding (CYP) generally refers to peacebuilding efforts of any person younger than 30. In this evaluation, participant data was organized in the following four groupings, and FGDs were organized separately with these groups (Table 1).

Efforts were made to include an equal number of boys and girls in FGDs, and efforts were made to reach children and youth from the most marginalized back­grounds, including children and youth from ethnic or social minorities.

3.4 Overview of 3M Evaluation Tools

Online mapping. In addition to using the participatory evaluation tools, the Global Partnership collaborated with other agencies to expand an existing web platform to allow users to add information on who is doing what and where to support children and youth as peacebuilders. The platform allows users to record information on different types of CYP work, when and where the work was conducted, and the number of young people from different age groups involved. It also allows importing of peace and violence indicators. However, development delays prevented the use of the platform until the end of our evaluation process. See more and register CYP initiatives here www.GPCY.com/map (Table 3).

Focus group discussions (FGDs). Several evaluation tools were designed for focus group discussion (FGD) facilitated among a group of 4-12 participants involved with a particular CYP initiative. The timeline and body map were two participatory evaluation tools that were used with separate age groups within each CYP initiative evaluated. The children and youth in context and pots and stones

Table 1 Definitions of Evaluation Participant Groupings

Table 2 Number of participants who participated in FGD or interview by gender, age, and location (excluding draw and write). Participants are only counted once in this table though some participants participated in multiple evaluation activities

11 Evaluating Children and Youth Participation in Peacebuilding in Colombia...

Table 3 Number of FGDs completed by tool and participant group

FGDs were primarily intended for use with case studies that focused on a more limited number of CYP initiatives to gather more data and evidence concerning the quality and impact of CYP. A brief description of the tools is provided.

Essential Tool: The Timeline FGD explored the history, developments, suc­cesses, and challenges of a CYP initiative over time. Participants identified success factors and shared their ideas to improve the impact of their peacebuilding initiatives.

Essential Tool: The Body Map FGD used body parts to explore changes in child and youth experiences before and after their participation in a particular peacebuilding initiative. It explored positive, negative, and unexpected outcomes, changes, and impact of their participation.

Children and Youth in Context FGD used a diagram of children and youth in the context of their families, schools, community/district, and national contexts to help identify their peacebuilding impact. This tool helped (1) identify objectives of CYP initiatives at different levels, (2) analyze positive or negative and expected or unexpected impact at different levels, and (3) identify other data sources to verify CYP impact.

The Pots and Stones on the Quality of Participation in Peacebuilding FGD was used to discuss and score eight principles relating to CYP quality and to share ideas to improve quality.

Draw and Write Poems and Stories on “Peace and Violence: Choices and Change.” This was an optional individual activity that allowed participants to share more about their CYP experiences and feelings, choices they faced to engage in peace or violence, or changes arising from peacebuilding or violent experiences.

Case studies. They provided more detailed information and analysis of the quality and impact of key CYP initiatives identified through the initial evaluation process. In each country 3-4 CYP initiatives which were considered “most effec­tive” in building peace were selected. Additionally, the Reflecting Peace Practice Matrix (CDA 2009) was also applied. This matrix was developed from the Reflecting on Peace Practice Project (RPP), which is an experience-based learning process that involves agencies whose programs attempt to prevent or mitigate violent conflict. Its purpose is to analyze experience at the individual program level across a broad range of agencies and contexts. Its goal is to improve the effectiveness of international peacebuilding efforts. RPP found that all programs work at two basic levels, the individual/personal level and/or the sociopolitical level. In this evaluation, one of the RPP matrices was applied to better understand conflict issues addressed, the vision and the extent to which (a) individual and sociopolitical changes were supported through the initiative and (b) whether more people and key people were involved.

Data analysis. Workshops in each country brought together LET members, country partnership steering team members, the country evaluator, and lead eval­uators. Children and youth were supported and encouraged to play active roles as facilitators, presenters, analysts, and advocates in these workshops. Each data analysis workshop enabled a review of data transcriptions from the FGDs, identi­fication of emerging themes, and development of recommendations. Furthermore, the global and country evaluators came together for a 4 day data analysis workshop in May 2015 to compare and contrast findings and themes across the three countries.

3.5 Lessons Learned on Involving Children and Youth in a Participatory Evaluation

In this section, the factors which influenced meaningful participation of children and youth in the evaluation process are described.

Capacity building and mentoring of children and youth as evaluators. The evaluation process consistently focused on capacity building and mentoring of youth and children as evaluators while also applying ethical guidelines ensuring their participation was safe, relevant, and appropriate to their developing capacity and time availability as volunteers. The only paid positions were the Local Eval­uation Team (LET) coordinators (mostly undertaken by youth), the country eval­uators, partnership coordinators, and the lead global evaluators. The initial 4-day capacity-building workshop provided a crucial opportunity to enhance the skills, knowledge, and confidence of children, youth and adult LET members, partnership coordinators, as well as the country evaluators. Furthermore, the country evalua­tor's field visits, phone calls, and emails provided opportunities to mentor and support the LET members in planning, data collection, data storage, documentation, and analysis.

While key efforts were made to apply the evaluation tools in similar ways to ensure comparability of data, it was also crucial for the LET members to internalize and gain ownership of the evaluation tools and methods. Good translation of the tools into the national language was an enabling factor when done well and a hindering factor where it was not done sufficiently well. Opportunities to pretest the tools were helpful in Nepal, and the use of the tools in their own peacebuilding initiatives before applying them in other initiatives also increased LET members' confidence and internalization of the tools in each country. “The dynamic of the Local Evaluation Team allowed us to feel confident in front of other initiatives” highlighted a LET member from Colombia.

Providing opportunities for children and youth to be part of planning, data collection, data analysis, and development of key messages and recommendations increased child and youth engagement, feelings of respect, and their sense of ownership of the tools and findings. Child and youth LET members expressed how they developed and strengthened their facilitation, communication, leadership, and evaluation skills and gained knowledge to strengthen their peacebuilding initiatives.

Access to budget, materials, and equipment. Budgetary and material support to the LETs was crucial for implementing planned evaluation activities. In Colom­bia and Nepal, the LET coordinators were paid a monthly wage in line with their additional responsibilities for coordination, support to LET members, transcription, and data storage. These paid positions significantly increased the efficiency and effectiveness of the LETs. In contrast miscommunication in the DRC context resulted in a lack of payment to LET coordinators which contributed to reduced motivation and delays in data collection and transcription. In early March, the DRC team employed a database manager to transcribe, verify previously transcribed data, and translate documents. Other LET members from all three counties were volunteers and were provided mobile phone credit, transport, and food allowances. Each LET was also provided with sufficient material such as flipchart, pens, and colored card to undertake the evaluation activities, and a camera was either purchased or borrowed for the duration of the evaluation. Refreshment and snacks were also provided to participants during activities.

Ethics and risks. Open channels for communication to address ethical concerns and risks and to ensure voluntary and informed consent, child safeguarding, and personal security of evaluation team members were crucial. In the Colombian context LET members recognized that there were some inherent risks for youth, children, and their accompanying adults to travel to new areas to collect data from CYP initiatives. To minimize risks, LET members always traveled to new areas in small groups, and they ensured that a member from that specific peacebuilding initiative and community was actively involved. In both the DRC and Nepal, LETs had to adapt their action plans to postpone planned activities due to political strikes and insecurity during the month of January 2015.

Complex ethical dilemmas regarding payment or nonpayment of LET members were carefully considered. Paid positions of LET coordinators were taken on by experienced youth, while other LET members who were volunteers were provided with phone, transport, and food allowances, as well as certificates recognizing the skills and knowledge they gained as evaluators. The decisions were informed by the consideration of existing local practices, national laws regarding child labor, and the principles of do no harm and best interests of the child. Sincere efforts were made to support planning with and by child and youth evaluators to ensure work plans that were realistic considering their roles as volunteers while also considering their existing responsibilities to study, support their households, and enjoy leisure time.

Geography, diversity, and travel. Efforts were made in each country to establish LETs in diverse urban, rural, and remote locations. Illustrating their commitment to inclusiveness, some LET members in Colombia walked for several hours and used small boats to reach some of the CYP initiatives. Similar efforts were made by LET members in Nepal and the DRC. However, time constraints also resulted in fewer FGDs being organized in remote locations. If more budget and time had been available, the Country Partnership Steering Team members (and District Advisory Committee members in Nepal) were interested to involve more CYP initiatives across a wider number of districts and regions of each country. “The model of the program is unique, so this should be continued in the long term and we would like to request to reach more child and youth initiatives” emphasized a district official in Nepal.

Selection and motivation of children and youth. Selection of experienced child and youth peacebuilders who were interested in volunteering time to the 3M evaluation process was a critical factor enhancing LET effectiveness. Having little time to select LET members was a hindering factor, and in the DRC and Nepal, there were some changes in LET composition after national capacity-building workshops. Child and youth LET members in Colombia and Nepal were primarily motivated by opportunities that the evaluation provided to build their capacity and to strengthen their peacebuilding initiatives. In all three countries, the evaluation enabled interaction and learning among children and youth who were engaged in other peacebuilding initiatives in different parts of the country.

Time availability. Time constraints were one of the most significant factors negatively affecting the meaningful participation of children and youth in the evaluation process. As recognized at the outset, children and youth are busy with school, household, and other responsibilities and have limited time availability for organizing or being part of data collection activities. It was crucial to make maximum use of nonschool days - weekends and school holidays - for organizing evaluation activities. Another time constraint concerned the length of time taken to use each of the evaluation tools. Although it was planned that one tool would take 90 min, it often took up to two hours to ensure clear introductions, icebreakers, and completion of all the questions within the tool. As a result a few participants left before the end of the FGD.

Coordination and planning among the country partnership members increased the success of the participatory evaluation process. Furthermore, collaborative participation and commitment among youth organizations, a national NGO, and INGOs in the Nepal partnership contributed to the effectiveness and success of the process in Nepal. Two youth organizations who were members of the steering team in Nepal commented how “Youth and local level organisations were able to sit together and work together with national and international NGOs in the Nepal Partnership and 3M evaluation. This type of collaboration and partnership increases peacebuilding by children and youth.”

4