It was early 2008. By this time, I had heard many stories from people in Mali, Cambodia, El Salvador, and Guatemala about how Saving for Change had changed their lives.
From their stories, it was becoming increasingly clear that savings groups made a difference in their ability to save to buy food between planting and the harvest, to be resilient in the face of droughts and unplanned emergencies, and to increase their sense of empowerment.
From the perspective of a government official, international donor organization, or investor, however, it is important not only to hear these stories but to see concrete evidence showing by how much a program has affected a community. Providing answers to this question helps decision makers understand the significance of a program such as Saving for Change, and with good results helps to continue support and funding for such initiatives. Given my career conducting evaluations of other organizations’ international projects, I was eager to examine Saving for Change from this perspective. My staff and I wanted to understand the impact of the groups as thoroughly as possible to make the program stronger. I wanted to learn whether what I believed was happening was borne out by the facts.
To do this, we set up two systems: one for monitoring and another for evaluation. Monitoring allowed us to track group performance on the basis of a set of simple data points, such as number of members, amounts saved and borrowed, loan repayment and outreach, and whether groups continue to save or disband. This would enable the staff to track the spread of the program and how much it cost. Evaluation asked the “so what” questions: Did those who joined groups change how they saved and borrowed? Did their businesses grow? Did their income increase? Did they experience less hunger? Were they more likely to send their children to school? Had their decision-making role in the household and the community changed?
The design of the evaluation started much earlier than 2008. In late 2005, a few months after the first groups in Mali were trained, I designed and launched an initial survey with the help of a Malian research organization.
As I visited groups with my local staff to determine what questions to include, I started with this question: What does it mean to be wealthy? One woman’s response remains with me today: “A rich person has enough to eat all year, has cattle and a plow to work the land, and at least a bed to sleep on. A poor person goes hungry, has no cattle, makes do with a hoe, and has nothing at home.”1Did having Saving for Change groups in a village lead to less hunger and more livestock—the measures of wealth as villagers had defined it so clearly for me before? We were about to find out.
In mid-2008, Oxfam America and Freedom from Hunger secured a major grant from the Bill & Melinda Gates Foundation to evaluate the impact of participating in Saving for Change groups in Mali and to build the program in Mali, Cambodia, and Guatemala. The grant set aside more than $1.25 million for a large-scale impact evaluation in Mali, where Saving for Change had grown most quickly.2 The Gates Foundation’s decision to investigate the impact of savings groups was made at the same time that several studies were published that were critical of MFIs for overstating the impact of their work.3 The Gates Foundation set aside so much money for research because it wanted there to be no question about the validity of the study. If the results were positive, this would justify further investment in this alternative way to promote rural financial services.
The three-year impact evaluation of Saving for Change in Mali combined a randomized controlled trial (RCT) and an in-depth anthropological study. Economists from Innovations for Poverty Action (IPA) collected baseline data in 2009, interviewing approximately six thousand women in 500 villages where Saving for Change had not yet been introduced. Of these 500 villages, 209 were then randomly selected to receive Saving for Change (“treatment” villages) and 291 were not (“control” villages). Local NGOs then began introducing the program in the selected communities. In 2012 a follow-up survey was conducted to measure the program’s impact after three years, comparing the changes in villages assigned to receive Saving for Change with changes in villages assigned as control sites.
Simultaneously, a team of anthropologists from the Bureau of Applied Research in Anthropology (BARA) at the University of Arizona brought a qualitative lens to the evaluation, conducting fewer but longer and deeper ethnographic case studies in carefully selected representative villages to explain and tease out complex dynamics. The BARA team analyzed nineteen villages, fifteen in the RCT study area and four where Saving for Change had existed since 2006. The anthropologists then tracked changes in these villages over the same three years.