Development of a participatory monitoring and evaluation strategy

This paper describes the process of developing a participatory monitoring and evaluation strategy for a Kenyan youth-based NGO. The iterative nature of the study including the process of narrowing down indicators to measure and methods to monitor/evaluate these is well documented. A discussion on the extent to which the process achieved participation and was empowering for the participants reflects on existing power relationships and cultural context of Kenya and points to the need to create opportunities for youth where they engage with the broader community. Lessons that emerge out of the study focus on the importance of prioritizing monitoring and evaluation, the potential of youth to carry out effective monitoring and evaluation, and the need for researchers to engage respectfully with communities and participants.     

How performance management regulations shape evaluation practice in South African municipalities

South African municipalities are taking hesitant steps towards institutionalising evaluation practice. This paper explores how evaluation is being institutionalised at a municipal level, looking at both the legislative and regulatory requirements in terms of planning, budgeting, and reporting, as well as opportunities for evaluative practice. A well-established performance management system, coupled with certain constraints to innovation and building an evaluative culture are contributing to the creation of an evaluation function that serves a monitoring purpose. A combination of desktop research and participatory action research was used to understand the municipal practice of evaluation. The results have implications for how decentralised public sector evaluation systems are understood.     

Between power and perception: Understanding youth perspectives in participatory monitoring and evaluation (PM&E) in Ghana

Participatory monitoring and evaluation (PM&E) approaches have been touted as an important concept for local participation in social interventions programmes. Utilizing a process analysis approaches, the PM&E data gathered through key informant interviews, formal surveys, and policy frameworks were analyzed. The study revealed three dominant power struggles between youth and programme implementers in three districts at the Greater Accra region, Ghana. The first and pervasive form of power dynamics involves the youth and programme implementers. The second involves a variety of arrangements with the government on one hand and implementers and youth on the other side. The third is who qualifies to be a beneficiary and for that matter participate in the PM&E. In these three forms of power struggles the paper reveals superior implementers control of who participate in the programme PM&E and at what stage in the process. The power imbalance between programme implementer and target beneficiaries impeded the level of beneficiaries’ participation in the PM&E.     

Assessment of the monitoring and evaluation system of a population-based program: Theory-driven evaluation approach

The purpose of this study was to assess the performance of a monitoring and evaluation system (M&E) for a population-based program and to identify factors influencing its performance. The state health department provided a grant, guide, and training for health districts to implement the program and report quarterly and annually to the M&E. Some health districts did not meet this requirement. We also found that the M&E did not adequately capture the actual work of the districts in communities as related to the work plan. We identified several factors that negatively affected the performances of the M&E. For example, the state and health districts did not have a shared vision on how to develop and implement the program, and what to report to the M&E. We proposed strategies that could improve the timeliness and content of reports to the M&E, as well as its utilization in the future.     

Power and perceptions in participatory monitoring and evaluation

This paper examines the assumptions that commonly underpin the design of participatory monitoring and evaluation (PM&E) within development programmes through the examination of a case study of a large rural development programme in Uganda. This case study reveals a mismatch between programme assumptions and participant perceptions, which stymied the implementation of PM&E. In this case, PM&E was based on the assumptions that programme and participant goals were compatible, participants were willing to volunteer and engage themselves for the greater good of their communities, and information sharing and communication was fairly free and open. However, farmers within the programme felt that the bureaucratic and accountability requirements of the programme were not their concern, and were acutely aware of power differences between farmers and programme officials, and between farmers of varying status. The key lesson to be drawn from this case is the need for a heightened awareness of power dynamics and political factors in the design of PM&E.     

Using a participatory evaluation design to create an online data collection and monitoring system for New Mexico's Community Health Councils

We present the collaborative development of a web-based data collection and monitoring plan for thirty-two county councils within New Mexico's health council system. The monitoring plan, a key component in our multiyear participatory statewide evaluation process, was co-developed with the end users: representatives of the health councils. Guided by the Institute of Medicine's Community, Health Improvement Process framework, we first developed a logic model that delineated processes and intermediate systems-level outcomes in council development, planning, and community action. Through the online system, health councils reported data on intermediate outcomes, including policy changes and funds leveraged. The system captured data that were common across the health council system, yet was also flexible so that councils could report their unique accomplishments at the county level. A main benefit of the online system was that it provided the ability to assess intermediate, outcomes across the health council system. Developing the system was not without challenges, including creating processes to ensure participation across a large rural state; creating shared understanding of intermediate outcomes and indicators; and overcoming technological issues. Even through the challenges, however, the benefits of committing to using participatory processes far outweighed the challenges.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

Endogenous socio-economic development in the European union — issues of evaluation

The public sector, in its policy statements and in the design of programmes of intervention, appears to be subscribing to a rhetoric of “grassroots participation”. At the same time, however, pressures are increasing for bureaucracies to adopt a managerialist modus operandi. This article considers the tensions that arise when participative service provision and programmes are subjected to evaluation scrutiny by managerialist bodies. The discussion takes place in the context of an EU-sponsored endogenous socio-economic development initiative. This prepares the ground for an exploration of participative evaluation.     

Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey

The demand for improved quality of health promotion evaluation and greater capacity to undertake evaluation is growing, yet evidence of the challenges and facilitators to evaluation practice within the health promotion field is lacking. A limited number of evaluation capacity measurement instruments have been validated in government or non-government organisations (NGO), however there is no instrument designed for health promotion organisations. This study aimed to develop and validate an Evaluation Practice Analysis Survey (EPAS) to examine evaluation practices in health promotion organisations. Qualitative interviews, existing frameworks and instruments informed the survey development. Health promotion practitioners from government agencies and NGOs completed the survey (n = 169). Principal components analysis was used to determine scale structure and Cronbach’s α used to estimate internal reliability. Logistic regression was conducted to assess predictive validity of selected EPAS scale. The final survey instrument included 25 scales (125 items). The EPAS demonstrated good internal reliability (α > 0.7) for 23 scales. Dedicated resources and time for evaluation, leadership, organisational culture and internal support for evaluation showed promising predictive validity. The EPAS can be used to describe elements of evaluation capacity at the individual, organisational and system levels and to guide initiatives to improve evaluation practice in health promotion organisations.     

Evaluation capacity building (ECB) interventions and the development of sustainable evaluation practice: An exploratory study

Evaluation capacity building (ECB) is a practice that can help organizations conduct and use evaluations; however, there is little research on the sustainable impact of ECB interventions. This study provides an empirical inquiry into how ECB develops sustained evaluation practice. Interviews were conducted with 15 organizational leaders from non-profits, higher education institutions, and foundations that “bought in” to ECB and were at least six months removed from an ECB contract. The result of this work highlights how sustained evaluation practice developed over time and what these practices looked like in real-world settings. A developmental, iterative cycle for how ECB led organizations to sustain evaluation practice emerged around key components to sustainability. First, leadership supported ECB work and resources were dedicated to evaluation. Staff began to conduct and use evaluation, which led to understanding the benefits of evaluation, and promoted value and buy-in to evaluation among staff. Common barriers and emerging sustainability supports not previously identified by ECB literature—the “personal” factor and ongoing ECB practitioner contact—are described. Practical tips for ECB practitioners to promote sustainability are also detailed.     

Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned

Organizational evaluation capacity (EC) has received significant attention in the evaluation research literature in the past decade. Much of the focus has been on defining organizational evaluation capacity, which can be thought of as the competencies and structures required to conduct high-quality evaluation studies (capacity to do), as well as the organization's ability to integrate evaluation findings into its decision-making processes (capacity to use). This paper seeks to contribute to this growing body of knowledge through a multiple case study of EC across three different organizations (e.g., non-profit, provincial government and federal government, herein named sectors); the novelty of this particular study is that each case study is based on the use of a common measurement tool developed by Bourgeois, Toews, Whynot and Lamarche (2013). The cross-case analysis presented in the paper reveals that evaluation capacity tends to be higher, both in terms of capacity to do and capacity to use, in organizations that have developed systematic mechanisms to institute an evaluation culture within their walls. Interestingly, however, we also found that capacity to use does not first require capacity to do, as evidenced in the non-profit organization under study.     

Mediating role of citizen empowerment in the relationship between participatory monitoring and evaluation and social sustainability

The paper explores the mediating influence of citizen empowerment on the relationship between participatory monitoring and evaluation (PM&E) and social sustainability as a contribution to the ongoing debate on sustainability. The empirical investigation took the form of mixed-methods approach with the objectives to determine the extent to which PM&E influences citizen empowerment; establish the extent to which citizen empowerment influences social sustainability; and establish the joint influence of PM&E and citizen empowerment on social sustainability. The study employed a concurrent parallel design, in which samples for quantitative and qualitative components were different but drawn from the same population, and data collected within the same timeframe. Quantitative data were analyzed through bivariate and multiple regression analyses. Conversely, the qualitative component utilized interpretive technique and recursive abstraction to organize, summarize and give impression on the causal mechanisms at play in the quantitative data. Evidence from the study revealed that citizen empowerment mediates the relationship between PM&E and social sustainability. The study also found that when PM&E and citizen empowerment are considered together, citizen empowerment seems to have a dominant influence on social sustainability than PM&E itself. Thus, the government and development practitioners should prioritize citizen empowerment as a precursor to the realization of social sustainability outcomes.     

Contextual influence on evaluation capacity building in a rapidly changing environment under new governmental policies

Evaluation capacity building (ECB) is a context-dependent process. Contextual factors affecting ECB implementation have been explored theoretically and practically, but their influence within a changing environment has seldom been discussed. This study examined essential context-sensitive parameters, particularly those involved in implementing new governmental policies regarding higher education. Taiwan was used as a case study for exploring the effect of contextual change on ECB attributes from the perspectives of training receivers and providers. Surveys and interviews were used for data collection and importance–performance analysis was applied for data analysis. Four prominent features were identified. First, the ECB attributes perceived as important by receivers were performed adequately, whereas those perceived as less important were performed less well. Second, under new policies, training provider designed training covering a wide range of ECB, whereas receivers focused on those can be directly applied in evaluation process. Third, in a small education system such as Taiwan’s, the complexity of peer review is high and ethical issues become important. Fourth, because the evaluation structure has been changed from single- to dual-track, receivers expect more training for institution staff, whereas providers insist on hierarchical training. Aligning ECB supply and needs is paramount for adaptation to new policies.     

Lapses,infidelities, and creative adaptations: Lessons from evaluation of a participatory market development approach in the Andes

Participatory approaches are frequently recommended for international development programs, but few have been evaluated. From 2007 to 2010 the Andean Change Alliance evaluated an agricultural research and development approach known as the “Participatory Market Chain Approach” (PMCA). Based on a study of four cases, this paper examines the fidelity of implementation, the factors that influenced implementation and results, and the PMCA change model. We identify three types of deviation from the intervention protocol (lapses, creative adaptations, and true infidelities) and five groups of variables that influenced PMCA implementation and results (attributes of the macro context, the market chain, the key actors, rules in use, and the capacity development strategy). There was insufficient information to test the validity of the PMCA change model, but results were greatest where the PMCA was implemented with highest fidelity. Our analysis suggests that the single most critical component of the PMCA is engagement of market agents – not just farmers – throughout the exercise. We present four lessons for planning and evaluating participatory approaches related to the use of action and change models, the importance of monitoring implementation fidelity, the limits of baseline survey data for outcome evaluation, and the importance of capacity development for implementers.     

The importance of networking in development co‐operation: A social network analysis‐based evaluation of an international development programme

This article aims to demonstrate that networking activities, if properly planned, applied and monitored, can help increase the long‐term impact of development co‐operation actions. We demonstrate the added value of networking within development actions, focusing on dynamics of trust and reciprocity and on the rationales beyond different collaboration patterns, demonstrating the impact of networking on program effectiveness, intercultural learning and sustainability. Using social network analysis coupled with participant observation, it was possible to trace the development of a large multi‐stakeholder development programme and to understand a number of dynamics not considered by the donor when evaluating the action, thereby reaching a better understanding of the challenges and success of the programme.     

A participatory evaluation framework in the establishment and implementation of transdisciplinary collaborative centers for health disparities research

ObjectiveWe describe the formulation and implementation of a participatory evaluation plan for three Transdisciplinary Collaborative Centers for Health Disparities Research funded by the National Institute of Minority Health and Health Disparities.MethodsAlthough different in scope of work, all three centers share a common goal of establishing sustainable centers in health disparities science in three priority areas – social determinants of health, men’s health research, and health policy research.ResultsThe logic model guides the process, impact, and outcome evaluation. Emphasis is placed on process evaluation in order to establish a “blue print” that can guide other efforts as well as assure that activities are being implemented as planned.ConclusionWe have learned three major lessons in this process: (1) Significant engagement, participation, and commitment of all involved is critical for the evaluation process; (2) Having a “roadmap” (logic model) and “directions” (evaluation worksheets) are instrumental in getting members from different backgrounds to follow the same path; and (3) Participation of the evaluator in the leadership and core meetings facilitates continuous feedback.     

Valuing and embracing complexity: How an understanding of complex interventions needs to shape our evaluation capacities building initiatives

This paper describes some of the main challenges of evaluating complex interventions, as well as the implications of such challenges for evaluation capacity building. It discusses lessons learned from a case study of an evaluation of Dancing with Parkinson’s, an organization that provides dance classes to people with Parkinson’s disease in Toronto, Canada. These implications are developed from a realist evaluation lens. Key lessons include the need to develop skills to understand program mechanisms and contexts, recognize multiple models of causality, apply mixed method designs, and ensure the successful scaling up and spread of an intervention.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

Problems and possibilities of development work among tribals: a participatory evaluation of a voluntary agency in India

Mandal, a church-related non-governmental voluntary agency operating in north-west India's Nasik district since 1966 was evaluated for performance in promoting self-sustaining economic growth and socio- political awareness in its beneficiaries. The research design incorporated 2 questions, the extent to which Mandal had used the integrated approach, and the extent to which it promoted structural change in the village. 2 levels of analysis were used: random samples of 315 individuals (163 beneficiaries and 152 non-beneficiaries) and data from the takatu's office for 20 involved villages and 8 control villages. Quantitative data were sought by examining files and doing closed interviews, and qualitative data were collected by observation and open-ended interviews. The variables used in the analysis were educational level, class category, caste, agricultural investment, family net income, individual consciousness and social participation. It was concluded that on the individual level some positive impact was discernible in terms of family income and agricultural investment. At the village level no quantitative improvement was measurable, but the effect of Mandal was symbolic in that it sets an example for social change. The Mandal has acquired a health, non-partisan secular image for social action.