Building national M&E systems in the context of changing aid modalities: The underexplored potential of National Evaluation Societies

Changes in the aid architecture have provided a renewed impetus for monitoring and evaluation (M&E), while simultaneously imposing a major reform agenda on the key players involved. More specifically, since 1999, aid-dependent countries have been facing pressure to strengthen their national M&E systems, while donors have been asked to refrain from using their own parallel systems and to rely instead on country systems. Surprisingly, attempts to strengthen national M&E frameworks have thus far largely overlooked the potential of national evaluation societies (NES). Similarly, NES have also remained off the academic radar. Our study aims to fill this gap by mapping key features of NES, as well as their perceived contributions to country-led M&E. In this effort, we rely upon evidence from our survey of 23 NES in Sub-Saharan African countries with Poverty Reduction Strategy Papers (PSRP). Our findings show that there is quite some diversity among NES. Overall, NES are active organisations, whose unique membership features a wide variety of national M&E stakeholders who potentially play key roles in country-led and localised M&E development. Major obstacles faced by NES include the lack of financial resources, donor support and political influence. Survey findings also demonstrate that the increasing interest of donors and governments in NES has yet to materialise into strategic support.     

Assessment of the monitoring and evaluation system of a population-based program: Theory-driven evaluation approach

The purpose of this study was to assess the performance of a monitoring and evaluation system (M&E) for a population-based program and to identify factors influencing its performance. The state health department provided a grant, guide, and training for health districts to implement the program and report quarterly and annually to the M&E. Some health districts did not meet this requirement. We also found that the M&E did not adequately capture the actual work of the districts in communities as related to the work plan. We identified several factors that negatively affected the performances of the M&E. For example, the state and health districts did not have a shared vision on how to develop and implement the program, and what to report to the M&E. We proposed strategies that could improve the timeliness and content of reports to the M&E, as well as its utilization in the future.     

A critical friend: Monitoring and evaluation systems,development cooperation and local government. The case of Tuscany

The role of monitoring and evaluation (M&E) systems in the field of development cooperation has globally increased in last decades. International and regional organizations, as well as states, local governments and NGOs have largely adopted the tools provided by M&E in order to enhance transparency, effectiveness and efficiency. The paper aims at verifying how and to what extent the implementation of M&E systems has affected the overall quality of international cooperation projects financed by a local government. After a literature review on M&E in development cooperation, the research analyzes the wide range of activities (evaluation ex-ante, mid-term, final, monitoring, consultancy) carried out by the Evaluation Team of the XY in the last eight years in behalf of an Italian local government: the Region of Tuscany. The paper reveals the strategic significance of adopting M&E systems in the medium-long term.     

Monitoring and evaluation design of Malawi’s Right Foods at the Right Time nutrition program

Child stunting is a public health problem in Malawi. In 2014, the Government of Malawi launched the Right Foods at the Right Time (RFRT) program in Ntchisi district delivering nutrition social and behavior change communication, a small-quantity lipid-based nutrient supplement to children 6–23 months, and nutrition sensitive activities. Monitoring and evaluation (M&E) systems are key aspects of successful program implementation. We describe these and the methodology for an impact evaluation that was conducted for this program. Two monitoring systems using traditional and electronic platforms were established to register and track program delivery and processes including number of eligible beneficiaries, worker performance, program participation, and to monitor input, output, and outcome indicators. The impact evaluation used comparative cross-sectional and longitudinal designs to assess impact on anthropometric and infant and young child feeding outcomes. Three cross-sectional surveys (base-, mid-, and end-line) and two longitudinal cohorts of children followed in 6-month intervals from 6 to 24 months of age, were conducted in sampled households in the program and a neighboring comparison district. Additional M&E included qualitative studies, a process evaluation, and a cost-effectiveness study. The current paper describes lessons from this program’s M&E, and demonstrates how multiple implementation research activities can inform course-correction and program scale-up.     

Global Convergence: Aging and Long-Term Care Policy Challenges in the Developing World

Aging around the world poses a global challenge in eldercare. This challenge is particularly felt in low- and middle-income countries (LMICs), where population aging outpaces the development of aged care policies and services. This Perspective highlights the phenomenon of global convergence in several unsettling trends and challenges shared across LMICs. These include the weakening of informal family care systems for the elderly, growing need for formal long-term care of the frail and disabled who can no longer be adequately supported by family members, and mounting pressures for policy responses to tackle these societal challenges. It is argued that policymakers should take a proactive stance. That is, when family care for the elderly falls short and family caregivers are increasingly under strain, the government should step in and step up support to fill in the gap by developing appropriate policies and a continuum of long-term care services that are accessible and affordable for the majority of older people in need. Three general principles are then suggested with regard to long-term care provision, financing, and quality assurance, which transcend national borders and can be used to guide long-term care policymaking across LMICs.     

A call for reconfiguring evaluation models,pedagogy, and practice: Beyond reporting media-centric outputs and fake impact scores

Evaluation is an essential part of strategic communication practices including public relations and corporate, government, and organizational communication. Evaluation can potentially demonstrate effectiveness and value, facilitate learning to improve practice, and provide insights to inform future strategy. In the age of digitalization of media and information, there are more readily available data to enable evaluation than ever before. However, despite half a century of theorizing and discussing measurement and evaluation (M&E), literature reports “stasis” and a “deadlock” well into the 21st century In some cases, invalid practices such as advertising value equivalents (AVEs) continue to be used. Furthermore, when M&E collectively is part of communication practice, evidence shows that reports continue to focus on outputs and algorithm-generated fake impact scores, rather than genuine outcomes and impact. Despite the body of literature on M&E, it is clear that more needs to be said – and done. This discussion calls for reconfiguring common evaluation models to overcome false logics and for a reconfiguring and broadening of pedagogy and practice to break the deadlock and realize the benefits of evaluation.     

Taking Stock of Monitoring and Evaluation Arrangements in the Context of Poverty Reduction Strategy Papers: Evidence from 20 Aid‐Dependent Countries in Sub‐Saharan Africa

Shifts in thinking and practice concerning aid have posed serious challenges in monitoring and evaluation (M&E) for both donors and recipients, but progress in reform remains slow. An important first step in any initiative involving M&E capacity development is the diagnosis of the systems' current status. This article presents a diagnostic checklist that captures issues of M&E policy: indicators, data collection and methodology; organisation; capacity‐building; participation of nongovernmental actors; and use. It applies it to a review of the PRSP M&E arrangements of 20 aid‐dependent countries in sub‐Saharan Africa to demonstrate comparative strengths and weaknesses.     

Barriers and facilitators to implementing evidence-based health innovations in low- and middle-income countries: A systematic literature review

The unsuccessful implementation of health innovations occurs frequently, leading to missed opportunities where improvements could have been made on various aspects of a health system. The purpose of this study is to identify, assess and synthesise the facilitators and barriers to sustainably implementing evidence-based health innovations in a low- and middle-income country (LMIC) context. To identify the LMIC specific facilitators and barriers, a systematic literature review was conducted. 79 studies were analysed, and the implementation barriers and facilitators identified in each study were extracted. The extracted barriers and facilitators were categorised and synthesized into one of seven concepts: context, innovation, relations and networks, institutions, knowledge, actors or resources. There were no substantial variations between the frequency that each concept was identified as a facilitator or barrier to implementation. However, resources, which includes time, human, financial and physical resources, was the most frequently mentioned concept; emphasising the need to focus on the resource situation in LMICs. This study contributes to the growing literature that aims to inform health system planners and evaluators in LMICs on effectively and sustainably implementing evidence-based health innovations.     

Reviewing the ‘march to standards’ in public relations: a comparative analysis of four seminal measurement and evaluation initiatives

To many, development and adoption of professional standards for measurement and evaluation (M&E) is one of the most promising approaches for advancing public relations practice. In recent years, there has been a surge in efforts to develop standards for M&E in different parts of the world. Prominent examples of this include standard terminologies, metrics, principles for best practice in the field, and evaluation frameworks. Regardless of their alleged importance, however, the acceptance and application of such M&E standards in the practice varies significantly. To better understand the process by which standards in this field are developed and adopted, this article draws on recent concepts from organization studies (cf. Slager, Gond, & Moon, 2012) to analyze the trajectories of four seminal standards attempts: The Barcelona Principles and the AMEC Integrated Evaluation Framework on an international level, the DPRG/ICV Framework used in German-speaking countries, and the GCS Framework in the United Kingdom. The article reveals, by way of an interpretive qualitative approach, the various strategies undertaken to a) develop common sets of terms and rules, b) engage relevant actors in the design, promotion, and implementation of proposed standards, and c) to reinforcing standards symbolically.     

Identifying and controlling for program-level differences in comparative cost analysis: lessons from the economic evaluation of the National Breast and Cervical Cancer Early Detection Program

Performing economic evaluations of established health care programs is essential to identify and control for underlying program-level variations and to make valid comparisons. At a time when the need for such evaluations is growing, health care professionals have limited information on the methodological challenges of performing these evaluations. In this study, we used the National Breast and Cervical Cancer Early Detection Program to illustrate these potential underlying variations. We performed site visits to four grantees and collected activity-based cost data from nine additional representative programs. We identified five specific types of cost factors that should be considered when evaluating and comparing health care programs: clinical services, service mix, in-kind contributions, indirect costs, and year-to-year expenditures of specific activities. A key lesson is that case studies and pilot testing should be performed before initiating cost analysis to identify underlying variation and to test appropriate methods to adequately control for these differences.     

Design of a collaborative monitoring and evaluation system for a community-based nutrition project in rural Bangladesh

Poor diet and undernutrition are common among children living in Bangladesh. To promote appropriate complementary feeding of young children, an economic development (ED) program involving income-generating asset transfer was implemented alongside a social and behavior change (SBC) program. This paper introduces a collaborative monitoring and evaluation (M&E) system in which diverse collaborators (“research group”, “implementation team”, and “coordinators”) facilitate M&E data acquisition by leveraging their comparative advantages. The implementation team built a monitoring system to track the ED (n = 2960) and SBC participants (n=∼10,000) over 12 months. Based on the baseline design and the monitoring records, the collaborators planned an impact evaluation introducing a quasi-experimental design using two cross-sectional surveys and a prospective cohort survey of child feeding and nutritional status. Using various data sources generated from the M&E system, the collaborators will also reveal the program impact pathway through which each intervention component is delivered, received, and utilized alongside the context-specific facilitators and barriers, including the programs’ uptake. The collaborative M&E system enables the sharing of program goals, strengthens collaborators' commitment to the program, and extends the understanding of the program's progress and evaluation activities.     

The MHSIP mental health report card. A consumer-oriented approach to monitoring the quality of mental health plans. Mental Health Statistics Improvement Program

Health care report cards have been endorsed as a mechanism for efficiently comparing key quantifiable aspects of performance across a range of health systems or plans. There are challenges in determining what to measure; how to gather and analyze data; and how to report, interpret, and use findings. Mental health has received little attention, and a consumer perspective is typically not included. The proposed MHSIP mental health report card (MMHRC) addresses these concerns. General issues for report cards are discussed, and the MMHRC is described in terms of content, data sources and quality, and analysis and reporting.     

Video–based observation in impact evaluation

This paper reviews the literature on the use of video–based observation (VBO) with particular attention to monitoring and evaluation (M&E) on development projects. While the use of video both as a research tool and as a strategy for supporting professional development is well–documented across several disciplines, the extent to which VBO has been utilized in M&E contexts is less clearly defined. In order to provide theoretically–grounded recommendations for the development and implementation of one organisation’s innovative VBO impact evaluation scheme, this review examines recent evidence of VBO in M&E contexts and draws on VBO literature more generally to identify its advantages and challenges together with advice for enhancing its effectiveness. Based on this analysis, the paper highlights a number of practical issues that should be considered when VBO is being developed for M&E in development contexts, particularly where videos are being made by participants themselves. The value of VBO in responding to COVID–19 and reducing carbon emissions is also noted.     

A critical review on current practices of the monitoring and evaluation in the preparation of strategic urban plans within the Egyptian context

Worldwide, strategic urban planning is found in very different contexts and planning schemes, where it is always a tool strongly linked to enhancing urban management. Under the umbrella of strategic urban planning different approaches can be perceived, in which within these broad shared characteristics there are significant variations. A key component in the strategic urban planning process is the monitoring and evaluation (M&E) element and these differences can be seen in the way that M&E is approached. Egypt is currently on the track to reform its planning system shifting from the conventional master plans towards strategic urban plans (SUPs). This paper reflects on the current practices of M&E the preparation of SUPs within the Egyptian context. It sheds light on the strengths and weaknesses of the employed techniques and concludes with some recommendations to improve the process of M&E, which builds upon the strengths in the employed techniques.     

Primary healthcare planning,bottleneck analysis and performance improvement: An evaluation of processes and outcomes in a Nigerian context

Operational planning of interventions defines roadmaps, timelines and resources necessary for translating policies into expected health outcomes along the evidence-policy-implementation continuum. However, bottlenecks often hinder the attainment of objectives and the timely delivery of intervention packages leading to sub-optimal performance of health systems. Bottleneck identification, analysis and removal approaches to planning, which requires key stakeholders’ participation, have been recommended to improve health system outcomes in LMICs. This study demonstrates how integration of participatory action research (PAR) within a quality improvement model can help navigate the complexities of health system bottleneck analyses, planning and performance improvement in a Nigerian sub-national context.The study is based on data collected between June 2016 and June 2017, from Chikun LGA in Kaduna State Nigeria. PAR was integrated into a quality improvement model called DIVA (Diagnose-Intervene-Verify-Adjust) applied across selected interventions (eMTCT, Antenatal care, skilled birth attendance, immunization and Integrated Management of Childhood Illnesses). PAR was used to identify and analyse health system bottlenecks, as well as develop, monitor implementation and follow-up on action plans to address them. Evaluations were conducted involving 2 cycles of DIVA. The outputs (bottleneck analysis charts, driver diagrams, operational plans, M/E reports, etc.) from each cycle of the DIVA process were collated and analysed. Bottlenecks identified include availability of human resources for health, availability of health commodities as well as geographical accessibility. These had implications on acceptability and quality of services. Mean Improvements recorded were 20.4%, 14.0% and 10.8% and 11.2%, 7.5%; 5.5% (across eMTCT, maternal health and child health interventions) in the 1 st and 2nd DIVA cycles respectively. This study highlights processes and outcomes of integrating PAR in quality improvement design and operations in health intervention programmes with a focus on health systems strengthening in a Nigerian context. Implementing the DIVA model using a PAR approach may be considered an effective strategy for planning and implementing health interventions in comparable settings.     

SenseMaker® as a monitoring and evaluation tool to provide new insights on gender-based violence programs and services in Lebanon

Monitoring and evaluation (M&E) of gender-based violence (GBV) programs is challenging in humanitarian settings. To address these challenges, we used SenseMaker® as a mixed methods M&E tool for GBV services and programs in Lebanon. Over a three-month period in 2018, a total of 198 self-interpreted stories were collected from women and girls accessing GBV programs from six service providers across five locations. The resultant mixed-methods analysis provided holistic and nuanced insights on how perceived benefits differed by type of GBV program, how motivations for accessing programs differed by location, and how feelings while accessing programs differed by participant nationality. SenseMaker reinforced the intersectionality between events leading up to the accessed services, the experiences of accessing the services, and subsequent outcomes as a result of having accessed the services, helping to contextualize the findings within the broader experiences of participating women and girls. Limited literacy and technology skills among participants proved to be a challenge and future work should investigate how technology might facilitate use of the tool among participants with lower literacy and technology skills in addition to exploring the feasibility and added value of SenseMaker as an M&E tool in acute humanitarian settings.     

Home- and center-based learning opportunities for preschoolers in low- and middle-income countries

Recent international development efforts have emphasized the importance of supporting early childhood development, yet little is known about the availability of early learning opportunities in low- and middle-income countries (LMICs). The present study uses nationally representative data from >163,000 three- and four-year-olds living in 63 countries to estimate the availability of in- and out-of-home early learning opportunities in LMICs. Results suggest that 71.9% of preschool-aged children experience high levels of at-home stimulation (e.g., reading, counting, drawing), 33.6% attend center-based early childhood care and education (ECCE) programming, 29.1% experience both, and 22.9% experience neither. Large geographical and socioeconomic disparities in learning opportunities were found both across and within countries, particularly for ECCE.     

Planned treatment and outcomes in residential youth care: Evidence from Sweden

A recurring theme in evaluations of Swedish residential youth care is that treatment is often unplanned. Using a data set of teenagers placed in youth care in 1991 (N = 357), we show that planned treatment — in the sense of a known expected duration of treatment — is strongly positively associated with treatment outcomes. In the short term, teenagers with planned treatment are 32% less likely to experience a treatment breakdown and 25% less likely to be reassigned to other forms of residential care after completed treatment. In the long term, teenagers with planned treatment are 21% less likely to engage in criminal behavior and 40% less likely to be hospitalized for mental health problems. The results are robust to controlling for a rich set of potentially confounding factors: Even though observable pre-treatment teenager characteristics explain about one fifth of the variation in criminal behavior 5-10 years after treatment, they have almost no predictive power for whether treatment is planned or unplanned.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.