Assessment of the monitoring and evaluation system of a population-based program: Theory-driven evaluation approach

The purpose of this study was to assess the performance of a monitoring and evaluation system (M&E) for a population-based program and to identify factors influencing its performance. The state health department provided a grant, guide, and training for health districts to implement the program and report quarterly and annually to the M&E. Some health districts did not meet this requirement. We also found that the M&E did not adequately capture the actual work of the districts in communities as related to the work plan. We identified several factors that negatively affected the performances of the M&E. For example, the state and health districts did not have a shared vision on how to develop and implement the program, and what to report to the M&E. We proposed strategies that could improve the timeliness and content of reports to the M&E, as well as its utilization in the future.     

Monitoring and evaluation design of Malawi’s Right Foods at the Right Time nutrition program

Child stunting is a public health problem in Malawi. In 2014, the Government of Malawi launched the Right Foods at the Right Time (RFRT) program in Ntchisi district delivering nutrition social and behavior change communication, a small-quantity lipid-based nutrient supplement to children 6–23 months, and nutrition sensitive activities. Monitoring and evaluation (M&E) systems are key aspects of successful program implementation. We describe these and the methodology for an impact evaluation that was conducted for this program. Two monitoring systems using traditional and electronic platforms were established to register and track program delivery and processes including number of eligible beneficiaries, worker performance, program participation, and to monitor input, output, and outcome indicators. The impact evaluation used comparative cross-sectional and longitudinal designs to assess impact on anthropometric and infant and young child feeding outcomes. Three cross-sectional surveys (base-, mid-, and end-line) and two longitudinal cohorts of children followed in 6-month intervals from 6 to 24 months of age, were conducted in sampled households in the program and a neighboring comparison district. Additional M&E included qualitative studies, a process evaluation, and a cost-effectiveness study. The current paper describes lessons from this program’s M&E, and demonstrates how multiple implementation research activities can inform course-correction and program scale-up.     

A critical friend: Monitoring and evaluation systems,development cooperation and local government. The case of Tuscany

The role of monitoring and evaluation (M&E) systems in the field of development cooperation has globally increased in last decades. International and regional organizations, as well as states, local governments and NGOs have largely adopted the tools provided by M&E in order to enhance transparency, effectiveness and efficiency. The paper aims at verifying how and to what extent the implementation of M&E systems has affected the overall quality of international cooperation projects financed by a local government. After a literature review on M&E in development cooperation, the research analyzes the wide range of activities (evaluation ex-ante, mid-term, final, monitoring, consultancy) carried out by the Evaluation Team of the XY in the last eight years in behalf of an Italian local government: the Region of Tuscany. The paper reveals the strategic significance of adopting M&E systems in the medium-long term.     

Advances in monitoring and evaluation in low- and middle-income countries

Data to inform and improve health care systems in low- and middle-income countries (LMICs) has been facilitated by the development of monitoring and evaluation (M&E) systems. The drivers of change in M&E systems over the last 50 years have included a series of health concerns that have animated global donors (e.g., family planning, vaccination campaigns, and HIV/AIDS); the data requirements of donors; improved national economies enabling LMICs to invest more in M&E systems; and rapid advances in digital technologies. Progress has included the training and expansion of an M&E workforce, the creation of systems for data collection and use, and processes for assessing and ensuring data quality. Controversies have included the development of disease-specific systems that do not coordinate with each other, and a growing burden on health care deliverers to collect data for a proliferating number of health and process indicators. Digital technologies offer the promise of real time data and quick adaptation but also raise ethical and privacy concerns. The desire for speed can cast large-scale evaluations, considered by some to be the gold standard, in an unfavorable light as slow and expensive. Accordingly, there is a growing demand for speedy evaluations that rely on routine health information systems and privately collected “big data” from electronic health records and social media.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

A critical review on current practices of the monitoring and evaluation in the preparation of strategic urban plans within the Egyptian context

Worldwide, strategic urban planning is found in very different contexts and planning schemes, where it is always a tool strongly linked to enhancing urban management. Under the umbrella of strategic urban planning different approaches can be perceived, in which within these broad shared characteristics there are significant variations. A key component in the strategic urban planning process is the monitoring and evaluation (M&E) element and these differences can be seen in the way that M&E is approached. Egypt is currently on the track to reform its planning system shifting from the conventional master plans towards strategic urban plans (SUPs). This paper reflects on the current practices of M&E the preparation of SUPs within the Egyptian context. It sheds light on the strengths and weaknesses of the employed techniques and concludes with some recommendations to improve the process of M&E, which builds upon the strengths in the employed techniques.     

A call for reconfiguring evaluation models,pedagogy, and practice: Beyond reporting media-centric outputs and fake impact scores

Evaluation is an essential part of strategic communication practices including public relations and corporate, government, and organizational communication. Evaluation can potentially demonstrate effectiveness and value, facilitate learning to improve practice, and provide insights to inform future strategy. In the age of digitalization of media and information, there are more readily available data to enable evaluation than ever before. However, despite half a century of theorizing and discussing measurement and evaluation (M&E), literature reports “stasis” and a “deadlock” well into the 21st century In some cases, invalid practices such as advertising value equivalents (AVEs) continue to be used. Furthermore, when M&E collectively is part of communication practice, evidence shows that reports continue to focus on outputs and algorithm-generated fake impact scores, rather than genuine outcomes and impact. Despite the body of literature on M&E, it is clear that more needs to be said – and done. This discussion calls for reconfiguring common evaluation models to overcome false logics and for a reconfiguring and broadening of pedagogy and practice to break the deadlock and realize the benefits of evaluation.     

Video–based observation in impact evaluation

This paper reviews the literature on the use of video–based observation (VBO) with particular attention to monitoring and evaluation (M&E) on development projects. While the use of video both as a research tool and as a strategy for supporting professional development is well–documented across several disciplines, the extent to which VBO has been utilized in M&E contexts is less clearly defined. In order to provide theoretically–grounded recommendations for the development and implementation of one organisation’s innovative VBO impact evaluation scheme, this review examines recent evidence of VBO in M&E contexts and draws on VBO literature more generally to identify its advantages and challenges together with advice for enhancing its effectiveness. Based on this analysis, the paper highlights a number of practical issues that should be considered when VBO is being developed for M&E in development contexts, particularly where videos are being made by participants themselves. The value of VBO in responding to COVID–19 and reducing carbon emissions is also noted.     

Building national M&E systems in the context of changing aid modalities: The underexplored potential of National Evaluation Societies

Changes in the aid architecture have provided a renewed impetus for monitoring and evaluation (M&E), while simultaneously imposing a major reform agenda on the key players involved. More specifically, since 1999, aid-dependent countries have been facing pressure to strengthen their national M&E systems, while donors have been asked to refrain from using their own parallel systems and to rely instead on country systems. Surprisingly, attempts to strengthen national M&E frameworks have thus far largely overlooked the potential of national evaluation societies (NES). Similarly, NES have also remained off the academic radar. Our study aims to fill this gap by mapping key features of NES, as well as their perceived contributions to country-led M&E. In this effort, we rely upon evidence from our survey of 23 NES in Sub-Saharan African countries with Poverty Reduction Strategy Papers (PSRP). Our findings show that there is quite some diversity among NES. Overall, NES are active organisations, whose unique membership features a wide variety of national M&E stakeholders who potentially play key roles in country-led and localised M&E development. Major obstacles faced by NES include the lack of financial resources, donor support and political influence. Survey findings also demonstrate that the increasing interest of donors and governments in NES has yet to materialise into strategic support.     

Reviewing the ‘march to standards’ in public relations: a comparative analysis of four seminal measurement and evaluation initiatives

To many, development and adoption of professional standards for measurement and evaluation (M&E) is one of the most promising approaches for advancing public relations practice. In recent years, there has been a surge in efforts to develop standards for M&E in different parts of the world. Prominent examples of this include standard terminologies, metrics, principles for best practice in the field, and evaluation frameworks. Regardless of their alleged importance, however, the acceptance and application of such M&E standards in the practice varies significantly. To better understand the process by which standards in this field are developed and adopted, this article draws on recent concepts from organization studies (cf. Slager, Gond, & Moon, 2012) to analyze the trajectories of four seminal standards attempts: The Barcelona Principles and the AMEC Integrated Evaluation Framework on an international level, the DPRG/ICV Framework used in German-speaking countries, and the GCS Framework in the United Kingdom. The article reveals, by way of an interpretive qualitative approach, the various strategies undertaken to a) develop common sets of terms and rules, b) engage relevant actors in the design, promotion, and implementation of proposed standards, and c) to reinforcing standards symbolically.     

Supporting the caregiver-child dyad’s relationship: An evaluation of implementation quality in the Chilean Crecer Jugando program

This study presents an evaluation of the implementation quality of the Chilean program Crecer Jugando (CJ), a 16 weekly sessions group-based parenting program for children 0 to 4 years old and their primary caregivers aiming at promoting positive caregiver-child interaction. The implementation of CJ in two public health care centers (HCC) in Chile’s Metropolitan Region was assessed based on Donabedian’s theoretical model, focusing on the dimensions of the program’s structure (e.g., infrastructure and supplies), processes (e.g., coordination of CJ team with the HCCs, participants’ attendance, CJ team interaction with participating children), and preliminary outcomes (i.e., parenting stress, caregiver-child interaction). A total of 63 main caregiver-child dyads participated in the study, which took place over a six-month period. Results indicated that the CJ program was feasible to be implemented in two HCCs and would benefit from improving the coordination with the HCCs and the quality of interaction of the CJ team with participating children. After participation in the CJ program, caregivers showed a decrease in their parenting stress. Lessons learned are discussed.     

SenseMaker® as a monitoring and evaluation tool to provide new insights on gender-based violence programs and services in Lebanon

Monitoring and evaluation (M&E) of gender-based violence (GBV) programs is challenging in humanitarian settings. To address these challenges, we used SenseMaker® as a mixed methods M&E tool for GBV services and programs in Lebanon. Over a three-month period in 2018, a total of 198 self-interpreted stories were collected from women and girls accessing GBV programs from six service providers across five locations. The resultant mixed-methods analysis provided holistic and nuanced insights on how perceived benefits differed by type of GBV program, how motivations for accessing programs differed by location, and how feelings while accessing programs differed by participant nationality. SenseMaker reinforced the intersectionality between events leading up to the accessed services, the experiences of accessing the services, and subsequent outcomes as a result of having accessed the services, helping to contextualize the findings within the broader experiences of participating women and girls. Limited literacy and technology skills among participants proved to be a challenge and future work should investigate how technology might facilitate use of the tool among participants with lower literacy and technology skills in addition to exploring the feasibility and added value of SenseMaker as an M&E tool in acute humanitarian settings.     

Using a logic model to evaluate the Kids Together early education inclusion program for children with disabilities and additional needs

Despite clear evidence that learning and social opportunities for children with disabilities and special needs are more effective in inclusive not segregated settings, there are few known effective inclusion programs available to children with disabilities, their families or teachers in the early years within Australia. The Kids Together program was developed to support children with disabilities/additional needs aged 0–8 years attending mainstream early learning environments. Using a key worker transdisciplinary team model, the program aligns with the individualised package approach of the National Disability Insurance Scheme (NDIS).AimThis paper reports on the use of a logic model to underpin the process, outcomes and impact evaluation of the Kids Together program.MethodsThe research team worked across 15 Early Childhood Education and Care (ECEC) centres and in home and community settings. A realist evaluation using mixed methods was undertaken to understand what works, for whom and in what contexts. The development of a logic model provided a structured way to explore how the program was implemented and achieved short, medium and long term outcomes within a complex community setting.Discussion and conclusionKids Together was shown to be a highly effective and innovative model for supporting the inclusion of children with disabilities/additional needs in a range of environments central for early childhood learning and development. The use of a logic model provided a visual representation of the Kids Together model and its component parts and enabled a theory of change to be inferred, showing how a coordinated and collaborative approached can work across multiple environments.     

A visual management tool for program planning,project management and evaluation in paediatric health care

This article describes the development and implementation of a custom-designed Excel-based visual management tool. The tool’s purpose was to support program planning and evaluation by our resource support team within a paediatric health care setting. Our aims in developing it were to 1) establish a streamlined process and supporting tools to efficiently plan and prioritize program directions and activities; 2) track progress; and 3) evaluate and report on our performance, outputs and outcomes. A collaborative approach based on the ADKAR (Awareness, Desire, Knowledge, Ability, Reinforcement) change management model and the LEADS (Lead self, Engage others, Achieve results, Develop coalitions, Systems transformation) leadership framework was used to guide the design and implementation processes. Team members reported high perceived effectiveness and efficiency with respect to the tool’s utility in supporting its proposed aims. A graded approach to building knowledge and skills in using the tool, to individual responsibility for data entry, and to accountability by team members facilitated its successful implementation. Administrative support is important for sustainability and continual improvement of the tool to address changing team needs over time.     

Challenges and successes to the implementation of a zoo conservation-education program

Zoo conservation-education programs have potential to address biodiversity loss, although evidence of their effectiveness to encourage social change is in its infancy. Moreover, how a program is implemented may influence program efficacy, yet there is little evidence of the factors that shape this process in zoos. Accordingly, through a process evaluation of Zoos Victoria’s ‘When Balloons Fly” (WBF) conservation-education program against marine debris, we identified barriers and enablers to the implementation of the program which can be addressed to improve future initiatives. Between April-May 2018, 24 Zoos Victoria professionals completed an online survey focussed on identifying challenges and successes in implementing WBF. Four participants additionally completed a follow-up telephone interview. Themes were identified and organised according to the model of diffusion in service organisations (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). Our results illuminate that features of the organisation (e.g., structure, culture) rather than features of individuals (e.g., skills, motivation) or characteristics of the conservation-education program itself (e.g., messaging), were most influential to implementation success. Based on these findings, attention to the organisational context is critical for promoting and evaluating the success of conservation-education programs and must be given significant attention alongside program characteristics and staff capability to deliver them.     

Taking Stock of Monitoring and Evaluation Arrangements in the Context of Poverty Reduction Strategy Papers: Evidence from 20 Aid‐Dependent Countries in Sub‐Saharan Africa

Shifts in thinking and practice concerning aid have posed serious challenges in monitoring and evaluation (M&E) for both donors and recipients, but progress in reform remains slow. An important first step in any initiative involving M&E capacity development is the diagnosis of the systems' current status. This article presents a diagnostic checklist that captures issues of M&E policy: indicators, data collection and methodology; organisation; capacity‐building; participation of nongovernmental actors; and use. It applies it to a review of the PRSP M&E arrangements of 20 aid‐dependent countries in sub‐Saharan Africa to demonstrate comparative strengths and weaknesses.     

Implementation of a Collaborative Care Initiative for PTSD and Depression in the Army Primary Care System

Collaborative care models hold promise as a means to improve the Military Health System (MHS) to better meet the mental health needs of U.S. military personnel. This article describes an Army-wide collaborative care initiative, RESPECT-Mil (“Re-engineering Systems of Primary Care Treatment for Depression and PTSD in the Military”), aimed at improving care for soldiers with depression and post traumatic stress disorder. Essential to the implementation of RESPECT-Mil has been the use of a centralized coordinating team to facilitate collaborative care practice and policy activities across the multiple levels of the MHS. The article explores the relevance of collaborative care frameworks to the social work profession and concludes with a discussion of future directions for the social work field to promote greater involvement in the integrated care movement.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

Using logic model and visualization to conduct portfolio evaluation

Portfolio evaluation is the evaluation of multiple projects with a common purpose. While logic models have been used in many ways to support evaluation, and data visualization has been used widely to present and communicate evaluation findings, adopting logic models for portfolio evaluation and using data visualization to share findings simultaneously is surprisingly limited in the literature. With the data from a sample portfolio of 209 projects which aims to improve the system of early care and education (ECE), this study illustrated how to use logic model and data visualization techniques to conduct a portfolio evaluation by answering two evaluation questions: “To what extent are the elements of a logic model (strategies, sub-strategies, activities, outcomes, and impacts) reflected in the sample portfolio?” and “Which dominant paths through the logic model were illuminated by the data visualization technique?” For the first question, the visualization technique illuminated several dominant strategies, sub-strategies, activities, and outcomes. For the second question, our visualization techniques made it convenient to identify critical paths through the logic model. Implications for both program evaluation and program planning were discussed.