An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

Assessment of the monitoring and evaluation system of a population-based program: Theory-driven evaluation approach

The purpose of this study was to assess the performance of a monitoring and evaluation system (M&E) for a population-based program and to identify factors influencing its performance. The state health department provided a grant, guide, and training for health districts to implement the program and report quarterly and annually to the M&E. Some health districts did not meet this requirement. We also found that the M&E did not adequately capture the actual work of the districts in communities as related to the work plan. We identified several factors that negatively affected the performances of the M&E. For example, the state and health districts did not have a shared vision on how to develop and implement the program, and what to report to the M&E. We proposed strategies that could improve the timeliness and content of reports to the M&E, as well as its utilization in the future.     

The participatory design of a performance oriented monitoring and evaluation system in an international development environment

This article illustrates the application of the impact monitoring and evaluation process for the design and development of a performance monitoring and evaluation framework in the context of human and institutional capacity development. This participative process facilitated stakeholder ownership in several areas including the design, development, and use of a new monitoring and evaluation system, as well their targeted results and accomplishments through the use of timely performance data gathered through ongoing monitoring and evaluation. The process produced a performance indicator map, a comprehensive monitoring and evaluation framework, and data collection templates to promote the development, implementation, and sustainability of the monitoring and evaluation system of a farmer's trade union in an African country.     

Evaluation capacity building (ECB) interventions and the development of sustainable evaluation practice: An exploratory study

Evaluation capacity building (ECB) is a practice that can help organizations conduct and use evaluations; however, there is little research on the sustainable impact of ECB interventions. This study provides an empirical inquiry into how ECB develops sustained evaluation practice. Interviews were conducted with 15 organizational leaders from non-profits, higher education institutions, and foundations that “bought in” to ECB and were at least six months removed from an ECB contract. The result of this work highlights how sustained evaluation practice developed over time and what these practices looked like in real-world settings. A developmental, iterative cycle for how ECB led organizations to sustain evaluation practice emerged around key components to sustainability. First, leadership supported ECB work and resources were dedicated to evaluation. Staff began to conduct and use evaluation, which led to understanding the benefits of evaluation, and promoted value and buy-in to evaluation among staff. Common barriers and emerging sustainability supports not previously identified by ECB literature—the “personal” factor and ongoing ECB practitioner contact—are described. Practical tips for ECB practitioners to promote sustainability are also detailed.     

A participatory evaluation framework in the establishment and implementation of transdisciplinary collaborative centers for health disparities research

ObjectiveWe describe the formulation and implementation of a participatory evaluation plan for three Transdisciplinary Collaborative Centers for Health Disparities Research funded by the National Institute of Minority Health and Health Disparities.MethodsAlthough different in scope of work, all three centers share a common goal of establishing sustainable centers in health disparities science in three priority areas – social determinants of health, men’s health research, and health policy research.ResultsThe logic model guides the process, impact, and outcome evaluation. Emphasis is placed on process evaluation in order to establish a “blue print” that can guide other efforts as well as assure that activities are being implemented as planned.ConclusionWe have learned three major lessons in this process: (1) Significant engagement, participation, and commitment of all involved is critical for the evaluation process; (2) Having a “roadmap” (logic model) and “directions” (evaluation worksheets) are instrumental in getting members from different backgrounds to follow the same path; and (3) Participation of the evaluator in the leadership and core meetings facilitates continuous feedback.     

Partnership evaluation: An application of a developmental framework to the Governance and Local Development project in Senegal

This study explores the valued-added of USAID/Senegal’s Governance for Local Development (GOLD) project’s integrated approach to working with four sector projects: two in health, one in water and sanitation, and one in nutrition. The study builds on a partnership evaluation framework that identifies practices related to preconditions, factors linked with success, implementation structures and processes, contribution of collaboration to performance, and collaboration outcomes. A survey instrument developed statements of positive practices and asked respondents to agree or disagree with each one. The analysis includes exploration of differences in perception. Respondents generally expressed agreement with the statements in the survey instrument associated with the various dimensions of effective partnerships. All projects mentioned the presence of partnership champions. Problems with joint action (e.g., planning, budgeting, managing) were the most frequently cited constraints. The study confirms most of the hypotheses embedded in the evaluation framework regarding partnership’s potential for creating synergies and improved outcomes and affirms the framework’s usefulness as a developmental evaluation tool. Positive outcomes result from intentional efforts to build relationships, cultivate champions, and design systems and processes that support a partnership culture and contribute to each partners’ success as well as the success of the partnership itself.     

Evaluation of the public health achievements made by projects supported by a federal contract mechanism at the Centers for Disease Control and Prevention (CDC), USA

In 2012, the Centers for Disease Control and Prevention (CDC) established the Achieving Public Health Impact through Research (APHIR) contract mechanism. APHIR provides CDC’s Centers, Institute, and Offices (CIOs) a mechanism that supports multiyear, high impact public health research. Awarded projects supported research on a wide range of topics (e.g., cancer surveillance, HIV education programs, development of biological assays, and evaluation of traumatic brain injury prevention programs) and achieved diverse outcomes (e.g., contribution to the body of knowledge in their field, changes in practice and health service delivery, and capacity building). This article describes how existing impact frameworks and a variety of methods and tools (key informant interviews, online survey, bibliometric analysis, Altmetric and document reviews) were used to identify the outcomes achieved by awarded projects. The approach discussed in this paper can be used to evaluate projects that involve a diversity of activities and outcomes.     

Political and organizational preconditions for application of the standards for educational evaluation : The standards for educational evaluation seen in a wider perspective in relation to policy making and organization

Some of the standards for educational evaluation refer to the political and organizational context in which evaluation projects take place: “conflicting interests,” “political viability,” and various standards which have to do with the identification of the evaluation object and of the most important groups and persons involved. Whereas the Joint Committee on Standards for Educational Evaluation seems to focus on the level of the individual research project, the present authors feel that in addition, contextual influences on evaluations should be analyzed at a higher level of aggregation. Political and organizational factors at the macro level can even be seen as preconditions for the application of the standards at project level. This point of view is illustrated by referring to a study of the functioning of evaluation research within some innovative programs in Dutch education.     

War outside, ceasefire inside: An analysis of the performance appraisal system of a public hospital in a zone of conflict

Our study examines the use of the performance appraisal system at Hebron Public Hospital (Palestine) during the second intifada, started in 2000. The aim of the article is to shed light on the reasons behind the use of performance appraisal systems in organizations operating in zones of conflicts, an area relatively neglected by HR scholars. To create the theoretical fundament we draw on mainstream literature on performance appraisal, contextualizing it to the Middle-Eastern context. From the literature analysis, we identify five guiding logics for the implementation and use of performance appraisal systems (Appendix A). We use a multi-method approach, qualitative and quantitative, to analyze the longitudinal performance evaluation data over the period 2000-2002 for about 250 individuals. These data are complemented with interviews and observations in the field.Our analysis shows that the trends evidenced in the quantitative analysis are similar to trends evident in Western contexts. However, these trends were not the consequence of the same five Western logics found in the literature. The qualitative study allows us to identify two additional logics for making sense of the performance appraisal system at Hebron Public Hospital: the need to find peace within the organization (organizational peacefulness logic); and the need to maintain order through the acceptance of the status quo (dominance logic). These results allow us to draw conclusions for theory and practice of HR management and to identify useful criteria for doing research in areas of conflicts.     

Practices and challenges in educational program evaluation in the Asia-Pacific region: results of a Delphi study

While educational program evaluation has become more important in recent years because of increasing governmental demands for accountability, little is known about the development of and issues in regard to this topic in the Asia-Pacific region. The findings from a Delphi study conducted in a number of relevant countries are provided in this article. Thirty panelists participated in three rounds of iterations to identify current status, challenges, and concerns in the near future. The results should be of value to educational evaluators, researchers, and decision makers.     

Politics and performance: the implications of emerging governance arrangements for urban management approaches and information systems

Challenges to traditional public sector approaches to urban government have had an impact on various aspects of governance and management, including political arrangements, decision making processes, ways of ensuring accountability, the allocation of roles and responsibilities between public sector agencies and other potential service providers and between different levels of government, and the mobilisation and allocation of financial resources for urban development. These are discussed, drawing in particular on recent research on urban poverty and governance in ten cities in the South. The changing approaches to urban government and management have been accompanied by attempts to define criteria by which urban governance arrangements and performance can be assessed, which are summarised and critically evaluated. Finally, the potential value of a particular set of tools (GIS) for emerging approaches to urban governance is assessed. Their potential for improving the information available to all governance actors is recognised. However, because of their cost and technological and managerial demands, as well as the limited effectiveness of spatial planning and inequalities in access to information, plans for their use must be financially, institutionally and politically realistic.     

Network assessment: Design of a framework and indicators for monitoring and self-assessment of a customized gender equality plan in the Mediterranean Engineering Education context

With more than 28 active members, the Mediterranean Engineering Schools Network (RMEI) is the most active network on gender equality in the Mediterranean area. Supported by the HORIZON2020 TARGET project ‘Taking a Reflexive Approach to Gender Equality for Institutional Transformation’, in conceptualizing gender equality learning and system change, the network took a context-specific approach based on a theory of change and other STEM organizational frameworks, to design a self-assessment framework and indicators, considering the systemic view of SDG5 horizontally connected to all SDGs, national context complexity, and transdisciplinary requirements. 95 simple, practical, user-friendly indicators were designed, fitting in the specificities of the Mediterranean-Engineering context. The analysis showed that the network realized its vision, made the ‘passage’ from theory to praxis on gender equality change, effectively developed meaningful processes/structures, formulated a policy statement, built a community of practice and inspired members. It also achieved trustful relationships and inspired outputs, effective communication, sharing of information and resources, and top management commitment. Critical aspects are a) the analysis in depth of issues linked to the existence of gender-based stereotypes and bias in engineering schools of the Mediterranean that entails tackling gender ideologies considering the whole national social system and existing structures; b) sustainability of gender equality structures created at the member institutions with the support of TARGET project which is depending on the willingness of institution’s leaders to continue/advance with gender-sensitive strategies in their institution.     

An evaluation in the choice of inputs and outputs in the efficiency measurement of police forces

The new Labour government has recently instigated an initiative to establish whether English and Welsh police forces should be ranked into groups based on an efficiency measure. A particular problem, however, is that policing includes many inputs and outputs (outcomes) that could potentially be utilised in efficiency models. This paper considers the problems associated with measuring relative police force efficiency given that a vast number of potential indicators (inputs and outputs) must be reduced to a handful to allow feasible estimation. In addition, it discusses the input and output variables utilised in the first ‘official’ analysis of English and Welsh police force efficiency [Demonstration Project, Home Office, London, UK, 2001].     

Articulating influence: Toward a research agenda for interpreting the evaluation of soft power,public diplomacy and nation brands

While the terms soft power, public diplomacy (PD) and nation brands have cemented their place in academic discourse during the early 21st century, the evaluation of these activities has not been given anywhere near the same level of attention. When describing how campaigns are evaluated, scholars tend to make assumptions based on the goals or outputs of an initiative rather than on the basis of reliable, empirical data on its results. Strong positivist tendencies within current scholarship usually lead to evaluation being considered in terms of methodology and best practice, typically with the assumption that certain preferred outcomes will be demonstrable if an ideal model is followed. Most significantly, such approaches seem to underplay the interests and objectives that inform and constrain choices surrounding modes of communication and evaluation. I argue here that PD activities are rarely the product of rational choices about communication options, and nor is PD evaluation the result of applying the “best” methodology. Rather, questions of PD and evaluation practices are bound together in complex organizational and power structures that generate pragmatic responses both to the “problem of influence” and the reporting of results. Through use of the concept of articulation, this article outlines a framework for interpreting evaluation practices from a contextualized perspective, which grasps how and why soft power practices assume certain forms.     

Implications of Information Systems for the Distribution of Authority and Decision Making in Human Service Organizations

No abstract available for this article.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Using the evaluation process as a lever for improving health and healthcare accessibility: The case of HCV services organization in Quebec

BackgroundThe evaluation process can be a lever to improve pathways of access to healthcare. The objective of this article is to show how an evaluation strategy can both contribute to knowledge development and have direct impacts on health services provision. We use the case of hepatitis C (HCV) services organization to illustrate the use and the value of this evaluative approach.MethodInspired by empowerment evaluation, the transformative–participatory approach involved overlapping phases of knowledge development and discussion with stakeholders. We conducted several knowledge development activities to discern the needs of people with HCV, the resources available, and the facilitators and impediments along the care pathway, starting from prevention and screening, all the way through to treatment. Using an overlapping approach allowed us to regularly transfer acquired knowledge back to the participants in the study settings and also to gather their impressions, interpretations, and suggestions during periods of deliberation.ResultsThe knowledge development activities made it possible to document the needs, resources, and experiences of people affected by HCV. In the discussion sessions, viable solutions were identified to improve health and healthcare access for people with HCV and to prioritize certain actions. This project demonstrated that using the evaluation process can enable an instrumental, conceptual use of results and, in fact, can have a transformative impact on services organization.     

Primary healthcare planning,bottleneck analysis and performance improvement: An evaluation of processes and outcomes in a Nigerian context

Operational planning of interventions defines roadmaps, timelines and resources necessary for translating policies into expected health outcomes along the evidence-policy-implementation continuum. However, bottlenecks often hinder the attainment of objectives and the timely delivery of intervention packages leading to sub-optimal performance of health systems. Bottleneck identification, analysis and removal approaches to planning, which requires key stakeholders’ participation, have been recommended to improve health system outcomes in LMICs. This study demonstrates how integration of participatory action research (PAR) within a quality improvement model can help navigate the complexities of health system bottleneck analyses, planning and performance improvement in a Nigerian sub-national context.The study is based on data collected between June 2016 and June 2017, from Chikun LGA in Kaduna State Nigeria. PAR was integrated into a quality improvement model called DIVA (Diagnose-Intervene-Verify-Adjust) applied across selected interventions (eMTCT, Antenatal care, skilled birth attendance, immunization and Integrated Management of Childhood Illnesses). PAR was used to identify and analyse health system bottlenecks, as well as develop, monitor implementation and follow-up on action plans to address them. Evaluations were conducted involving 2 cycles of DIVA. The outputs (bottleneck analysis charts, driver diagrams, operational plans, M/E reports, etc.) from each cycle of the DIVA process were collated and analysed. Bottlenecks identified include availability of human resources for health, availability of health commodities as well as geographical accessibility. These had implications on acceptability and quality of services. Mean Improvements recorded were 20.4%, 14.0% and 10.8% and 11.2%, 7.5%; 5.5% (across eMTCT, maternal health and child health interventions) in the 1 st and 2nd DIVA cycles respectively. This study highlights processes and outcomes of integrating PAR in quality improvement design and operations in health intervention programmes with a focus on health systems strengthening in a Nigerian context. Implementing the DIVA model using a PAR approach may be considered an effective strategy for planning and implementing health interventions in comparable settings.     

Codifying explicit and tacit practitioner knowledge in community social pediatrics organizations: Evaluation of the first step of a knowledge transfer strategy

In this article, we the present the evaluation of the first phase of a KT initiative, the codification of the tacit and explicit knowledge of community social pediatrics (CSP) practitioners. As increasing numbers of professionals began working in the CSP field, it became necessary to capture the know-how to make it better understood. This first phase of the initiative took advantage of one of the KM stages, the codifying process that is absent from KT models. The evaluation aim to answer the question: “What is the CSPs practitioners level of familiarity with the codification document, as well as their perceptions of its utility and its contribution to the understanding of the CSP model”. The results, show that most respondents had consulted the codification document and were able to describe in detail parts of it. The results also show that they considered it to be a useful and necessary tool to guide practices and helpful in understanding the model. The results of this evaluation is paving the way for the next steps of the KT initiative.