An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

A comprehensive health service evaluation and monitoring framework

ObjectiveTo develop a framework for evaluating and monitoring a primary health care service, integrating hospital and community services.MethodA targeted literature review of primary health service evaluation frameworks was performed to inform the development of the framework specifically for remote communities. Key principles underlying primary health care evaluation were determined and sentinel indicators developed to operationalise the evaluation framework. This framework was then validated with key stakeholders.ResultsThe framework includes Donabedian's three seminal domains of structure, process and outcomes to determine health service performance. These in turn are dependent on sustainability, quality of patient care and the determinants of health to provide a comprehensive health service evaluation framework. The principles underpinning primary health service evaluation were pertinent to health services in remote contexts. Sentinel indicators were developed to fit the demographic characteristics and health needs of the population. Consultation with key stakeholders confirmed that the evaluation framework was applicable.ConclusionData collected routinely by health services can be used to operationalise the proposed health service evaluation framework. Use of an evaluation framework which links policy and health service performance to health outcomes will assist health services to improve performance as part of a continuous quality improvement cycle.     

The participatory design of a performance oriented monitoring and evaluation system in an international development environment

This article illustrates the application of the impact monitoring and evaluation process for the design and development of a performance monitoring and evaluation framework in the context of human and institutional capacity development. This participative process facilitated stakeholder ownership in several areas including the design, development, and use of a new monitoring and evaluation system, as well their targeted results and accomplishments through the use of timely performance data gathered through ongoing monitoring and evaluation. The process produced a performance indicator map, a comprehensive monitoring and evaluation framework, and data collection templates to promote the development, implementation, and sustainability of the monitoring and evaluation system of a farmer's trade union in an African country.     

Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

The Influence of Family and Community Social Capital on Educational Achievement*

Abstract It is increasingly recognized that families and communities are important in helping youths develop the knowledge and skills they need to obtain technologically sophisticated jobs, which are an emerging part of the global economy. In this paper we adopt social capital as a framework for examining the influence of family and community on promoting educational achievement among public school students. We explore more fully the role of community social capital in influencing educational performance beyond that attributed to family social capital. Using data from the National Education Longitudinal Survey (NELS), we find that both process and structural attributes of family social capital are key factors affecting high school students' educational achievement. Process and structural attributes of community social capital also help youths to excel, though they contribute less strongly to achievement. These findings suggest that policies designed to promote educational achievement must extend beyond the school and must seek to strengthen social capital in the family and the community.     

Multimethod strategy for assessing program fidelity: the national evaluation of the revised G.R.E.A.T. program

This study reports the results of the process evaluation component of the Process and Outcome Evaluation of the Gang Resistance Education and Training (G.R.E.A.T.) program. The process evaluation consisted of multiple methods to assess program fidelity: (a) observations of G.R.E.A.T. Officer Trainings (G.O.T); (b) surveys and interviews of G.R.E.A.T.-trained officers and supervisors; (c) surveys of school personnel; and (d) 'on-site,' direct observations of officers delivering the G.R.E.A.T. program in the study sites. Results illustrate a high level of program fidelity, providing greater confidence in any subsequent outcome results.     

Middle school reform through data and dialogue. Collaborative evaluation with 17 leadership teams

This report describes a 2-year, longitudinal study of one school district's effort to link site-based, collaborative evaluation with formal, centralized program evaluation. Participants formed a research team in partnership with a local university. Team members assisted leadership teams in identifying issues for informal, site-based assessments and then used the information to monitor progress toward established goals. Participants collaborated in designing and conducting informal assessments of student achievement, school climate and safety, discipline, and parent involvement. Leadership teams used these data in developing their school improvement plans. Researchers and administrators used these data to revise the districtwide survey. At the end of the 2-year cycle, analysis of school improvement plans showed that collaborative evaluation is creating a connection between dialogue and data. These 17 middle schools are approaching school improvement in a more integrated fashion by actively involving key stakeholders (students, parents, and teachers) in the evaluation process.     

Multidisciplinary eHealth survey evaluation methods

This paper describes the development process of an evaluation framework for describing and comparing web survey tools. We believe that this approach will help shape the design, development, deployment, and evaluation of population-based health interventions. A conceptual framework for describing and evaluating web survey systems will enable the developers of health interventions to identify, select, and use the most appropriate and relevant survey tools. The primary contributors included survey methodologists, computer scientists, informaticians, technical communications specialists, educators, evaluators, clinical practitioners, and health services researchers, who together developed a comprehensive ontology to describe the electronic survey domain. This paper will describe the survey ontology development process and detail lessons learned in the creation of the web survey system evaluation framework.     

Development of a fostering purpose intervention

Evolutionary Evaluation is a framework for understanding the evolution of programs. Just as programs grow and develop, so can standalone interventions. This paper focused on building and evaluating a purpose intervention to scaffold adolescents and young adults’ search for purpose. Four studies were conducted at each phase of the intervention’s lifespan to assess if modifications were needed and to test the viable validity of using online activities to increase both the search for and the discovery of a purpose in life. These studies can be viewed through the lens of the Evolutionary Evaluation framework. Study 1 evaluated the intervention at its initiation phase, which consisted of developing and testing the intervention activities. Study 2 tested the intervention during the development phase to determine which activities most effectively increased rates of purpose. Study 3 evaluated the intervention in its stability phase by experimentally testing all activities together in a full curriculum via Amazon’s Mechanical Turk. Finally, study 4 assessed the dissemination of the intervention by testing the full curriculum with over 500 high school and college students. These studies suggest that the fostering purpose intervention cultivated purpose among adolescents and young adults. Additionally, the success of the intervention demonstrates that the Evolutionary Evaluation framework may be an effective model with which to develop an intervention, from initiation through dissemination phases.     

Towards an assessment for organizational participatory research health partnerships: A systematic mixed studies review with framework synthesis

Within the health sciences, organizational participatory research (OPR) is defined as a blend of research and action, in which academic researchers partner with health organization members. OPR is based on a sound partnership between all stakeholders to improve organizational practices. However, little research on the evaluation of OPR health partnership exists. This systematic mixed studies review sought to produce a new theoretical model that structures the evaluation of the OPR processes and related outcomes of OPR health partnerships. Six bibliographic databases were searched together with grey literature sources for OPR health partnership evaluation questionnaires. Six questionnaires were included, from which a pool of 95 OPR health partnership evaluation items were derived. The included questionnaires were appraised for the quality of their origin, development and measurement properties. A framework synthesis was performed using an existing OPR framework by organizing questionnaire items in a matrix using a hybrid thematic analysis. This led to our proposed Organizational Participatory Research Evaluation Model (OPREM) that includes three axes, Trust, Collective Learning and Sustainability (with specific dimensions) and 95 items. This model provides information to help stakeholders comprehensively structure the evaluation of their partnerships and subsequent improvement; thus, potentially helping to improve health organization practices.     

Facilitating evaluations of innovative,competence-based assessments: Creating understanding and involving multiple stakeholders

Schools are held more responsible for evaluating, quality assuring and improving their student assessments. Teachers’ lack of understanding of new, competence-based assessments as well as the lack of key stakeholders’ involvement, hamper effective and efficient self-evaluations by teachers of innovative, competence-based assessments (CBAs). While evaluating two CBAs in Agricultural Vocational Education and Training institutions, two interventions in the evaluation process aimed to tackle these problems were examined: (1) starting with explicating the CBA in the teacher team using a concrete explication format and (2) qualitatively involving key stakeholders (i.e., teachers, students and employers) in the evaluation of the CBA through mixed-group interviews. Quantitative and qualitative analysis, as well as stakeholders’ perceptions are used to find indications for the added value of these interventions for evaluation and further improvement of the CBAs. Results show that external facilitation is needed to make both interventions work.However, under this condition, explicating the CBA led to more complete, concrete and shared understandings of the actual CBA among teachers and mixed-groups interviews resulted in more concrete and elaborate evaluations of the CBAs’ quality and more ideas for improvement. Both interventions can facilitate building up elaborate, more valid and concrete arguments for CBA quality in self-evaluations, certainly in the case of evaluating innovative assessments. Lessons learned will provide guidelines for incorporating the interventions into other evaluations of innovative programs.     

Functional capacity evaluation: An essential component of South African occupational therapy work practice services

South African occupational therapists have been involved in the delivery of work practice services since the inception of the profession in this country. Initial development in this area of practice appeared slow but following the acceptance of South Africa's first democratic Constitution in 1996, many opportunities for service delivery were facilitated by the implementation of disability equity legislation. One of the key services offered by practitioners is Functional Capacity Evaluation (FCE). Although the outcomes of FCE are many, the process in essentially the same. This entails: referral, the initial interview, evaluation of performance components and ADL, vocational evaluation, obtaining collateral and the closure interview, the work visit, hypothesis testing and reporting. This article reports on the Functional Capacity Evaluation process as followed by South African occupational therapists in various contexts.     

Analysing facilitating and hindering factors for implementing gender equality interventions in R&I: Structures and processes

This article analyses the facilitating and hindering factors that have affected the implementation of gender equality interventions in research and innovation in Europe. It applies the evaluation framework developed in the EFFORTI project that recognizes the complexity of evaluating gender equality interventions in R&I, the importance of factoring in context to any sound evaluation as well as the need to distinguish between the design and implementation of interventions in evaluations. It is based on the analysis of 19 empirical case studies carried out throughout Europe and focuses on those structural and procedural factors that have either facilitated or hindered the implementation process of these interventions. Findings include how the governance framework; top-management commitment; bottom-up participation; framing synergies with other initiatives, strategies for tackling resistance; resources; sustainability of actions; gender competence, experience and knowledge and transparency, targets, standards and monitoring; and accessible data and information all contributed to the successful implementation of the interventions.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts.The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts. The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

Examining the link between program implementation and behavior outcomes in the lifestyle education for activity program (LEAP)

Lifestyle Education for Activity Program (LEAP) was a comprehensive, school-based intervention designed to promote physical activity in high school girls. The intervention focused on changes in instructional practices and the school environment to affect personal, social, and environmental factors related to physical activity. Multiple process evaluation tools and an organizational assessment tool were developed to monitor program implementation from a framework called the LEAP essential elements, which characterized complete and acceptable intervention delivery; secular trends were also monitored. Using process data, LEAP intervention schools were categorized into low- and high-implementing groups and compared with control schools on nine essential elements assessed at the organizational level. The Wilcoxon scores test revealed that low- and high-implementing intervention, and control schools differed significantly on two of nine administrator-reported organizational-level components: having a physical activity team and having a faculty-staff health promotion program. A mixed-model analysis of covariance indicated that, compared to control schools, a greater percentage of girls in high-implementing schools reported engaging in vigorous physical activity. Process evaluation can be used to understand the relationship between level of implementation and successful program outcome.     

Constructing a strategy map for banking institutions with key performance indicators of the balanced scorecard

This study presents a structural evaluation methodology to link key performance indicators (KPIs) into a strategy map of the balanced scorecard (BSC) for banking institutions. Corresponding with the four BSC perspectives (finance, customer, internal business process, and learning and growth), the most important evaluation indicators of banking performance are synthesized from the relevant literature and screened by a committee of experts. The Decision Making Trial and Evaluation Laboratory (DEMATEL) method, a multiple criteria analysis tool, is then employed to determine the causal relationships between the KPIs, to identify the critical central and influential factors, and to establish a visualized strategy map with logical links to improve banking performance. An empirical application is provided as an example. According to the expert evaluations, the three most essential KPIs for banking performance are customer satisfaction, sales performance, and customer retention rate. The DEMATEL results demonstrate a clear road map to assist management in prioritizing the performance indicators and in focusing attention on the strategy-related activities of the crucial indicators. According to the constructed strategy map, management could better invest limited resources in the areas that need improvement most. Although these strategy maps of the BSC are not universal, the research results show that the presented approach is an objective and feasible way to construct strategy maps more justifiably. The proposed framework can be applicable to institutions in other industries as well.     

Building School,Family, and Community Partnerships: Developing a Theoretical Framework

School, family, and community partnerships are among the suite of strategies advocated for improved school outcomes in which social workers can play a valuable role. Such partnerships are complex to implement in practice and there is little systematic research to guide the practitioner. This study evaluated an early intervention partnership program involving school, family, community, and philanthropic partners implemented in two primary schools in rural Victoria, Australia. The findings highlighted challenges in the process of evaluation and program implementation and in the opportunities this provided to identify key enabling factors. These involved having: a shared vision and aims; democratic governance; a supportive policy and organisational environment including external funding; workers with skills and clear roles; activities for project momentum; and ongoing review and evaluation. These factors can be used by social workers and others working with schools as a valuable conceptual framework to facilitate school, family, and community partnership processes.     

Evaluating the design and implementation fidelity of an adapted Plan-Do-Study-Act approach to improve health system performance in a Nigerian state

BackgroundThe Plan-Do-Study-Act (PDSA) cycle is fundamental to many quality improvement (QI) models. For the approach to be effective in the real-world, variants must align with standard elements of the PDSA. This study evaluates the alignment between theory, design and implementation fidelity of a PDSA variant adapted for Nigeria’s health system performance improvement.MethodsAn iterative consensus building approach was used to develop a scorecard evaluating new conceptual indices of design and implementation fidelity of QI interventions (design and implementation index, defects and gaps) based on Taylor’s theoretical framework.ResultsDesign (adaptation) scores were optimal across all standard features indicating that design was well adapted to the typical PDSA. Conversely, implementation fidelity scores were only optimal with two standard features: prediction-based test of change and the use of data over time. The other features, use of multiple iterative cycles and documentation had implementation gaps of 17 % and 50 % respectively.ConclusionThis study demonstrates how both adaptation and implementation fidelity are important for success of QI interventions. It also presents an approach for evaluating other QI models using Taylor’s PDSA assessment framework as a guide, which might serve to strengthen the theory behind future QI models and provide guidance on their appropriate use.