Facilitating commitment, consensus, credibility, and visibility through collaborative foreign assistance project evaluations

The four key organizations involved in foreign assistance projects—those of the donor, technical advisors, host country counterparts, and beneficiaries—hold systematically different stakes in the project implementation process. As a result, each tends to reject or mistrust the goals of the other participants and resists collaborating with them during the project evaluation process. Yet, several potentially beneficial outcomes of collaboration for each of these organizations can be demonstrated, including enhanced commitment, consensus, credibility, and visibility. The evaluator plays a key role as facilitator of such a mutually beneficial process of collaboration by convincing each stakeholder that collaboration will provide the forum to promote rather than to sabotage individual stakes. In some cases, however, modified collaborative approaches may be necessary in order to bring to bear the issues and concerns of all stakeholders.     

Improving the quality of care of veterans with diabetes. A simulation intervention for psychiatric nurses

The primary objective of this study was to provide and evaluate strategies beneficial in preparing psychiatric nurses to exercise clinical judgment needed for adequate assessment and care of veterans with diabetes. Psychiatric nurses completed a pretest evaluation of their knowledge of care for veterans with diabetes. During the intervention stage, a human patient simulator was used to create an interactive scenario for participants to propose a diagnosis and administer treatment. Clinical judgment attributes were evaluated during the simulation by instructors using a rubric. Participants then completed a posttest evaluation. In addition, data about medical transfer rates for veterans with diabetes were provided 1 month before and 1 month after project implementation. All participants (N=20) demonstrated an increase on posttest scores. Medical emergencies decreased from 55% to 20% after project implementation, indicating improvement in the quality of care of veterans with diabetes.     

Systematic dissemination of a preschool physical activity intervention to the control preschools

For public health interventions to have a meaningful impact on public health, they must be disseminated to the wider population. Systematic planning and evaluation of dissemination efforts can aid translation from experimental trials to larger dissemination programs. The Study of Health and Activity in Preschool Environments (SHAPES) was a group-randomized intervention trial conducted in 16 preschools that successfully increased the physical activity of preschool age children. Following the completion of the research study protocol, the intervention was abbreviated, modified and implemented in four preschools that participated as control preschools in the original research study. The purposes of the current study were to describe the process of refining the intervention for dissemination to the control preschools, and to assess the acceptability of the resulting abbreviated intervention delivery. Five overarching behavioral objectives, informed by process evaluation, data from the original trial and collaboration with intervention teachers, were used to guide the implementation. Teachers in the dissemination classrooms reported high levels of acceptability, potential for sustainability of the program, and positive results in knowledge, skills, and child outcomes. Researchers can include a systematic approach to dissemination of effective intervention elements to the control participants in experimental studies to inform future dissemination efforts and begin to bridge the dissemination gap.     

Stakeholder perceptions of contextual factors related to PBIS implementation in high need schools

Positive behavior intervention and supports (PBIS) is a thoroughly researched three-tiered framework for preventing and treating challenging behavior in schools. While more than 20 years of research contribute to the literature base, less is known about PBIS implementation in high-needs school settings, particularly those that serve students predominantly from low-income families and majority minority student populations across urban and rural districts. This study investigated the critical contextual factors of PBIS implementation from the stakeholder perspective. Specifically, we surveyed 161 participants including teachers, school staff members, and administrators who were PBIS team members and had received technical assistance in PBIS implementation. The completed surveys were analyzed with participants and their schools coded for district type, Title I funding, and position in order to determine how the perceptions of issues such as buy-in, administrator support, PBIS in secondary school settings, and poverty affect PBIS implementation and team needs. The factor analysis and resulting structure along with statistically significant results for each factor are presented along with implications.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Stakeholder driven indicators for eHealth performance management

The goal of the present article is to compile a corpus of indicators of eHealth development evaluation that would essentially reflect stakeholder approaches and complement technical indicators of assessment of an eHealth system. Consequently, the assessment of the development of an eHealth system would reflect stakeholder approaches and become an innovative solution in attempting to improve productivity of IT projects in the field of health care. The compiled minimum set of indicators will be designed to monitor implementation of the national eHealth information system. To ensure reliability of the quality research, the respondents were grouped in accordance to the geographical distribution and diversity of the levels and types of the represented jobs and institutions. The applied analysis implies several managerial insights on the hierarchy of eHealth indicators. These insights may be helpful in recommending priority activities in implementation of an eHealth data system on the national or international level. The research is practically useful as it is the first to deal with the topic in Lithuania and its theoretical and practical aspect are particularly relevant in implementation of an eHealth data system in Lithuania. The eHealth assessment indicators presented in the article may be practically useful in two aspects: (1) as key implementation guidelines facilitating the general course of eHealth system development and (2) as a means to evaluate eHealth outcomes.     

Social Studies in the Modern Era: A Case Study of Effective Teachers' Use of Literacy and Technology

The purpose of this research project was to highlight the practices and philosophies of two effective—but different—social studies teachers who balance the demands of teaching in the modern era while honoring their own philosophies for teaching social studies. This project was ground in the theoretical framework provided by TPACK and used a case study methodology for its research design. While the pedagogical content knowledge of the participants was strong and technology was used abundantly for instructional purposes, this research raised questions regarding how teachers can most effectively use technology to enhance instruction by helping students conceptualize content knowledge and apply their learning in new ways.     

Post occupancy evaluation of primary schools in Saudi Arabia

The physical school environment has been shown to be important in helping children fulfill their academic potential and in providing appropriate working conditions for staff. However, few tools have been developed that enable multi stakeholder consultation which takes into account the opinions of young students. In Saudi Arabia there has been widespread investment in schools, but few guidelines have been provided to assist design or continuous evaluation. A Post Occupancy Evaluation (POE) method was developed and used to evaluate three international primary schools in Saudi Arabia. The methods identified weaknesses in the three schools and differences in responses from the three groups consulted (children, teachers and parents). Conclusions drawn from the study are that greater efforts need to be made to draw together research about how school facilities can support teaching and learning, increase effectiveness and levels of satisfaction. POE, when used in conjunction with checklists could be used as a means of driving up standards of educational facilities.     

The problems and prospects in the teaching of mixed methods research

There are pedagogical challenges USA students and instructors face within mixed methods classrooms. Instructors of mixed methods are often self-taught, lacking adequate training in both qualitative and quantitative approaches to research. Students are not often trained in both research approaches. These dual training gaps can result in deep pedagogical issues compromising students’ ability to fully understand mixed methods research praxis and leaving teachers feeling ill equipped to address students’ learning concerns. To tackle the myriad of challenges confronted in the mixed methods classroom requires structural changes to the current way graduate training programs in social research methods are organized and taught. Developing a team-based teaching approach to mixed methods research that provides students with instructors who have the requisite qualitative and quantitative knowledge can serve as a pedagogical model that can begin at least to address the current methods and methodological skills gap in the teaching of mixed methods research.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.     

Do public relations professionals understand corporate governance issues well enough to advise companies on stakeholder relationship management?

The King Reports on Governance for South Africa are internationally respected for proposing integrated reporting in a triple bottom-line business context to improve corporate governance. The most recent report, King III, views stakeholder relationship management as a key tenet of corporate governance. This raises the question whether public relations professionals (PRPs) understand the principles of corporate governance well-enough to inform and guide organizations on the management of stakeholder relationships. The views of senior PRPs at selected top performing companies were elicited on their practice of stakeholder relationship management in line with corporate governance principles. It was found that their knowledge on corporate governance was limited, although they recognized its importance. They propose six role functions for managing stakeholder relationships on a strategic or managerial level; these role functions are aligned with the King III principles on stakeholder relationships. The following three were of particular importance: developing a corporate strategy, giving advice to senior management and managing crisis communication. Different terminologies were used to build relations with stakeholders and different approaches were used to profile stakeholders. It is recommended that organizations leverage their public relations functions on strategic and managerial levels in support of their corporate governance efforts. In turn, PRPs are encouraged to ensure a deep knowledge on corporate governance issues when counseling senior management on building stakeholder relationships.     

Evaluating systems change in a juvenile justice reform initiative

Evaluating comprehensive, interagency initiatives to reform human services systems presents substantial challenges to traditional research models. Outcomes are observed at the system level rather than the individual level, and the validity of study results may be challenged on a variety of dimensions, particularly small sample sizes and measurement error. We report the results of a cross-site evaluation of the first phase of Reclaiming Futures, a five-year effort to improve services and interventions for justice-involved youth in 10 communities across the United States. We present the evaluation findings and discuss implications for similar research endeavors. The methods used in this study may be useful for other system-level evaluation efforts. A social network analysis analyzes changes in the size, density, and cohesiveness of stakeholder networks. Bivariate models test for associations between Reclaiming Futures implementation and key stakeholder perceptions of effective system change. Multivariate models are used to explore selection effects on item response. In general, we find positive correlations between successful implementation of Reclaiming Futures and indicators of effective and efficient system change.     

Building School,Family, and Community Partnerships: Developing a Theoretical Framework

School, family, and community partnerships are among the suite of strategies advocated for improved school outcomes in which social workers can play a valuable role. Such partnerships are complex to implement in practice and there is little systematic research to guide the practitioner. This study evaluated an early intervention partnership program involving school, family, community, and philanthropic partners implemented in two primary schools in rural Victoria, Australia. The findings highlighted challenges in the process of evaluation and program implementation and in the opportunities this provided to identify key enabling factors. These involved having: a shared vision and aims; democratic governance; a supportive policy and organisational environment including external funding; workers with skills and clear roles; activities for project momentum; and ongoing review and evaluation. These factors can be used by social workers and others working with schools as a valuable conceptual framework to facilitate school, family, and community partnership processes.     

The inherently political nature of program evaluators and evaluation research

This paper discusses the importance of addressing the political dynamics inherent in any evaluation research process. The theoretical orientation contributing to the denial and avoidance of politics is articulated and contrasted with one which posits politics as normative and useful. A connection is drawn between stakeholder and case study approaches to evaluation. It is argued that both are effective routes to evaluations which facilitate decision making. A case example of a study of Community Development Block Grant agencies is used to detail problems accompanying avoidance of politics. Guidelines for effectively addressing and using politics are offered.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

From plan to action: Adapting evaluation to serve the developmental needs of a newly-funded multidisciplinary research center

The Southeastern Coastal Center for Agricultural Health and Safety (SCCAHS) is one of many newly-funded federal research centers, housing five multidisciplinary research projects and seven pilot projects, and serving a multi-state region. In the early stages of such a complex project, with multiple teams separated by geography and disciplines, the evaluation program has been integral in connecting internal and external stakeholders at the center and project levels. We used a developmental evaluation (DE) framework to respond to the complex political environment surrounding agricultural health and safety in the southeast; to engage external stakeholders in guiding the center’s research and outreach trajectories; to support center research teams in a co-creation process to develop logic models and tailored indicators; and to provide timely and feedback within the center to address communications gaps identified by the evaluation program. By using DE principles to shape monitoring and evaluation approaches, our evaluation program has adapted to the dynamic circumstances presented as our center’s progress has been translated from a plan in a grant proposal to implementation.     

Development of an evaluation framework for publicly funded R&D projects: The case of Korea's Next Generation Network

For decades, efforts have been made globally to measure the performance of large-scale public projects and to develop a framework to perform such measurements due to the complexity and dynamics of R&D and stakeholder interests. Still, limitations such as the use of a simply modified model and the lack of a comprehensive viewpoint are prevalent in existing approaches. In light of these research gaps, this study suggests a practical model to evaluate the performance of large-scale and publicly funded projects. The proposed model suggests a standard matrix framework of indices that evaluates the performance of particular elements in an industrial ecosystem in vertical categories and the economic and technological outcomes of those elements in horizontal categories. Based on the application of a balanced scorecard, this study uses mixed methodologies such as social network analysis, inter-industry analysis, and the analytic hierarchy process. Finally, the model evaluates the performance of Korea's Next Generation Network project as a case study.     

An evaluation capacity building toolkit for principal investigators of undergraduate research experiences: A demonstration of transforming theory into practice

This paper describes the approach and process undertaken to develop evaluation capacity among the leaders of a federally funded undergraduate research program. An evaluation toolkit was developed for Computer and Information Sciences and Engineering¹ Research Experiences for Undergraduates² (CISE REU) programs to address the ongoing need for evaluation capacity among principal investigators who manage program evaluation. The toolkit was the result of collaboration within the CISE REU community with the purpose being to provide targeted instructional resources and tools for quality program evaluation. Challenges were to balance the desire for standardized assessment with the responsibility to account for individual program contexts. Toolkit contents included instructional materials about evaluation practice, a standardized applicant management tool, and a modulated outcomes measure. Resulting benefits from toolkit deployment were having cost effective, sustainable evaluation tools, a community evaluation forum, and aggregate measurement of key program outcomes for the national program. Lessons learned included the imperative of understanding the evaluation context, engaging stakeholders, and building stakeholder trust. Results from project measures are presented along with a discussion of guidelines for facilitating evaluation capacity building that will serve a variety of contexts.