IMPLEMENTATION OF PARENT MANAGEMENT TRAINING AT THE NATIONAL LEVEL: THE CASE OF NORWAY

This article describes early aspects of the nationwide implementation of an evidence‐based program (EBP) in Norway and the design for studying program fidelity over time. The EBP is the Parent Management Training, Oregon Model (PMTO). The project is a combination of a ‘top down’ initiative at the governmental and ministry level, and a ‘bottom up’ initiative from practitioners seeking effective interventions in the prevention and remediation of behavior problems in children and youth. The main components of the implementation strategy were to (a) establish a national implementation and research center; (b) provide for regional and local participation at county and municipal levels; (c) establish a comprehensive therapist recruitment, training, and maintenance program; (d) create a network for collaboration, supervision, and quality control; (e) conduct clinical outcome research; and (f) study the implementation process. Following the training of three successive generations of PMTO specialists, a study was designed to examine how individual, family, interventionist, organizational, and community characteristics influence program adoption and implementation, as well as how these factors impact upon how PMTO specialists provide the intervention to families with competent adherence to the model. Within the framework of the collaborative North American and Norwegian project, challenges and facilitators in the process of the continental crossing of the PMTO model are discussed.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Design of a collaborative monitoring and evaluation system for a community-based nutrition project in rural Bangladesh

Poor diet and undernutrition are common among children living in Bangladesh. To promote appropriate complementary feeding of young children, an economic development (ED) program involving income-generating asset transfer was implemented alongside a social and behavior change (SBC) program. This paper introduces a collaborative monitoring and evaluation (M&E) system in which diverse collaborators (“research group”, “implementation team”, and “coordinators”) facilitate M&E data acquisition by leveraging their comparative advantages. The implementation team built a monitoring system to track the ED (n = 2960) and SBC participants (n=∼10,000) over 12 months. Based on the baseline design and the monitoring records, the collaborators planned an impact evaluation introducing a quasi-experimental design using two cross-sectional surveys and a prospective cohort survey of child feeding and nutritional status. Using various data sources generated from the M&E system, the collaborators will also reveal the program impact pathway through which each intervention component is delivered, received, and utilized alongside the context-specific facilitators and barriers, including the programs’ uptake. The collaborative M&E system enables the sharing of program goals, strengthens collaborators' commitment to the program, and extends the understanding of the program's progress and evaluation activities.     

Partnership evaluation: An application of a developmental framework to the Governance and Local Development project in Senegal

This study explores the valued-added of USAID/Senegal’s Governance for Local Development (GOLD) project’s integrated approach to working with four sector projects: two in health, one in water and sanitation, and one in nutrition. The study builds on a partnership evaluation framework that identifies practices related to preconditions, factors linked with success, implementation structures and processes, contribution of collaboration to performance, and collaboration outcomes. A survey instrument developed statements of positive practices and asked respondents to agree or disagree with each one. The analysis includes exploration of differences in perception. Respondents generally expressed agreement with the statements in the survey instrument associated with the various dimensions of effective partnerships. All projects mentioned the presence of partnership champions. Problems with joint action (e.g., planning, budgeting, managing) were the most frequently cited constraints. The study confirms most of the hypotheses embedded in the evaluation framework regarding partnership’s potential for creating synergies and improved outcomes and affirms the framework’s usefulness as a developmental evaluation tool. Positive outcomes result from intentional efforts to build relationships, cultivate champions, and design systems and processes that support a partnership culture and contribute to each partners’ success as well as the success of the partnership itself.     

Breastfeeding interventions in Kansas: A qualitative process evaluation of program goals and objectives

Across the state of Kansas, eighteen public health departments received funding through the 2011 Breastfeeding Grant Initiative to start a breastfeeding intervention. The main objective of this study was to evaluate the progress toward program goals and objectives. This study was a process evaluation. Qualitative data were collected from recipient health departments at two time-points during the program year. Structured, open-ended questions were asked through telephone interviews. This study examined: (1) progress toward program goals and objectives, (2) problems encountered during implementation, and (3) evaluation measures employed to assess program impact. All health departments reported making significant progress toward program goals and objectives and reported successful collaboration with other healthcare providers. The use of breast pumps, educational classes, and professional training of staff were reported as providing the best outcome in the promotion of breastfeeding. The majority of respondents did not measure program impact. From a public health perspective, it is important that infants receive breast milk for the first six months of life. It appears that goals and objectives set a priori guided health departments with the administration of their breastfeeding program. Results may be used to enhance and sustain delivery of breastfeeding support programs in Kansas communities.     

Exploring the Potential of Shadow Controls in the Evaluation of Children’s Services

This article describes and presents a framework for an under‐used evaluation technique in the context of an evaluation of a programme for disaffected young people. Shadow controls—the use of expert judgement to estimate the success of a programme—are often dismissed in research design as an unreliable form of comparison, but can be useful in situations where there is limited scope for a control group or to enhance the causal inference attributable to non‐experimental evaluations. The exercise described uses a practice tool as a structure for making predictions about the situations of the young people on the programme, assuming they do not receive an intervention. These predictions (shadow controls) are then compared to outcome data for the young people at the end of the programme. The results of the exercise provide some important messages about the programme’s effectiveness and the potential for strengthening non‐experimental evaluation methods. The article also discusses how the method can usefully inform evaluations of social programmes and encourage agency and user collaboration.     

Monitoring and evaluation design of Malawi’s Right Foods at the Right Time nutrition program

Child stunting is a public health problem in Malawi. In 2014, the Government of Malawi launched the Right Foods at the Right Time (RFRT) program in Ntchisi district delivering nutrition social and behavior change communication, a small-quantity lipid-based nutrient supplement to children 6–23 months, and nutrition sensitive activities. Monitoring and evaluation (M&E) systems are key aspects of successful program implementation. We describe these and the methodology for an impact evaluation that was conducted for this program. Two monitoring systems using traditional and electronic platforms were established to register and track program delivery and processes including number of eligible beneficiaries, worker performance, program participation, and to monitor input, output, and outcome indicators. The impact evaluation used comparative cross-sectional and longitudinal designs to assess impact on anthropometric and infant and young child feeding outcomes. Three cross-sectional surveys (base-, mid-, and end-line) and two longitudinal cohorts of children followed in 6-month intervals from 6 to 24 months of age, were conducted in sampled households in the program and a neighboring comparison district. Additional M&E included qualitative studies, a process evaluation, and a cost-effectiveness study. The current paper describes lessons from this program’s M&E, and demonstrates how multiple implementation research activities can inform course-correction and program scale-up.     

A visual management tool for program planning,project management and evaluation in paediatric health care

This article describes the development and implementation of a custom-designed Excel-based visual management tool. The tool’s purpose was to support program planning and evaluation by our resource support team within a paediatric health care setting. Our aims in developing it were to 1) establish a streamlined process and supporting tools to efficiently plan and prioritize program directions and activities; 2) track progress; and 3) evaluate and report on our performance, outputs and outcomes. A collaborative approach based on the ADKAR (Awareness, Desire, Knowledge, Ability, Reinforcement) change management model and the LEADS (Lead self, Engage others, Achieve results, Develop coalitions, Systems transformation) leadership framework was used to guide the design and implementation processes. Team members reported high perceived effectiveness and efficiency with respect to the tool’s utility in supporting its proposed aims. A graded approach to building knowledge and skills in using the tool, to individual responsibility for data entry, and to accountability by team members facilitated its successful implementation. Administrative support is important for sustainability and continual improvement of the tool to address changing team needs over time.     

How do new members affect the relationship between principal investigator’s network position and academic output of granted funds?

The paper focuses on how to improve academic output of a granted fund when the adding of new members changes principal investigator’s network position. This objective is refined by exploring how new members affect the relationship between principal investigator’s network position and the academic output of granted funds, and whether this effect is similar in high-output and low-output collaboration networks. New members are divided into two groups, namely, international collaboration and inbound mobility. Using negative binomial regression on research funds supported by the National Natural Science Foundation of China, the results indicate that new members indeed have moderating effects on the relationship between principal investigator’s network position and the academic output of granted funds. Moreover, we find that this moderating effect in high-output collaboration networks is larger than that in low-output collaboration networks. Our findings provide practical implications for the decision makers to design funding planning and for the principal investigators to improve academic output.     

Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.     

Building stronger weak ties among a diverse pool of emergent nonprofit leaders of color

This article explores theoretical underpinnings of social capital and strong and weak ties as they relate to relationships within, between, and among six cohorts of promising nonprofit leaders of color in a northwestern U.S. city. Using mixed methods, including network analyses, it considers the impact and potential of a university‐community collaboration to deliver a program to promising emergent leaders nominated for their talent and commitment to the work of their community‐based organizations, often in competition for limited resources. The article considers the program's implicit effect on relationships and reveals intersector connections among alumni. Findings show substantial alumni movement among organizations, increasing the potential for collaboration and mentoring relationships. We conclude with recommendations for programs seeking long‐term impact and discuss a series of unanticipated findings and the questions they engender, many critical as communities and organizations become more diverse and social needs do not diminish.     

Will the children use it?—A RE-AIM evaluation of a local public open space intervention involving children from a deprived neighbourhood

Knowledge on how to improve public open spaces in deprived neighbourhoods to increase active living among children is scarce and comprehensively evaluated public open space interventions are needed. Firstly, the aim was to explore if involving 39 local fifth-grade children (10–11 years old) from a deprived neighbourhood in creating playable installations in a public open space influenced their use of this space. Secondly, we wanted to explore if the Reach, Effectiveness, Adoption, Implementation and Maintenance framework (RE-AIM) was useful for evaluating different stages in the intervention project. RE-AIM was applied through a convergent mixed-methods triangulation design using survey, accelerometer, GPS, and interviews as data sources. Effectiveness outcomes revealed that on average the space was used less by the 39 children after the intervention. The implementation and maintenance dimensions revealed aspects of why most children involved in the project did not use the space after intervention. The evaluation cast light on children’s perceptions of their role, and importance of maintenance when the intervention was completed. In future, all dimensions of built environmental projects would benefit from being planned and evaluated in a collaboration with all project partners using an evaluation framework integrated and applied from the beginning of the project.     

Teaching Note—Asserting Social Work’s Role in Developing an Interprofessional Education Project

ABSTRACT

Interprofessional (IP) education is an essential component of today’s health care education. IP education has been recognized and supported for its potential to educate workforce-ready health care clinicians with the knowledge and skills, necessary to collaboratively deliver high-quality, client-centered care. While social work’s reflective, patient-centered, and holistic approach to care is invaluable in addressing healthcare challenges at all levels, social work’s lack of involvement in IP health care team practice is evident in the development of IP education curriculum with other health care disciplines. This article describes how one School of Social Work at a regional public university became part of a longitudinal IP education project in collaboration with a large R-1 research university, creating an IP opportunity for Master’s in Social Work (MSW) students. It also discusses the process taken for social work inclusion in the IP project, an experiential elective course designed to supplement the project, as well as Transformative Learning Theory, the conceptual framework used.     

The participatory design of a performance oriented monitoring and evaluation system in an international development environment

This article illustrates the application of the impact monitoring and evaluation process for the design and development of a performance monitoring and evaluation framework in the context of human and institutional capacity development. This participative process facilitated stakeholder ownership in several areas including the design, development, and use of a new monitoring and evaluation system, as well their targeted results and accomplishments through the use of timely performance data gathered through ongoing monitoring and evaluation. The process produced a performance indicator map, a comprehensive monitoring and evaluation framework, and data collection templates to promote the development, implementation, and sustainability of the monitoring and evaluation system of a farmer's trade union in an African country.     

A principles-focused evaluation of community mental health short-term case management

Principles-focused evaluations allow evaluators to appraise each principle that guides an organization or program. This study completed a principles-focused evaluation of a new community mental health intervention called Short Term Case Management (STCM) in Toronto, Canada. STCM is a time limited intervention for clients to address unmet needs and personalized goals over 3 months. Findings showcase that a principles-focused evaluation, assessing whether program principles are guiding, useful, inspiring, developmental and/or evaluable (GUIDE), is a practical formative evaluation approach. Specifically, offering an understanding of a novel intervention, including its key components of assessment and planning, support plan implementation and evaluation and care transitions. Findings also highlight that STCM may work best for those clients ready to participate in achieving their own goals. Future research should explore how best to apply the GUIDE framework to complex interventions, including multiple principles, to increase evaluation feasibility and focus.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

Design of an educational strategy based on Intervention Mapping for nutritional health promotion in Child Care Centers

Early childhood is a critical period for instilling healthy habits to prevent overweight and obesity. This paper describes the development of an educational intervention for the promotion of healthy eating and physical activity among two-to-four-year-old children in public child care centers (CCCs) in Mexico City. Following the Intervention Mapping (IM) protocol, we developed the Bright Futures multidisciplinary intervention. First, a formative research process to identify the personal and environmental determinants of childhood overweight and obesity, behavioral outcomes and performance objectives was conducted. Then, a matrix of change objectives by intersecting the performance objectives with the determinants was integrated. Bright Futures lasted six months and included 24 weekly sessions, each composed of five phases: warm-up, theory, active movements, relaxation, and hydration. Ad hoc interactive teaching materials focused on recreational activities, and formulated plans for the adoption, implementation and process/impact evaluation of the intervention was developed. IM successfully guided the design of a theory-driven and evidence-based intervention for children in CCCs within a socio-ecological and participatory planning framework. This is one of the first studies in Mexico to use IM in the context of CCCs.     

Using intervention-oriented evaluation to diagnose and correct students’ persistent climate change misconceptions: A Singapore case study

The evaluation of classroom-based educational interventions is fraught with tensions, the most critical of which is choosing between focusing the inquiry on measuring the effects of treatment or in proximately utilizing the data to improve practice. This paper attempted to achieve both goals through the use of intervention-oriented evaluation of a professional development program intended to diagnose and correct students’ misconceptions of climate change. Data was gathered, monitored and analyzed in three stages of a time-series design: the baseline, treatment and follow-up stages. The evaluation itself was the ‘intervention’ such that the data was allowed to ‘contaminate’ the treatment. This was achieved through giving the teacher unimpeded access to the collected information and to introduce midcourse corrections as she saw fit to her instruction. Results showed a significant development in students’ conceptual understanding only after the teacher's decision to use direct and explicit refutation of misconceptions. Due to the accessibility of feedback, it was possible to locate specifically at which point in the process that the intervention was most effective. The efficacy of the intervention was then measured through comparing the scores across the three research stages. The inclusion of a comparison group to the design is recommended for future studies.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.