Providing context to the implementation of health promoting schools: A case study

PurposeIssues related to program context are important components of population health interventions and particularly among complex, adaptive initiatives in schools. Health Promoting Schools (HPS) is a global approach to support early development of healthy behaviors among children and youth. The purpose of this study was to explicate the practical and contextual processes that influenced implementation of HPS in schools in Nova Scotia (Canada).MethodsUsing a case-study approach, data was collected using interviews (n = 14) with principals, teachers and parents, observations collected during school visits and document review in five diverse schools.ResultsCase study schools reported a variety of HPS practices and three key themes emerged that provided context to the processes that facilitated their implementation. The results suggest that although school characteristics (theme 1), like staff allocation, physical location and resources, are important, these barriers can be mitigated by building organizational capacity (theme 2) and establishing a supportive school community culture (theme 3).ConclusionsThe study provided insight to the variability in implementation by describing how contextual barriers were experienced and mitigated by schools. Establishing a broad system to support HPS, with collaboration across health and education sectors, could help to progress adoption, implementation and sustainability of HPS.     

Similarities and differences in program registers: A case study

Researchers, clinicians, and other professionals are increasingly in need of cost-effective, evidence-based programs and practices. However, these individuals may lack the time and, for some, the required expertise to search for and identify such interventions. To address this concern, several online registers that list or categorize programs according to their empirical evidence of effectiveness have been established. Although these registers are designed to simplify the task of selecting evidence-based interventions, the use of distinct review processes and standards by each register creates discrepancies in final program classifications, which can pose a challenge for users. The present case study highlights three programs that have been evaluated by more than one register and have received similar or different classifications. Reasons for inconsistencies are discussed, and several recommendations for evaluating organizations and register users are provided to enhance the functionality and ease of use of online program registers.     

A mixed-methods evaluation using effectiveness perception surveys,social network analysis,and county-level health statistics: A pilot study of eight rural Indiana community health coalitions

Community health coalitions (CHCs) are a promising approach for addressing disparities in rural health statistics. However, their effectiveness has been variable, and evaluation methods have been insufficient and inconsistent. Thus, we propose a mixed-methods evaluation framework and discuss pilot study findings. CHCs in our pilot study partnered with Purdue Extension. Extension links communities and land grant universities, providing programming and support for community-engaged research. We conducted social network analysis and effectiveness perception surveys in CHCs in 8 rural Indiana counties during summer 2017 and accessed county-level health statistics from 2015-16. We compared calculated variables (i.e., effectiveness survey k-means clusters, network measures, health status/outcomes) using Pearson’s correlations. CHC members’ positive perceptions of their leadership and functioning correlated with interconnectedness in their partnership networks, while more centralized partnership networks correlated with CHC members reporting problems in their coalitions. CHCs with highly rated leadership and functioning developed in counties with poor infant/maternal health and opioid outcomes. Likewise, CHCs reporting fewer problems for participation developed in counties with poor infant/maternal health, poor opioid outcomes, and more people without healthcare coverage. This pilot study provides a framework for iterative CHC evaluation. As the evidence grows, we will make recommendations for best practices that optimize CHC partnerships to improve local health in rural areas.     

The context of program implementation and evaluation: A pilot study of interorganizational differences to improve child welfare reform efforts

This article contributes to the growing literature on evaluation and implementation science by examining the interaction between staff perceptions of organizational strength with perceptions and indicators of program fidelity. As part of a pilot project related to the evaluation of the Family to Family initiative, a survey was distributed to employees within two urban child welfare agencies with a total of 410 respondents across both sites, for a combined response rate of 72.2%. Survey results were analyzed both in terms or respondents' perception of their agency as well as in relation to measures of program performance and workload. Multivariate models show that organizational indicators are the most significant and positive predictors of perceived program implementation. Specifically, staff who positively perceived the availability of information within their agency also believed that the programs were well implemented in their agency. These findings suggest that as the value of program changes are articulated within an organization, the implementation of the initiative is perceived to improve.     

Challenges and successes to the implementation of a zoo conservation-education program

Zoo conservation-education programs have potential to address biodiversity loss, although evidence of their effectiveness to encourage social change is in its infancy. Moreover, how a program is implemented may influence program efficacy, yet there is little evidence of the factors that shape this process in zoos. Accordingly, through a process evaluation of Zoos Victoria’s ‘When Balloons Fly” (WBF) conservation-education program against marine debris, we identified barriers and enablers to the implementation of the program which can be addressed to improve future initiatives. Between April-May 2018, 24 Zoos Victoria professionals completed an online survey focussed on identifying challenges and successes in implementing WBF. Four participants additionally completed a follow-up telephone interview. Themes were identified and organised according to the model of diffusion in service organisations (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). Our results illuminate that features of the organisation (e.g., structure, culture) rather than features of individuals (e.g., skills, motivation) or characteristics of the conservation-education program itself (e.g., messaging), were most influential to implementation success. Based on these findings, attention to the organisational context is critical for promoting and evaluating the success of conservation-education programs and must be given significant attention alongside program characteristics and staff capability to deliver them.     

Presidential youth fitness program implementation: An antecedent to organizational change

IntroductionGrounded in organizational change theory, the purpose of this study was to investigate the efficacy of the Presidential Youth Fitness Program (PYFP) and its association with healthy cultures within schools.MethodsUsing a qualitative approach, data were collected through interviews, site visits and artifacts across 374 schools. An explanatory collective case study approach was used to identify key events related to implementation.ResultsPivotal antecedents to organizational change included prolonged, continual PD, direct support of PYFP implementation, and recognition. Further, three key themes of leveling of the playing field, strategically overcoming barriers, and recruiting teacher fitness champions were identified.ConclusionsCreating a healthy school culture was an unexpected, but feasible outcome stemming from the implementation of the PYFP. A collective effort, led by physical education teachers and fitness champions and embraced by the administration, faculty, and community, is necessary for the school culture to unfreeze from its present status.     

Using client exit interviews to illuminate outcomes in program logic models: a case example

This paper describes how client exit interviews can assist human service administrators and workers to better understand the outcomes their programs are designed to accomplish. Specifically, the qualitative component of a demonstration family literacy program evaluation is used to illustrate how client input can be used to fine-tune the outcomes component of a program's logic model. An analysis of semi-structured exit interviews with 35 clients, who were randomly selected from all 89 served in the first year of the program, resulted in revision to the program's original logic model, creating explicit ‘testable’ pathways to achieving intended outcomes.     

An evaluation case: The implementation and evaluation of a problem-solving training program for adolescents

This study recounts the implementation and evaluation of a problem-solving skills training program for adolescents. Sixty-six students from the seventh, eighth, and ninth grades of a local junior high school were identified by guidance counselors as needing problem-solving training—48 male and 18 female students between the ages of 11 and 16. The hypothesis that students included in the training sessions would demonstrate better problem-solving skills, a more internal locus of control, and a higher level of self-esteem than control group students was tested via a pretest-posttest Control Group Design. The results of a stepwise discriminant analysis of the data supported the hypothesis. An analysis of grade level differences indicated that the program had less effect on the seventh-grade training group. Implications for program modification and the limitations of the evaluation findings are discussed.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

A visual management tool for program planning,project management and evaluation in paediatric health care

This article describes the development and implementation of a custom-designed Excel-based visual management tool. The tool’s purpose was to support program planning and evaluation by our resource support team within a paediatric health care setting. Our aims in developing it were to 1) establish a streamlined process and supporting tools to efficiently plan and prioritize program directions and activities; 2) track progress; and 3) evaluate and report on our performance, outputs and outcomes. A collaborative approach based on the ADKAR (Awareness, Desire, Knowledge, Ability, Reinforcement) change management model and the LEADS (Lead self, Engage others, Achieve results, Develop coalitions, Systems transformation) leadership framework was used to guide the design and implementation processes. Team members reported high perceived effectiveness and efficiency with respect to the tool’s utility in supporting its proposed aims. A graded approach to building knowledge and skills in using the tool, to individual responsibility for data entry, and to accountability by team members facilitated its successful implementation. Administrative support is important for sustainability and continual improvement of the tool to address changing team needs over time.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

A principles-focused evaluation of community mental health short-term case management

Principles-focused evaluations allow evaluators to appraise each principle that guides an organization or program. This study completed a principles-focused evaluation of a new community mental health intervention called Short Term Case Management (STCM) in Toronto, Canada. STCM is a time limited intervention for clients to address unmet needs and personalized goals over 3 months. Findings showcase that a principles-focused evaluation, assessing whether program principles are guiding, useful, inspiring, developmental and/or evaluable (GUIDE), is a practical formative evaluation approach. Specifically, offering an understanding of a novel intervention, including its key components of assessment and planning, support plan implementation and evaluation and care transitions. Findings also highlight that STCM may work best for those clients ready to participate in achieving their own goals. Future research should explore how best to apply the GUIDE framework to complex interventions, including multiple principles, to increase evaluation feasibility and focus.     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

Validity in an evaluation of Healthy Families Florida—A program to prevent child abuse and neglect

Evidence-based continuums have become a guide for identifying the level of evidence in evaluations of home visiting programs conducted to determine their effectiveness in preventing child abuse and neglect. While randomized controlled trials are required for the highest levels of evidence, quasi-experimental designs have also been specified as an appropriate alternative. Using a quasi-experimental evaluation of a home visiting program to prevent child abuse and neglect that adheres to the Healthy Families America model, this paper describes and illustrates how types of validity can be improved. More specifically, we address how threats to internal validity can be identified and reduced through statistical techniques; how construct validity may be strengthened using state records to measure the outcomes; and how external validity is affected by including or excluding study participants. After applying a variety of statistical adjustments to reduce selection bias, we found that the outcomes favored the home visiting program and increased after accounting for covariates that contributed to child abuse and neglect. This was true across the statistical techniques (traditional covariate and propensity score adjustment) used. For evaluations using quasi-experimental designs, recommendations relevant to the illustrations in the paper are also presented.     

Breastfeeding interventions in Kansas: A qualitative process evaluation of program goals and objectives

Across the state of Kansas, eighteen public health departments received funding through the 2011 Breastfeeding Grant Initiative to start a breastfeeding intervention. The main objective of this study was to evaluate the progress toward program goals and objectives. This study was a process evaluation. Qualitative data were collected from recipient health departments at two time-points during the program year. Structured, open-ended questions were asked through telephone interviews. This study examined: (1) progress toward program goals and objectives, (2) problems encountered during implementation, and (3) evaluation measures employed to assess program impact. All health departments reported making significant progress toward program goals and objectives and reported successful collaboration with other healthcare providers. The use of breast pumps, educational classes, and professional training of staff were reported as providing the best outcome in the promotion of breastfeeding. The majority of respondents did not measure program impact. From a public health perspective, it is important that infants receive breast milk for the first six months of life. It appears that goals and objectives set a priori guided health departments with the administration of their breastfeeding program. Results may be used to enhance and sustain delivery of breastfeeding support programs in Kansas communities.     

Maltreatment prevention through early childhood intervention: A confirmatory evaluation of the Chicago Child-Parent Center preschool program

Increased recognition of the consequences associated with child maltreatment has led to greater emphasis on its prevention. Promising maltreatment prevention strategies have been identified, but research continues to suffer from methodological limitations and a narrow focus on select prevention models. This investigation uses data from the Chicago Longitudinal Study to examine mediating mechanisms that link the Chicago Child-Parent Center preschool program to a reduction in overall child maltreatment and, more specifically, child neglect. We use structural equation modeling to test child, family, and school measures hypothesized to mediate the effects of CPC participation on maltreatment and neglect. Results indicate that a substantial proportion of the program's impacts can be accounted for by family support processes, including increased parent involvement in school and maternal educational attainment as well as decreased family problems. The CPC program's association with reduced school mobility and increased attendance in higher-quality schools also significantly mediated its effects on maltreatment and neglect. Further, a decrease in troublemaking behavior contributed modestly to mediating the program's association with maltreatment but not neglect. We discuss the implications of these results for the field of maltreatment prevention.     

Publishing systematic, pragmatic case studies in program evaluation: rationale and introduction to the special issue

The present Special Issue proposes that the creation of a database of systematic, pragmatically focused case studies in program planning and evaluation will be useful to both theorist and practitioner. This knowledge-base can (a) facilitate cross-case comparisons for inductively deriving generalizations of best practice; (b) allow practitioners to use the experience of past cases to guide their practice in present cases; and (c) further explore the foundational nature of evaluation as a discipline. This introduction first explores why there are so few published evaluation case studies. It then describes the development of a pragmatic paradigm, including a common analytic framework for facilitating the conduct, documentation, and discussion of individual case studies and for organizing them into a database. The introduction concludes with a cross-case summary comparison of the three cases in this first part of the Special Issue. In the next issue of this journal, a second part of the Special Issue will be presented. This will include three additional case studies and an analysis of all six of the case studies in terms of what we have learned about the theoretical and practical values—as well as the challenges—of creating a database of systematic, pragmatic case studies in evaluation.