Process evaluation of a preschool physical activity intervention using web-based delivery

Background and purposePreschool/childcare settings offer a practical target for physical activity interventions. Online learning programs have the potential for greater public health reach and impact. The SHAPES-Dissemination (SHAPES-D) project adapted the original SHAPES in-person intervention for online delivery to teachers. The purpose of this paper is to describe the implementation monitoring and process evaluation for the SHAPES-D project.MethodsNine preschools with 26 classrooms participated. A total of 41 teachers were trained via online learning to implement the SHAPES-D program in their classrooms. The dose received, completeness, and fidelity of implementation were assessed through website metrics, teacher surveys and interviews, and classroom observations.ResultsDose received was adequate (73%). Observed completeness and physical activity enjoyment fidelity were high (100%), although moderate-to-vigorous physical activity fidelity and social environment fidelity were low (25% each). Overall implementation was high (91%).DiscussionResults indicate that the online method of delivery is viable for dissemination. The online delivery system provides an easy method of monitoring dose received. This may be the first structural intervention to monitor dose received through web metrics.ConclusionThe adaptation of an in-person intervention to an online delivery system increases the potential for dissemination of a successful program to increase physical activity in preschool settings.     

Examining the link between program implementation and behavior outcomes in the lifestyle education for activity program (LEAP)

Lifestyle Education for Activity Program (LEAP) was a comprehensive, school-based intervention designed to promote physical activity in high school girls. The intervention focused on changes in instructional practices and the school environment to affect personal, social, and environmental factors related to physical activity. Multiple process evaluation tools and an organizational assessment tool were developed to monitor program implementation from a framework called the LEAP essential elements, which characterized complete and acceptable intervention delivery; secular trends were also monitored. Using process data, LEAP intervention schools were categorized into low- and high-implementing groups and compared with control schools on nine essential elements assessed at the organizational level. The Wilcoxon scores test revealed that low- and high-implementing intervention, and control schools differed significantly on two of nine administrator-reported organizational-level components: having a physical activity team and having a faculty-staff health promotion program. A mixed-model analysis of covariance indicated that, compared to control schools, a greater percentage of girls in high-implementing schools reported engaging in vigorous physical activity. Process evaluation can be used to understand the relationship between level of implementation and successful program outcome.     

An evaluation of transit procurement training

We evaluated a training course called “Orientation to Transit Procurement”, designed and conducted by the National Transit Institute. This course is designed to provide Federal Transit Administration (FTA) grantees an overview of regulations and best practices related to the procurement process. Our objective in conducting the evaluation was to understand how transit agency staff made changes in procurement practices in response to the course training. The evaluation was mixed mode: an Internet survey followed by in-depth interviews with a small group of respondents. Survey respondents were also provided with an open-ended question providing us with additional context for our evaluation. Results show that the training is substantially successful at meeting the goal of improving procurement practices at transit agencies; indeed, most respondents report making changes at their agencies as the proximate result of the training. This was at odds with our exploration of knowledge of procurement topics, as most respondents gave inaccurate answers on multiple-choice “knowledge questions”. This may have been due to question structure or, more likely, the nature of online surveys. Suitable training on the procurement of information technology was also a main concern. The lack of training in this area is indicative of the broader challenge facing public transit agencies in how to incorporate new forms of technology into their existing practices and bureaucratic structures.     

Physical activity and nutrition program for adults with metabolic syndrome: Process evaluation

IntroductionThe Vietnam Physical Activity and Nutrition (VPAN) program aimed to improve physical activity and nutrition for adults aged 50–65 years with Metabolic Syndrome in Vietnam. The VPAN program consisted of a range of resources and strategies, including an information booklet, resistance band, face-to-face education sessions, and walking groups. This process evaluation assessed the participation, fidelity, satisfaction, and reasons for completing and not-completing the VPAN.MethodsData were collected by mixed-methods from a sample of 214 intervention participants. Quantitative data were collected via surveys (n = 163); qualitative data via face-to-face exit interviews with intervention program completers (n = 10) and non-completers (n = 10), and brief post education session discussions.ResultsMost participants (87%–96%) reported the program resources and strategies useful, assisting them to increase their physical activity level and improving their diet. The education sessions were the most preferred strategy (97%) with high attendance (>78% of participants). The main reasons for withdrawal were work commitments and being too busy.ConclusionThe evaluation indicated that the program reached and engaged the majority of participants throughout the six-month intervention. The combination of printed resources and face-to-face intervention components was a suitable approach to support lifestyle behavioural change in the Vietnamese population.     

Using mixed method approach in measuring effects of training in firms: Case study of the European Social Fund support

Public support of training in firms corresponds to the long-term importance of the quality of human capital in the competitiveness of firms and nations. Thus, the EU supports such training via the European Social Fund (ESF). The evaluation community evaluates the support by using either qualitative or quantitative methods. The simultaneous application of these two approaches is rare. The purpose of this paper is to combine quantitative (counterfactual impact evaluation) and qualitative (qualitative comparative analysis) methods in order to fill the methodological gap. Based on the combination of both approaches, it explores their strengths, complementarity and disadvantages to evaluate public support for employee training in the Czech Republic. The combination of methods makes it possible to identify not only the impacts but also their causes. Linking the ESF support to corporate competitiveness is crucial for demonstrating the effectiveness of public spending.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

The impact of a faculty training program on inclusive education and disability

This paper describes the knowledge gained by 20 faculty members following their participation in a training program on inclusive education and disability. The study, which was conducted at an university in Spain, aimed to design, implement and evaluate a program for training faculty members to respond in an inclusive manner to the needs of students with disabilities. An initial, formative and summative qualitative evaluation was carried out and four instruments were used for collecting the data: group and individual interviews, written open-ended questionnaires and observations. The data were analyzed inductively, using a category and code system. The results reveal that, after the training program, faculty considered what they had learned to be useful for their professional practice and highlighted that they felt better-informed and better-trained in relation to disability and were more aware of the needs of students with disabilities. Finally, in the conclusions section, the paper discusses the results in relation to those reported by other studies, and offers some recommendations for universities planning to implement training policies designed to build more inclusive learning environments.     

An innovative medical and dental hygiene clinic for street youth: Results of a process evaluation

Canada has a noteworthy reputation for high quality health care. Nonetheless, street youth are one of our most vulnerable yet underserved populations. Consequently, a medical and dental clinic was created in downtown Ottawa, Ontario to respond to their needs. The purpose of this study is to describe a process evaluation of the clinic during its first year of operation with a focus on program fidelity, dose, reach, and satisfaction. A mixed methods approach was used involving interviews with providers, focus groups with street youth, analysis of Electronic Medical Record (EMR) data, and supplemental information such as document reviews. The evaluation identified areas that were working well along with challenges to program implementation. Areas of concerns and possible solutions were presented to the management team that then helped to plan and make improvements to the clinic. Our evaluation design and working relationship with clinic management promoted the integration of real-time evidence into program improvements.     

Lessons learned using a values-engaged approach to attend to culture,diversity, and equity in a STEM program evaluation

Evaluation must attend meaningfully and respectfully to issues of culture, race, diversity, power, and equity. This attention is especially critical within the evaluation of science, technology, engineering, and mathematics (STEM) educational programming, which has an explicit agenda of broadening participation. The purpose of this article is to report lessons learned from the implementation of a values-engaged, educative (Greene et al., 2006) evaluation within a multi-year STEM education program setting. This meta-evaluation employed a case study design using data from evaluator weekly systematic reflections, review of evaluation and program artifacts, stakeholder interviews, and peer review and assessment. The main findings from this study are (a) explicit attention to culture, diversity, and equity was initially challenged by organizational culture and under-developed evaluator–stakeholder professional relationship and (b) evidence of successful engagement of culture, diversity, and equity emerged in formal evaluation criteria and documents, and informal dialogue and discussion with stakeholders. The paper concludes with lessons learned and implications for practice.     

Breastfeeding interventions in Kansas: A qualitative process evaluation of program goals and objectives

Across the state of Kansas, eighteen public health departments received funding through the 2011 Breastfeeding Grant Initiative to start a breastfeeding intervention. The main objective of this study was to evaluate the progress toward program goals and objectives. This study was a process evaluation. Qualitative data were collected from recipient health departments at two time-points during the program year. Structured, open-ended questions were asked through telephone interviews. This study examined: (1) progress toward program goals and objectives, (2) problems encountered during implementation, and (3) evaluation measures employed to assess program impact. All health departments reported making significant progress toward program goals and objectives and reported successful collaboration with other healthcare providers. The use of breast pumps, educational classes, and professional training of staff were reported as providing the best outcome in the promotion of breastfeeding. The majority of respondents did not measure program impact. From a public health perspective, it is important that infants receive breast milk for the first six months of life. It appears that goals and objectives set a priori guided health departments with the administration of their breastfeeding program. Results may be used to enhance and sustain delivery of breastfeeding support programs in Kansas communities.     

Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.     

Framing diversity: A qualitative content analysis of public relations industry publications

This qualitative content analysis used framing theory to examine how gender and diversity have been framed over the past decade in modern public relations industry publications. Findings revealed that diversity is most often set aside in its own issue or column and not integrated throughout the whole of the publications. Most articles about diverse populations mentioned primarily Hispanic or Latino populations and Blacks or African Americans. Diversity is framed as different, a journey or process, a language problem, a responsibility, inclusion, transforming workplace culture, enhancing the profession, resource intensive, power driven, needing to be justified, and impacting the bottom line. Disparities in public relations still exist among gender, racial, and ethnic groups with less power and must be addressed more fully in public relations professional publications.     

Assessment of the monitoring and evaluation system of a population-based program: Theory-driven evaluation approach

The purpose of this study was to assess the performance of a monitoring and evaluation system (M&E) for a population-based program and to identify factors influencing its performance. The state health department provided a grant, guide, and training for health districts to implement the program and report quarterly and annually to the M&E. Some health districts did not meet this requirement. We also found that the M&E did not adequately capture the actual work of the districts in communities as related to the work plan. We identified several factors that negatively affected the performances of the M&E. For example, the state and health districts did not have a shared vision on how to develop and implement the program, and what to report to the M&E. We proposed strategies that could improve the timeliness and content of reports to the M&E, as well as its utilization in the future.     

Pragmatic ex-ante evaluation using an innovative conceptual framework: The case of a high-tech entrepreneurship program for women

Program designers at local project level usually design interventions under conditions of scarce resources and bounded rationality lacking sufficient information about the process and impact mechanisms of the complex social programs they design and implement. This paper proposes a model for pragmatic ex ante evaluation using an innovative conceptual framework for practitioners working in the field to improve gender equality in research and innovation.The presented and applied model is pertinent to the theory-oriented evaluation tradition using a logic frame and a theory of change approach adapted to the special requirements of ex-ante evaluation. The model is illustrated by the case study of a program for the promotion of women entrepreneurship in the STEM fields.It is concluded that in a setting with limited resources and information, ex-ante evaluations should aim at defining clear conditions of proportionality and need to take into account the extent to which contextual factors hinder or facilitate the implementation of a program, and the specific institutional and system mechanisms at play, in order to enable proactive risk management from the beginning.     

Process evaluation of integrated West Nile virus surveillance in northern Italy: an example of a One Health approach in public health policy

West Nile virus (WNV) is endemic in the Po valley area in northern Italy. Regional health authorities have implemented integrated WNV surveillance following a One Health approach, based on collaboration between human, animal and environmental health institutions. We evaluated this integrated WNV surveillance system in Emilia-Romagna, Lombardy and Piedmont regions by means of a process evaluation. We examined the system’s implementation fidelity, dose delivered and received, reach, and we identified strengths and weaknesses in the system. Qualitative and semi-quantitative data were obtained from three regional focus groups. Data were discussed in a follow up focus group, where participants suggested recommendations for improving the surveillance system.Inter-institutional and interdisciplinary integration and the creation of a 'community of practice' were identified as key elements for effective surveillance. We identified differences in the degree of interdisciplinary integration in the three regions, likely due to different epidemiological situations and years of experience in surveillance implementation. Greater collaboration and sharing of information, public engagement and economic assessments of the integrated surveillance approach would facilitate its social recognition and guarantee its sustainability through dedicated funding. We demonstrate that a transdisciplinary research approach based on process evaluation has value for designing and fine-tuning integrated health surveillance systems.     

Diversity training for the community aged care workers: A conceptual framework for evaluation

Older Australians are an increasingly diverse population, with variable characteristics such as culture, sexual orientation, socioeconomic status, and physical capabilities potentially influencing their participation in healthcare. In response, community aged care workers may need to increase skills and uptake of knowledge into practice regarding diversity through appropriate training interventions. Diversity training (DT) programs have traditionally existed in the realm of business, with little research attention devoted to scientifically evaluating the outcomes of training directed at community aged care workers. A DT workshop has been developed for community aged care workers, and this paper focuses on the construction of a formative evaluative framework for the workshop. Key evaluation concepts and measures relating to DT have been identified in the literature and integrated into the framework, focusing on five categories: Training needs analysis; Reactions; Learning outcomes, Behavioural outcomes and Results The use of a mixed methods approach in the framework provides an additional strength, by evaluating long-term behavioural change and improvements in service delivery. As little is known about the effectiveness of DT programs for community aged care workers, the proposed framework will provide an empirical and consistent method of evaluation, to assess their impact on enhancing older people’s experience of healthcare.     

An evaluation case: The implementation and evaluation of a problem-solving training program for adolescents

This study recounts the implementation and evaluation of a problem-solving skills training program for adolescents. Sixty-six students from the seventh, eighth, and ninth grades of a local junior high school were identified by guidance counselors as needing problem-solving training—48 male and 18 female students between the ages of 11 and 16. The hypothesis that students included in the training sessions would demonstrate better problem-solving skills, a more internal locus of control, and a higher level of self-esteem than control group students was tested via a pretest-posttest Control Group Design. The results of a stepwise discriminant analysis of the data supported the hypothesis. An analysis of grade level differences indicated that the program had less effect on the seventh-grade training group. Implications for program modification and the limitations of the evaluation findings are discussed.     

Implementation and evaluation of tourism industry: Evidentiary case study of night market development in Taiwan

Taiwan’s night markets are the most popular and unique characteristics of domestic and foreign tourists’ night life and have great potential value for tourism development. However, development of the night market is vulnerable to the negative impact of COVID-19. To discover the potential value and evaluation of the night market, we interviewed 46 experts from several industries and occupations in Taiwan to investigate the relationships between brand equity, benefits, motivations of and satisfaction with night market tourism and development. The results show that travel motivation has both direct and indirect effects on brand equity, benefits and satisfaction. Furthermore, customer satisfaction is the most critical performance attribute of night market tourism, which may be influenced by brand equity, benefit, and motivation. Managerial implications and future research directions are discussed.     

Evaluation of a cross-cultural training program for Pakistani educators: Lessons learned and implications for program planning

In this paper, we share the results of a summative evaluation of PEILI, a US-based adult professional development/training program for secondary school Pakistani teachers. The evaluation was guided by the theories of cultural competence (American Psychological Association, 2003, Bamberger, 1999, Wadsworth, 2001) and established frameworks for the evaluation of professional development/training and instructional design (Bennett, 1975, Guskey, 2002, King, 2014, Kirkpatrick, 1967). The explicit and implicit stakeholder assumptions about the connections between program resources, activities, outputs, and outcomes are described. Participant knowledge and skills were measured via scores on a pre/posttest of professional knowledge, and a standards-based performance assessment rubric. In addition to measuring short-term program outcomes, we also sought to incorporate theory-driven thinking into the evaluation design. Hence, we examined participant self-efficacy and access to social capital, two evidenced-based determinants or “levers” that theoretically explain the transformative space between an intervention and its outcomes (Chen, 2012). Data about program determinants were collected and analyzed through a pre/posttest of self-efficacy and social network analysis. Key evaluation findings include participant acquisition of new instructional skills, increased self-efficacy, and the formation of a nascent professional support network. Lessons learned and implications for the design and evaluation of cross-cultural teacher professional development programs are discussed.     

An alternative evaluation approach for the problem-solving training program: A utilization-focused evaluation process

A utilization-focused approach in evaluating this program would have placed more emphasis on clearly identifying evaluation users, their information needs, and likely use of findings. With the advantages of hindsight, it appears that such an approach might have led to greater focus on critical implementation issues that affected the eventual decision not to continue the program despite seemingly positive outcomes evaluation findings. Other methods options are also discussed, including the potential usefulness of critical case studies. Finally, there is discussion of how to prepare decision makers for utilization and work with them to interpret and apply findings.