Process evaluation of a preschool physical activity intervention using web-based delivery

Background and purposePreschool/childcare settings offer a practical target for physical activity interventions. Online learning programs have the potential for greater public health reach and impact. The SHAPES-Dissemination (SHAPES-D) project adapted the original SHAPES in-person intervention for online delivery to teachers. The purpose of this paper is to describe the implementation monitoring and process evaluation for the SHAPES-D project.MethodsNine preschools with 26 classrooms participated. A total of 41 teachers were trained via online learning to implement the SHAPES-D program in their classrooms. The dose received, completeness, and fidelity of implementation were assessed through website metrics, teacher surveys and interviews, and classroom observations.ResultsDose received was adequate (73%). Observed completeness and physical activity enjoyment fidelity were high (100%), although moderate-to-vigorous physical activity fidelity and social environment fidelity were low (25% each). Overall implementation was high (91%).DiscussionResults indicate that the online method of delivery is viable for dissemination. The online delivery system provides an easy method of monitoring dose received. This may be the first structural intervention to monitor dose received through web metrics.ConclusionThe adaptation of an in-person intervention to an online delivery system increases the potential for dissemination of a successful program to increase physical activity in preschool settings.     

Working on Womanhood (WOW): A participatory formative evaluation of a community-developed intervention

The well-documented disparities in availability, accessibility, and quality of behavioral health services suggest the need for innovative programs to address the needs of ethnic minority youth. The current study aimed to conduct a participatory, formative evaluation of “Working on Womanhood” (WOW), a community-developed, multifaceted, school-based intervention serving primarily ethnic minority girls living in underserved urban communities. Specifically, the current study aimed to examine the feasibility, acceptability, and initial promise of WOW using community-based participatory research (CBPR) and represented the third phase of a community-academic partnership. Qualitative and quantitative data were collected from 960 WOW participants in 21 urban public schools, as well as WOW counselors, parents, and school staff over the course of one academic year. Results demonstrated evidence of acceptability of WOW and noteworthy improvements for WOW participants in targeted outcomes, including mental health, emotion regulation, and academic engagement. Findings also indicated several challenges to implementation feasibility and acceptability, including screening and enrollment processes and curriculum length. Additionally, we discuss how, consistent with participatory and formative research, findings were used by program implementers to inform program improvements, including modifications to screening processes, timelines, curriculum, and trainings – all in preparation for a rigorous effectiveness evaluation.     

Natural experiment methodology for research: a review of how different methods can support real-world research

In particular research domains, the randomized control trial (RCT) is considered to be the only means for obtaining reliable estimates of the true impact of an intervention. However, an RCT design would often not be considered ethical, politically feasible, or appropriate for evaluating the impact of many policy, programme, or structural changes common in public health research. As such, researchers must use alternative yet robust research methods for determining the impact of such interventions. The evaluation of natural experiments (i.e. an intervention not controlled or manipulated by researchers), using various experimental and non-experimental design options can provide an alternative to the RCT. The following review highlights (a) the importance of evaluating natural experiments; (b) design considerations associated with evaluating natural experiments; (c) methods for reducing bias in natural experimental studies; and (d) the potential benefits of targeted systems to enable natural experiments in emerging priority domains moving forward.     

The American Stop Smoking Intervention Study. Conceptual framework and evaluation design.

Reducing tobacco use, especially cigarette smoking, is a public health priority. The American Stop Smoking Intervention Study (ASSIST) was initiated in 1991 to prevent and reduce tobacco use primarily through policy-based approaches to alter the social-political environment. This article describes the conceptual design, research framework, evaluation components, and analytic strategies that are guiding the evaluation of this demonstration research endeavor. The ASSIST evaluation is a unique analysis of the complex relationships between the social context, public health activity at the state level, tobacco use, and individual behavior. The measures of tobacco control activity developed for this evaluation may be useful in ongoing national cancer control surveillance efforts, and the lessons learned will enhance the development of tobacco control programs.     

A flexible framework for planning and evaluating early-stage health interventions: FRAME-IT

Health interventions exhibit three stages of maturity: early-, mid-, and late-stages. Early-stage interventions have innovative content necessitating evaluation; however existing evaluation frameworks omit constructs and guidelines relevant to this evaluation. Early-stage interventions require planning and evaluation that supports creating, testing, and exploring content to establish general feasibility and enable refinement for further testing, prior to randomised controlled trialling and wider dissemination. Feasibility, Reach-out, Acceptability, Maintenance, Efficacy, Implementation, Tailorability (FRAME-IT) was developed for a mixed methods feasibility study of a novel well-being intervention. FRAME-IT was conceived as a complementary framework to Reach, Efficacy, Adoption, Implementation, Maintenance (RE-AIM; Glasgow et al., 1999) which is better suited for mid- and late- stage interventions. FRAME-IT is proposed to support: (1) early-stage intervention planning and design, by guiding research focus and data sourcing strategy with relevant constructs; (2) comprehensive evaluation, by including constructs appropriate for early-stage interventions, i.e. feasibility, acceptability, and tailorability; (3) future intervention scalability, by including and adapting some of RE-AIM’s constructs to encourage a smoother translation of research into practice as interventions are scaled-up.     

How Academic Research Shapes Labor and Social Policy

How academic research affects labor and social policy is viewed through a program evaluation framework that highlights the difficulties of determining the causal impact of such research on public policy. The effect is illustrated by examples. My conclusion is that academic research can have a modest to substantial impact on policy. Its impact is enhanced if it has a number of key characteristics: high quality; reputable researchers involved; synthesized and translated into a language understood by policy makers, the general public, and the media; credible champions who will broker and defend it, in the political process or in the public realm; timeliness; and, political acceptability.     

Latino youth participation in community-based participatory research to reduce teen pregnancy disparities

BackgroundCommunity-based participatory research can provide a framework to build community capacity to do health equity research, particularly from community members who may not typically participate in research design and intervention.PurposeThe purpose of this paper is to describe a community-based coalition's partnership and engagement with Latino youth throughout the research process addressing health disparities in unintended teen pregnancy rates in a local community. Israel and colleagues' components of CBPR provide a framework to develop youth participation throughout the research process.MethodHigh school and college Latino youth participated in health equity research from community assessments to design an intervention and dissemination of results.ResultsWorking with youth can improve the integrity and validity of the research process and can also provide benefits to the community and individual youth members, resulting in increased community capacity for health equity research.DiscussionLessons learned about the direct and indirect benefits and challenges are presented. Community-based partnerships working with youth should consider documenting the individual and collective impact of community engagement on the youth from the onset of participation.     

Working with fathers of at-risk children: Insights from a qualitative process evaluation of an intensive group-based intervention

This article is based on qualitative research with fathers who attended Mellow Dads, an intensive ‘dads only’ group-based intervention underpinned by attachment theory for fathers of at-risk children. Specifically the article draws on data from a process evaluation of the programme in order to explore the challenges of engaging men in effective family work. The methods used to undertake the process evaluation included participant observation of one complete Mellow Dads course, interviews with fathers and facilitators, interviews with the intervention author and a study of programme documentation. The article focuses on the theoretical underpinning of the programme, its acceptability to the fathers and the challenges faced by facilitators in delivering the programme as intended. The fathers appreciated the efforts of facilitators to make the group work, valued the advice on play and parenting style as well as the opportunity to meet other fathers in similar circumstances. However, there were obstacles that impacted on the effectiveness of the programme. These included the considerable time required to get the men to attend in the first place and then to keep them coming, the lack of practice of parenting skills when fathers were not living with their children, and the difficulties of sharing personal information. The challenges identified raise questions about how much change can be expected from vulnerable fathers and whether programmes designed for mothers can be applied to fathers with little adaptation. The article aims to contribute to ongoing dialogue about the best way to successfully engage fathers in children's well-being, and raises the question as to whether working with fathers requires different skill-sets and approaches from the more familiar social work territory of working with mothers.     

A sea change on the island continent: frameworks for risk assessment,prevention and intervention in child health in Australia

This article overviews prevention and early intervention approaches focusing specifically on their relevance to the health of children and young people in Australia. Australian public health has a sound track record although concealed within the aggregate profile are a number of sub-populations with poorer health indicators. Recognition of this has increased efforts to improve the health of children and young people especially in exploring the impact of social environments within the communities where children are raised. This paper examines emerging research in this area drawing out key lessons and learning from Australian experience in the field of early intervention and prevention in community settings.     

Dissemination as success: Local emergency management communication practices

Local emergency managers are charged by the U.S. Department of Homeland Security to increase the number of community members with emergency plans and kits. This outcome remains unrealized in part based upon in-depth interviews with 37 emergency managers in eight states revealing that emergency managers conceptualize communication as dissemination rather than as a process of research and evaluation. This study calls for local emergency managers to conduct outcome evaluations of risk communication campaigns to increase public participation.     

The next decade in evaluation research

The field of evaluation research has developed rapidly in the past few years. Its growth is marked by:(1) establishment of an'entrepreneurial industry of considerable proportions; (2) emergence of a new profession as evidenced by the founding of professional associations, publication of books and journals, and the development of specialized training programs; (3) recognition of the importance of evaluation in the federal bureaucracy by the establishment of key executive positions in the various offices of the Secretaries; (4) quality control efforts by a variety of stakeholders and watchdog groups; and (5) efforts to address methodological problems and to maximize the dissemination and utilization of studies. In the future, continued federal funding in health, education, and welfare will be dependent on demonstrated program impact, and “sunset legislation.” Moreover, there will be fewer important “policy” evaluations and more administrative and managerial efforts. Evaluation questions, increasingly, will be specified either by Congress or the administering agency, including the setting of design and methodological requirements for studies. Opportunities for investigator-inspired studies and basic methodological development will decrease.     

Development of an intervention to improve mental health for obstetric fistula patients in Tanzania

Obstetric fistula is a debilitating childbirth injury that has been associated with high rates of psychological distress. Global efforts have helped to link women to surgical repair, but thus far no evidence-based interventions exist to address the psychological needs of these women during the hospital stay. In this paper, we describe the development of a psychological intervention for women in Tanzania who are receiving surgical care for an obstetric fistula. The intervention was developed based on theories of cognitive behavioral therapy and coping models. Content and delivery were informed by qualitative data collection with a range of stakeholders including women with fistula, and input from a study advisory board. The resulting intervention was six individual sessions, delivered by a trained community health nurse. The session topics were (1) recounting the fistula story; (2) creating a new story about the fistula; (3) loss, grief and shame; (4) specific strategies for coping; (5) social relationships; and (6) planning for the future. A trial run of the intervention revealed that the intervention could be delivered with fidelity and was acceptable to patients. A future randomized control trial will evaluate the efficacy of this intervention to address the mental health symptoms of this population.     

Consumer Evaluation Research: Implications for Methodology,Social Policy,and the Role of the Sociologist*

Consumer evaluation research is defined and analyzed as an experimental model for establishing a link between social science research and public policy. The paper reports the results of an 18-month longitudinal study of an experimental citizen evaluation group. Participant observation was the method used to organize the data. The impact of the model on methodological issues, such as objectivity, researcher autonomy, and control over research decision-making, is discussed. Implications of the CER model for integrating social research and public policy formulation are analyzed.     

Transition to Work: Effects of Preparedness and Goal Construction on Employment and Depressive Symptoms

This study examines the mediating role of employment preparedness in improving employment, mental health, and construction of work‐life goals among young vocational school graduates who participated in the School‐to‐Work effectiveness trial. The trial included a 1‐week intervention program that focused on enhancing employment preparedness. In this trial, 416 graduates of secondary vocational institutes were randomized into a control and experimental group. All the study participants were assessed at baseline, immediately after the intervention, and 10 months later. The results showed that the School‐to‐Work intervention increased employment preparedness, which in turn increased employment at the 10‐month follow‐up. Furthermore, employment predicted work‐life‐related personal goals and lower financial strain, which in turn was associated with lower depressive symptoms. The implications for theory and practice are discussed.     

Dissemination and Implementation Research in Marriage and Family Therapy: An Introduction and Call to the Field

Despite the considerable resources allocated to research to promote public health, interventions capable of benefiting individuals and families are not finding their way into regular practice. An important avenue for addressing this problem is through dissemination and implementation (D&I) science, a burgeoning research area focusing on translating empirical knowledge into everyday practice. This article begins by suggesting ways in which MFTs are uniquely equipped to contribute to and benefit from D&I research. We will then provide an overview of D&I research, outlines key D&I models, and highlight examples of family intervention research relevant to MFTs using the key models. Finally, we conclude by providing the field with important next steps to advance the presence of MFT research within D&I scholarship.     

Statewide dissemination of trauma-focused cognitive-behavioral therapy (TF-CBT)

Despite evidence linking childhood trauma to subsequent social, emotional, psychological, and cognitive problems, many children who have experienced trauma do not receive mental health treatment that has been proven to be effective. Large-scale dissemination of evidence-based practices (EBPs) is one possible solution to enhance the current negative state of mental health treatment for these children. This article describes a dissemination effort of an EBP (i.e., Trauma-Focused Cognitive-Behavioral Therapy [TF-CBT]) for childhood symptoms of post-traumatic stress disorder throughout Arkansas. The effort targeted mental health professionals within child advocacy centers and community mental health centers across the state. The article describes the process of dissemination and implementation. Lessons learned and recommendations for future dissemination efforts are highlighted.     

Identifying the norms of professional practice: Reviewing PRSA's Silver Anvil award-winning campaigns

This study uses content analysis of 420 Public Relations Society of America Silver Anvil award-winning campaigns to assess how well the ROPES process of campaign management has been institutionalized into the public relations industry. By analyzing 42 years of award-winning campaigns, the research found notable trends in relation to the research practices, goal and objective planning, strategy design and implementation, campaign evaluation, and on-going efforts to cultivate relationships with stakeholders. These trends are discussed in relation to the continued struggles of public relations educators to professionalize the industry and move away from stereotypes as publicity seekers and special event planners.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.     

Feasibility and acceptability of a child sexual abuse prevention program for childcare professionals: comparison of a web-based and in-person training

Recent prevention research has established the efficacy of some child sexual abuse prevention programs targeting adults; however, less is known about the feasibility of implementing such programs. The current study examines the feasibility and acceptability of a child sexual abuse prevention program for child care professionals provided in two different formats: in person and Web based. The sample consisted of 188 child care professionals from a large-scale, multisite, randomized controlled trial. Findings indicate that both in-person and online training formats are feasible to implement and acceptable to professionals. When comparing formats, the in-person format was favored in terms of comfort level and likelihood of sharing information with others. These findings have significant implications for dissemination of child sexual abuse prevention programs for child care professionals.