Process evaluation of “Girls on the Run”: Exploring implementation in a physical activity-based positive youth development program

Many positive youth development programs rely on physical activity as a primary program component. Referred to as physical activity-based youth development programs, these program designs have great potential for promoting healthy youth development. This study examined how one such physical activity-based positive youth development program was implemented in order to identify design features critical to maximizing positive youth outcomes. This mixed method, multi-site process evaluation of Girls on the Run (GOTR) utilized focus groups, site visits, and self-report implementation checklists. Implementation scores were calculated to assess implementation fidelity across twenty-nine sites, and qualitative data were inductively analyzed to identify factors influential for implementation. Results reveal variability in how GOTR was implemented. Five themes emerged from the data that represented factors serving as facilitators or barriers to programmatic implementation. These included contextual/environmental factors (e.g., parental involvement, relationships with school personnel), organizational factors (e.g., implementation support and responsiveness of staff), program-specific factors (e.g., curriculum design), coach factors (e.g., existing relationships with participants, responsiveness to participant's needs), and youth factors (e.g., behavioral and discipline issues). Study findings have implications for improving the design of physical activity-based and other positive youth development programs, with relevance to evaluators, program planners, youth development leaders, and others working with children and youth.     

Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

Evaluation of a complex, multisite, multilevel grants initiative

The Safe Schools/Healthy Students (SS/HS) national evaluation seeks to assess both the implementation process and the results of the SS/HS initiative, exploring factors that have contributed to or detracted from grantee success. Each site is required to forge partnerships with representatives from education, mental health, juvenile justice, and law enforcement, coordinating and integrating their efforts and working together to contribute to comparable outcomes (e.g., reduced violence and alcohol and drug use, improved mental health services). The evaluation uses multiple data collection techniques (archival data, surveys, site visits, interviews, and focus groups) from a variety of sources (project directors, community partners, schools, and students) over several years. Certain characteristics of the SS/HS initiative represent unique challenges for the evaluation, including the absence of common metrics for baseline, outcome data, and lack of comparison group. A unifying program theory was required to address these challenges and synthesize the large amounts of qualitative and quantitative information collected. This article stresses the role of program theory in guiding the evaluation.     

Monitoring and evaluation design of Malawi’s Right Foods at the Right Time nutrition program

Child stunting is a public health problem in Malawi. In 2014, the Government of Malawi launched the Right Foods at the Right Time (RFRT) program in Ntchisi district delivering nutrition social and behavior change communication, a small-quantity lipid-based nutrient supplement to children 6–23 months, and nutrition sensitive activities. Monitoring and evaluation (M&E) systems are key aspects of successful program implementation. We describe these and the methodology for an impact evaluation that was conducted for this program. Two monitoring systems using traditional and electronic platforms were established to register and track program delivery and processes including number of eligible beneficiaries, worker performance, program participation, and to monitor input, output, and outcome indicators. The impact evaluation used comparative cross-sectional and longitudinal designs to assess impact on anthropometric and infant and young child feeding outcomes. Three cross-sectional surveys (base-, mid-, and end-line) and two longitudinal cohorts of children followed in 6-month intervals from 6 to 24 months of age, were conducted in sampled households in the program and a neighboring comparison district. Additional M&E included qualitative studies, a process evaluation, and a cost-effectiveness study. The current paper describes lessons from this program’s M&E, and demonstrates how multiple implementation research activities can inform course-correction and program scale-up.     

An evaluation of immediate outcomes and fidelity of a drug abuse prevention program in continuation high schools: project towards no drug abuse (TND)

The present study provides an implementation fidelity, process, and immediate outcomes evaluation of Project Towards No Drug Abuse (TND), a drug prevention program targeting continuation high school youth (n=1426) at risk for drug abuse. A total of 24 schools participated in three randomized conditions: TND Only, TND and motivational interviewing follow-up, and no treatment control. Fidelity was high: across program schools the curriculum was implemented as intended and was received favorably by students. Relative to controls, intervention conditions produced effects on hypothesized mediators, including greater gains in program related knowledge, greater reductions in drug use intentions, and positive changes in motivation. However, few generalizations to attitudes and intentions regarding risky sexual behavior were found. The pattern of results suggests that the experimental manipulations worked as intended.     

A qualitative exploratory study of substance abuse prevention outcomes in a heterogeneous prevention system

The prevention of abuse of alcohol and other drugs is a concern for parents, policy-makers, educators, and social service professionals. Prevention programs are sponsored by many different types of social and educational agencies using a variety of intervention strategies. This article reports a study of a sample of such programs in the state of Nevada. The overall prevention system in the state espouses a "risk and resiliency" approach to prevention. Focus group methodology was used to study perception of outcomes of these programs from the viewpoints of various program stakeholders (youth participating in the programs, parents of participants and program staff). Analysis of the qualitative data yielded findings about potential outcomes as well as implicit program theories. Implications for future planning efforts as well as further evaluation efforts are discussed.     

Application Review of Journey Mapping

Journey Mapping is a unique Internet-driven program evaluation tool supporting both qualitative and quantitative data collection. Successful implementation of the tool requires minimal training. The Journey Mapping website is updated on a regular basis to include new features; all users log on and work with the latest version. Journey Mapping supports collection of program outcome data, as well as data for program refinements. Realizing the full potential and benefits of Journey Mapping may depend on integrating the system with staff job designs and the overall mission of the program.     

Medical students' and tutors' experiences of directed and self-directed learning programs in evidence-based medicine: a qualitative evaluation accompanying a randomized controlled trial

This qualitative study aims to interpret the results of a randomized controlled trial comparing two educational programs (directed learning and self-directed learning) in evidence-based medicine (EBM) for medical students at the University of Oslo from 2002 to 2003. There is currently very little comparative educational research in this field. In the trial, no statistically significant differences between the study groups were shown for any outcomes considered (EBM knowledge, skills, and attitudes). Further analysis suggests that main reason for the negative trial results was that the majority of students learned equally effectively, whichever program they received, although implementation of the educational programs was not complete because of varying attendance. This study illustrates a stepwise evaluation model that might be useful in evaluating other socially complex interventions.     

Critical Variables for Infusing Community Content into a Clinical Program

ABSTRACT

This empirical study seeks to identify effective methods for infusing community material into a clinical social work program. It draws from Diffusion of Innovation Theory (Rogers, 1995b) and the Transtheoretical Model (Prochaska, DiClemente, &;Norcross, 1992) to guide implementation. Based on qualitative and quantitative data from a variety of sources, the results identify variables critical in the change process and some preliminary outcomes of that change. Findings suggest that teaching community content can be enhanced through developing a pool of outside community guest speakers and field trips, addressing the varying faculty development needs of professors teaching macro and micro social work courses, and defining and redefining the concept of community during implementation and change processes.     

Implementation and process evaluation of a school-based drug abuse prevention program: Project Towards No Drug Abuse

The present study provides an implementation, process, and immediate outcomes evaluation of the classroom component of Project Towards No Drug Abuse (TND). This project involves development and evaluation of a school-based drug abuse prevention curriculum for continuation high school youth, who are at relatively high risk for drug abuse. Three randomized conditions were evaluated: a standard care, classroom only, and classroom plus school-as-community. The latter condition was an enhanced school-based condition which involved outside-of-classroom meetings and activities. Implementation was high in both program conditions even though this was a higher risk context. Process evaluation data were favorable and did not vary between the two program conditions. Immediate outcomes data (knowledge) was higher in the two program conditions than in the standard care condition. Regarding the classroom program, addition of extra-classroom activities does not appear to alter the quality of delivery of the program.     

A model-guided process evaluation: Office-based prescribing and pharmacy dispensing of methadone

This article presents an exemplar of a model-guided process evaluation that specifies the treatment model, assesses its implementation, monitors the fidelity of the model throughout the project, assesses model exposure and absorption, and helps understand the program's intermediate effects (proximal outcomes) as well as final effects (distal outcomes). The New Mexico study on office-based prescribing and community pharmacy dispensing of methadone is a research demonstration project that phases a small group of female methadone maintenance patients out of methadone clinics and into a program where they will obtain their scheduled doses of methadone at pharmacies that work in collaboration with physicians and a social worker.The patient's methadone treatment will in this way become part of their overall health care. Early detection of problems of implementation (e.g., the omission of program content or the delivery of inaccurate information) enables the researcher to make adjustments before the problems become unmanageable and the integrity of the original research design is compromised. A model-guided process evaluation can critically inform health services research demonstrations designed for enabling continuous, ongoing feedback and improvement of client-related services.     

Examining the Student's Decision to Pursue a Joint Graduate Degree Program in Social Work and the Law

In response to economic and educational trends, an increasing number of American universities are offering joint graduate degree programs in social work and the law. However, little is known about what drives a student to pursue study in these programs. This research study involved qualitative interviews with nine students in the United States either currently enrolled in a joint degree program in social work and law or who considered the joint degree program, but chose a single degree program instead. Interviews focused on what factors drew students to, or deterred students from, dual matriculation. Grounded theory analysis was used to identify themes across interviews. The findings can inform development of new joint degree programs in social work and law, as well as guide improvement of recruitment and retention practices for existing programs.     

Evaluation of a cross-sector community initiative partnership: Delivering a local sport program

Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders’ satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.     

A Qualitative Evaluation of a Mentor Program for At-Risk Youth: The Participants' Perspective

This article presents a qualitative evaluation of the first year of a mentor program for at-risk high school youth in a low income urban setting with high rates of youth and violent crime. Pre and posttest data were collected employing a standardized set of open-ended questions regarding the program and the mentees' relationships with their mentors. Overall there appears to be congruence between the mentees' expectations at pretest and posttest outcomes. Most were overwhelmingly positive about the program, developed a valued relationship with their mentors, and secured concrete benefits as well (e.g., employment and greater academic achievement). Two mentors who had a leadership role were also interviewed. Case studies of four of the adolescents are presented along with implications for practice.     

Interviewer as instrument: accounting for human factors in evaluation research

This methodological study examines an original data collection model designed to incorporate human factors and enhance data richness in qualitative and evaluation research. Evidence supporting this model is drawn from in-depth youth and adult interviews in one of the largest policy/program evaluations undertaken in the United States, the Drug, Alcohol, and Tobacco Education evaluation (77 districts, 118 schools). When applying the explicit observation technique (EOT)--the strategic and nonjudgmental disclosure of nonverbal human factor cues by the interviewer to the respondent during interview--data revealed the observation disclosure pattern. Here, respondents linked perceptions with policy or program implementation or effectiveness evidence. Although more research is needed, it is concluded that the EOT yields richer data when compared with traditional semistructured interviews and, thus, holds promise to enhance qualitative and evaluation research methods. Validity and reliability as well as qualitative and evaluation research considerations are discussed.     

An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.     

An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.     

Qualitative cost–benefit evaluation of complex,emergent programs

This paper discusses a methodology used for a qualitative cost–benefit evaluation of a complex, emergent program. Complex, emergent programs, where implementation varies considerably over time and across sites to respond to local needs and opportunities, present challenges to conventional methods for cost–benefit evaluation. Such programs are characterized by: ill-defined boundaries of what constitutes the intervention, and hence the resources used; non-standardized procedures; differing short-term outcomes across projects, even within the same long-term goals; and outcomes that are the result of multiple factors and co-production, making counter-factual approaches to attribution inadequate and the use of standardized outcome measures problematic. The paper discusses the advantages and limitations of this method and its implications for cost–benefit evaluation of complex programs.     

Building School,Family, and Community Partnerships: Developing a Theoretical Framework

School, family, and community partnerships are among the suite of strategies advocated for improved school outcomes in which social workers can play a valuable role. Such partnerships are complex to implement in practice and there is little systematic research to guide the practitioner. This study evaluated an early intervention partnership program involving school, family, community, and philanthropic partners implemented in two primary schools in rural Victoria, Australia. The findings highlighted challenges in the process of evaluation and program implementation and in the opportunities this provided to identify key enabling factors. These involved having: a shared vision and aims; democratic governance; a supportive policy and organisational environment including external funding; workers with skills and clear roles; activities for project momentum; and ongoing review and evaluation. These factors can be used by social workers and others working with schools as a valuable conceptual framework to facilitate school, family, and community partnership processes.