Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.     

Similarities and differences in program registers: A case study

Researchers, clinicians, and other professionals are increasingly in need of cost-effective, evidence-based programs and practices. However, these individuals may lack the time and, for some, the required expertise to search for and identify such interventions. To address this concern, several online registers that list or categorize programs according to their empirical evidence of effectiveness have been established. Although these registers are designed to simplify the task of selecting evidence-based interventions, the use of distinct review processes and standards by each register creates discrepancies in final program classifications, which can pose a challenge for users. The present case study highlights three programs that have been evaluated by more than one register and have received similar or different classifications. Reasons for inconsistencies are discussed, and several recommendations for evaluating organizations and register users are provided to enhance the functionality and ease of use of online program registers.     

Assessing program impact using latent growth modeling: a primer for the evaluator

Latent growth modeling (LGM) has emerged as a flexible analytic technique for modeling change over time because it can describe developmental processes at both the inter- and intra-individual levels. The LGM method can also provide a means for testing the contribution of other variables in order to explain variability in growth trajectories. This paper didactically illustrates the use of LGM as an analytical tool in program evaluation. Specifically, a hypothetical evaluation of a high school drug prevention program was used to demonstrate: (a) how LGM can be used to assess the longitudinal impact of a prevention program by comparing treatment and control populations with respect to individual differences in initial status and in rate of change; and (b) how predictors of initial status (post-intervention) and growth selected on the basis of a particular program theory can be incorporated in the model to explain program impact. Some advantages and limitations of using LGM in program evaluation are highlighted.     

Using client exit interviews to illuminate outcomes in program logic models: a case example

This paper describes how client exit interviews can assist human service administrators and workers to better understand the outcomes their programs are designed to accomplish. Specifically, the qualitative component of a demonstration family literacy program evaluation is used to illustrate how client input can be used to fine-tune the outcomes component of a program's logic model. An analysis of semi-structured exit interviews with 35 clients, who were randomly selected from all 89 served in the first year of the program, resulted in revision to the program's original logic model, creating explicit ‘testable’ pathways to achieving intended outcomes.     

Evaluation of a cross-sector community initiative partnership: Delivering a local sport program

Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders’ satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.     

Perceptions of participants and staff of implementing a physical activity program in rural primary care

Addressing the lack of physical activity (PA) in rural adults is vital because of the role it plays in the risk for many chronic diseases. The purpose of the study was to explore the feasibility of conducting a 12-week intervention utilizing a Fitbit to increase PA behavior in inactive rural adults recruited from a primary care clinic. Semi-structured interviews were conducted with study participants (n=10) and a focus group was held with nurses (n=7) from the primary care clinic. Data were analyzed using the process of immersion/crystallization. Participants and nurses both felt the program had a positive impact on PA and that the program was needed in the community. Study participants most often reported their favorite part of the study was being able to track their activity as well as the goal setting. However, study participants reported barriers to continued participation in PA related to a lack of time and ability to be active in cold weather. Further, nursing staff reported barriers in regard to a lack of resources (staffing and money) These barriers need to be examined further and addressed in order to implement a sustainable PA program that can be maintained through a primary care clinic.     

Evaluating the impact of a healthy corner store initiative on food access domains

Healthy Corner Store Initiatives (HCSIs) are one food access strategy that aims to improve food environments by supporting local food stores in expanding their healthy offerings. This study presents the evaluation results from one such HCSI. Fresh Foods Here evaluated its network of eight stores in Columbus, Ohio between November 2013 and August 2014. Data from invoices, inventories, rapid market assessments, and customers surveys were analyzed for evidence of impact on the service delivery and personal domains of food access. For some indicators, initial gains were concentrated between pre- and interim evaluation periods, followed by either a leveling off or decline between interim and post. However, overall results were promising, with increases noted in the number of healthy items ordered by store owners, in daily foot traffic and transactions, and in consumer confidence for certain healthy behaviors. Lessons learned concerning the operation and evaluation of HCSIs are shared.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

Evaluation of a multi-site program designed to strengthen relational bonds for siblings separated by foster care

Sibling relationships in foster care settings have received increased attention in recent years. Despite growing evidence regarding the protective potential of sibling relationships for youth in care, some sibling groups continue to experience foster care related separation, and few programs exist to address the needs of these youth.This study describes and evaluates Camp To Belong, a multi-site program designed to provide short-term reunification to separated sibling groups through a week-long summer camp experience. Using a pre-test post-test survey design, this paper examines changes in youth ratings of sibling conflict and sibling support across camps located in six geographically distinct regions of the United States. The effects of youth age, number of prior camp exposures, and camp location were tested using multilevel modeling procedures.Findings suggest that participation in Camp To Belong may reduce sibling conflict, and improvements in sibling support are noted for youth who have had prior exposure to the camp’s programming. Camp-level variance in the sibling support outcome highlight the complex nature of relationships for siblings separated by foster care, and suggest the need for additional research. Lessons learned from this multi-site evaluation and future directions are discussed.     

The Father Friendly Initiative within Families: Using a logic model to develop program theory for a father support program

The transition to fatherhood, with its numerous challenges, has been well documented. Likewise, fathers’ relationships with health and social services have also begun to be explored. Yet despite the problems fathers experience in interactions with healthcare services, few programs have been developed for them. To explain this, some authors point to the difficulty practitioners encounter in developing and structuring the theory of programs they are trying to create to promote and support father involvement (Savaya, R., & Waysman, M. (2005). Administration in Social Work, 29(2), 85), even when such theory is key to a program's effectiveness (Chen, H.-T. (2005). Practical program evaluation. Thousand Oaks, CA: Sage Publications). The objective of the present paper is to present a tool, the logic model, to bridge this gap and to equip practitioners for structuring program theory. This paper addresses two questions: (1) What would be a useful instrument for structuring the development of program theory in interventions for fathers? (2) How would the concepts of a father involvement program best be organized? The case of the Father Friendly Initiative within Families (FFIF) program is used to present and illustrate six simple steps for developing a logic model that are based on program theory and demonstrate its relevance.     

Presidential youth fitness program implementation: An antecedent to organizational change

IntroductionGrounded in organizational change theory, the purpose of this study was to investigate the efficacy of the Presidential Youth Fitness Program (PYFP) and its association with healthy cultures within schools.MethodsUsing a qualitative approach, data were collected through interviews, site visits and artifacts across 374 schools. An explanatory collective case study approach was used to identify key events related to implementation.ResultsPivotal antecedents to organizational change included prolonged, continual PD, direct support of PYFP implementation, and recognition. Further, three key themes of leveling of the playing field, strategically overcoming barriers, and recruiting teacher fitness champions were identified.ConclusionsCreating a healthy school culture was an unexpected, but feasible outcome stemming from the implementation of the PYFP. A collective effort, led by physical education teachers and fitness champions and embraced by the administration, faculty, and community, is necessary for the school culture to unfreeze from its present status.     

Defining,illustrating and reflecting on logic analysis with an example from a professional development program

Program designers and evaluators should make a point of testing the validity of a program's intervention theory before investing either in implementation or in any type of evaluation. In this context, logic analysis can be a particularly useful option, since it can be used to test the plausibility of a program's intervention theory using scientific knowledge. Professional development in public health is one field among several that would truly benefit from logic analysis, as it appears to be generally lacking in theorization and evaluation. This article presents the application of this analysis method to an innovative public health professional development program, the Health Promotion Laboratory. More specifically, this paper aims to (1) define the logic analysis approach and differentiate it from similar evaluative methods; (2) illustrate the application of this method by a concrete example (logic analysis of a professional development program); and (3) reflect on the requirements of each phase of logic analysis, as well as on the advantages and disadvantages of such an evaluation method. Using logic analysis to evaluate the Health Promotion Laboratory showed that, generally speaking, the program's intervention theory appeared to have been well designed. By testing and critically discussing logic analysis, this article also contributes to further improving and clarifying the method.     

Evaluability assessment of a community development leisure program in Spain

This paper presents the results of an evaluability assessment (EA) conducted in Spain. EAs are assessments that minimize the uncertainty and risk inherent in a subsequent evaluation (e.g. an impact or results assessment) that may require excessive resources. We designed our own EA model by combining three theoretical approaches, with a holistic and pedagogic vision of the process and a flexible methodology to enable adaptation to different programs and contexts, thus fostering transferability. The EA was carried using a qualitative methodology: documentary analysis, seven semi-structured interviews with stakeholders, eight direct observation sessions and a check-list for politicians and experts. The outcome report helped promote organizational learning, provided the program with a logical and assessable theoretical model, detected its strengths and helped overcome the barriers to efficient assessment. Based on these achievements, an evaluation of the results was recommended, with repeated pretest-posttest measurements and an equivalent control group.     

Cultivating knowledge of resiliency and reintegration among military youth through a national youth leadership program

Youth in military families are frequently challenged by the adjustment demands associated with the deployment and reintegration of a parent. A positive youth development approach was undertaken by the Boys and Girls Clubs of America to develop and implement a Military Teen Ambassadors (MTA) training for youth in military families that would facilitate knowledge of resiliency and reintegration and foster leadership skills to build assets for themselves as well as their peers within their local communities. To determine if MTA was functioning as intended and to refine future programming, this preliminary formative study assessed perceived participant learning outcomes associated with MTA on variables pertaining to knowledge acquisition, perceived skill acquisition, and community needs awareness. Data were collected prior to the training, immediately following the training, and 6 months after the training. Repeated measures analysis indicated significant mean increases over time in knowledge and awareness of resiliency and reintegration; perceived leadership skills; and community awareness. Qualitative findings provided triangulation in the aforementioned areas. These findings strengthen the body of knowledge on resiliency by demonstrating that the 7 Cs model may be an effective strategy to incorporate into leadership development programs seeking to build knowledge of resiliency among military youth. Study limitations, lessons learned, and recommendations for further research are delineated.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.     

Feasibility of bidirectional text messages in evaluating a text-based nutrition education program for low-income parents: Results from the Text2BHealthy program

Text messages are increasingly used in the delivery of health education programs. One appealing aspect of this approach is the possibility of remotely collecting participant data to use in program tailoring or evaluation. The purpose of the present study is to test the feasibility of using text messages to collect participant data. Using data from 33 texted evaluation questions sent through the Text2BHealthy nutrition education program for low-income parents (n = 108−1521) response rates under different incentive and prompting strategies were examined. Response rates are generally low across a pilot year and three program years, ranging from 10–55%. While incentives seemed to be ineffective at improving response rates, results indicate that prompting participants to respond may increase response rates. Individuals who respond to an initial question are highly likely to respond to a follow-up question (88–99%) and to report positive behaviors (68–100%). Responses received through text may be unrepresentative and positively biased. Text messages may be a supplemental data collection strategy in nutrition education programs, but low response rates and response bias undermine data quality.     

Utilizing the RE-AIM framework to determine effectiveness of a preschool intervention program on social-emotional outcomes

Social-emotional issues in preschoolers continue to be an area of concern across the nation. Models to determine effective implementation practices are needed. The Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) framework is one model that can be used to evaluate program implementation in preschool settings. The Jump Start program provided short-term intervention to children in low-income preschools from multiethnic backgrounds. The goal was to promote prosocial behaviors while minimizing problem behaviors. Various evidenced-based practices (i.e., I Can Problem Solve, play therapy, Pyramid Model) were utilized. 305 children from 73 childcare centers participated in the program. The majority of participants were from ethnic minority backgrounds and resided in high-poverty areas of the county. The RE-AIM framework was utilized to determine program outcomes. Results showed successes on each level of RE-AIM with an at-risk population. The Jump Start program significantly increased positive behaviors and decreased challenging behaviors in preschool-aged children. Results indicated medium to large effect sizes. One year following program participation, the majority of children who participated in the program were at decreased risk of special education services and expulsion.     

Maltreatment prevention through early childhood intervention: A confirmatory evaluation of the Chicago Child-Parent Center preschool program

Increased recognition of the consequences associated with child maltreatment has led to greater emphasis on its prevention. Promising maltreatment prevention strategies have been identified, but research continues to suffer from methodological limitations and a narrow focus on select prevention models. This investigation uses data from the Chicago Longitudinal Study to examine mediating mechanisms that link the Chicago Child-Parent Center preschool program to a reduction in overall child maltreatment and, more specifically, child neglect. We use structural equation modeling to test child, family, and school measures hypothesized to mediate the effects of CPC participation on maltreatment and neglect. Results indicate that a substantial proportion of the program's impacts can be accounted for by family support processes, including increased parent involvement in school and maternal educational attainment as well as decreased family problems. The CPC program's association with reduced school mobility and increased attendance in higher-quality schools also significantly mediated its effects on maltreatment and neglect. Further, a decrease in troublemaking behavior contributed modestly to mediating the program's association with maltreatment but not neglect. We discuss the implications of these results for the field of maltreatment prevention.