Pragmatic ex-ante evaluation using an innovative conceptual framework: The case of a high-tech entrepreneurship program for women

Program designers at local project level usually design interventions under conditions of scarce resources and bounded rationality lacking sufficient information about the process and impact mechanisms of the complex social programs they design and implement. This paper proposes a model for pragmatic ex ante evaluation using an innovative conceptual framework for practitioners working in the field to improve gender equality in research and innovation.The presented and applied model is pertinent to the theory-oriented evaluation tradition using a logic frame and a theory of change approach adapted to the special requirements of ex-ante evaluation. The model is illustrated by the case study of a program for the promotion of women entrepreneurship in the STEM fields.It is concluded that in a setting with limited resources and information, ex-ante evaluations should aim at defining clear conditions of proportionality and need to take into account the extent to which contextual factors hinder or facilitate the implementation of a program, and the specific institutional and system mechanisms at play, in order to enable proactive risk management from the beginning.     

Effects of Africentric Socialization on Psychosocial Outcomes in Black Girls: The Critical Role of Gender

African American social work scholars recommend rigorous research and evaluation of Africentric interventions as one approach to building culturally appropriate, evidence-based treatments for the Black community. Following this approach, this pilot study evaluates the Imani Rites of Passage program, a 15-week Africentric curriculum with 10 Black adolescent females (mean age = 12.40), comparing their pre- and post-intervention data to that of a no-intervention group of 13 Black males (mean age = 12.08). The program evaluation is based on the degree of correspondence between theory and the measurement model spelled out by the cognitive-cultural view of African-American identity. The findings for Black females were consistent with the cognitive-cultural model in some ways but not in others, and they were more complex than the predictions. Future Africentric interventions guided by the cognitive-cultural model must consider gender differences in the outcomes. Implications for social work research and practice are discussed.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

Methodological strategies for evaluating youth gang prevention programs

This article reflects on common challenges and lessons learned during the evaluation of gang prevention programs based on case studies of three federally funded Canadian programs. Elements of evaluation design, implementation, data analysis and reporting of results are discussed. More specifically, the article highlights issues that occur when evaluating community projects focused on preventing extreme risks for violence and the complexity of working in potentially dangerous and/or unstable work environments. Topics covered include the problem with quasi-experimental designs, model fidelity adherence, program documentation, client recruitment and retention, and data collection. Recommendations are provided to improve evaluations of youth gang prevention programs and similar community-based interventions that focus on the prevention of youth violence.     

Issues in the Evaluation of an Online Prevention Exercise

ABSTRACT

The rapid growth of computer and Internet technologies and their acceptance as a viable medium for human services delivery has opened up new possibilities for providing effective prevention programming. This article reviews some of the major online prevention websites and evaluations of their effectiveness. In addition, it reports the struggle to evaluate an interactive multimedia anger management exercise that is part of a teen substance abuse prevention website. The evaluation provided some support that a 30-minute exposure to a web-guided prevention exercise could increase teens' prevention knowledge and that completing the online exercise as supplemental homework reinforces the classroom experience. However, positive changes in other measures of behavior change were not supported. The study pointed to many issues that occur when developing and evaluating online interventions in real-world environments.

This background section presents the advantages of online prevention, a brief overview of prevention research and its advantages, and an overview of prevention websites and their evaluation.     

Evaluation of community-based AIDS education and risk reduction projects in ethnic and racial minority communities: A survey of projects funded by the U.S. public health service

The purpose of this study was to describe proposed program evaluation methods used to determine effectiveness of selected federally funded AIDS education and risk reduction interventions focused on ethnic and racial minority populations. Funding for AIDS prevention programs has grown significantly since 1985, when the first allocation of $11 million was earmarked for general AIDS prevention and education. In 1987, funds specifically targeting minority populations were set aside from AIDS prevention and education programs. Nine agencies/divisions within the Department of Health and Human Services (DHHS) were identified as having funded AIDS prevention and education projects targeted toward ethnic and racial minority populations. The grants and contracts reviewed for this study were limited to those programs receiving Federal funding through DHHS. Few programs provided information within the broader context of sex education and drug education. The modification of needle-sharing behavior among IV drug users and unprotected sex with an IV drug user were the two most frequent risk factors targeted. All 63 programs used at least one formative evaluation method. The most frequently used formative techniques were monitoring achievement of project objectives and counting the number of people participating in program activities. Forty programs counted the amount of AIDS literature distributed. Thirty-three programs conducted a baseline assessment of AIDS knowledge, and 33 programs conducted a baseline assessment of risk behaviors for HIV infection. Results from this study support the National Academy of Sciences recommendation that while summative evaluation will ultimately be valuable, it would be premature to begin developing outcome evaluation strategies at the present time. It is incumbent upon program planners and evaluators to ensure that premature program evaluations are not made in an effort to eliminate direct federal funding of minority community-based strategies to stop the spread of HIV infection.     

Suicide prevention on college campuses: What works and what are the existing gaps? A systematic review and meta-analysis

Abstract

Objective: To examine the effects of universal and targeted suicide prevention programs on relevant outcomes in college campuses. Methods: College suicide prevention programs published from 2009 to 2018 were assessed on outcomes including knowledge, skills, self-efficacy, suicidal ideation, and suicidal behaviors. Effects of the interventions on outcome variables with sufficient studies to warrant meta-analysis (ie, knowledge, skills, and self-efficacy) were meta-analyzed. Studies reporting on the remaining outcomes (ie, suicidal ideation and behaviors) were systematically reviewed. Results: Significant increases in suicide prevention knowledge, skills, and self-efficacy were observed in universal prevention interventions that typically employed gatekeeper prevention strategies. Evidence of reductions in suicidal ideation and behaviors was observed across targeted suicide prevention programs for at-risk students. Conclusion: Prevention programs are beneficial for training those likely to come in contact with people endorsing suicidality, but further research is needed to show that suicide interventions can consistently have significant effects on suicidal students as well.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

PEGASE Program: Identification of babies in the child protection system at risk of developmental delays or disorders and the implementation of a ‘supra-optimal’ care pathway

The importance of the first three years of life for children's development and the consequences for their general health in adulthood have been widely documented in the international literature, including the benefits of early identification and intervention programs. Additionally, convergent research has highlighted wide-ranging negative trajectories attributable to both the short and long-term consequences of abuse and neglect and their neurobiological impact on health and development. However, formal identification processes and the use of standardised screening tools in child protection remain relatively rare despite the potential societal impact of child maltreatment in terms of public and economic health. Moreover, delays between the initial identification of mental health problems and health and developmental disorders, and the introduction of appropriate interventions persist. Abused and neglected infants risk an exacerbation of the consequences of early maltreatment due to insufficient access to prompt assessment and treatment, including appropriate responses to their health needs. The aim of this article is to present the PEGASE program, an innovative French program for very young children in care, which is modelled on the care of premature babies. By setting up an early care pathway, it aims to ensure adequate medical follow-up – both somatic and psychiatric – through standardised examination and tailored interventions in order to limit the adverse health and developmental consequences of abuse. The presentation of the PEGASE program is supplemented by a case study.     

Evaluation of a cross-sector community initiative partnership: Delivering a local sport program

Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders’ satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.     

Tailored program evaluation: Past, present, future

This paper discusses measurement issues related to the evaluation of computer-tailored health behavior change programs. As the first generation of commercially available tailored products is utilized in health promotion programming, programmers and researchers are becoming aware of the unique challenges that the evaluation of these programs presents. A project is presented that used an online tailored health behavior assessment (HBA) in a worksite setting. Process and outcome evaluation methods are described and include the challenges faced, and strategies proposed and implemented, for meeting them. Implications for future research in tailored program development, implementation, and evaluation are also discussed.     

Prevention of Child Sexual Abuse by Targeting Pre-Offenders Before First Offense

The population of potential child abuse offenders has largely been unstudied. In the current study, we examine whether a six-component model used for primary diabetes prevention could be adapted to child sexual abuse pre-offenders, whereby individuals who are prone to sexual abuse but have not yet committed an offense can be prevented from committing a first offense. The six components include: define and track the magnitude of the problem, delineate a well-established risk factor profile so that at-risk persons can be identified, define valid screening tests to correctly rule in those with the disease and rule out those without disease, test effectiveness of interventions—the Dunkelfeld Project is an example, produce and disseminate reliable outcome data so that widespread application can be justified, and establish a system for continuous improvement. By using the diabetes primary prevention model as a model, the number of victims of child sexual abuse might be diminished.     

Theory and method in treatment evaluation

We outline a conceptual model of evaluation research that has both guided and flowed from our research on the process and outcome of treatment for alcohol abuse and for depression. The conceptual model encompasses three basic processes: the process of treatment selection, the process of treatment, and the process underlying the onset and course of the disorder or problem being treated. A theory guiding a program evaluation can focus on one or more of these processes, and findings from evaluations can contribute to the development of better theories of these processes. We discuss how theories can guide both experimental and naturalistic evaluations, as well as aid in the estimation of treatment effects with either type of design. We then consider the role of theory and method in exploring patient-treatment matching. Finally, we discuss the implications of theory-guidance for the utility of evaluations—particularly the utility for evaluation researchers.     

Evaluation considerations for community-based gender-informed health interventions

Evaluations of gender-based interventions have been consistently criticized for their lack of methodological rigor. This is largely due to the complex design of many of the interventions, coupled with difficulties in measuring the outcome and impact of these interventions. This article proposes a number of ways to improve these evaluations both at the community and individual level. We recommend use of organizational theory and narrative inquiry methods, such as the appreciative inquiry technique, to examine how communities design gender-based interventions. In addition, we suggest a variety of methods to measure the effects of these interventions on gender norms in the community for example, policy analysis, multilevel modeling, and social conversations. With respect to measuring outcomes at the individual level, we argue for more rigorous evaluation designs in order to improve internal and external validity claims. Additionally, we suggest that evaluations should incorporate different methodologies, for example autobiographical narratives, which allows one to give saliency to the subjective voices of participants. Finally, we emphasize that evaluation designs need to document the long term effects of intervention programs and define the expected outcomes with greater specificity.     

Impact of a classroom behavior management intervention on teacher risk ratings for student behavior

Classroom behavior management interventions have been used successfully with drug prevention programs to prevent subsequent antisocial behavior and substance use among youth. This article presents results from implementation of the All Stars Challenge, a classroom-based behavior management component to a drug prevention program for fifth graders. Risk ratings for shyness and lack of awareness of social norms among high-risk students who received the All Stars Challenge were reduced compared with fifth graders who did not receive the intervention. In contrast, physical and social aggressivity among low-risk students who received the program increased when compared to similar control students.     

Challenges for environmental education evaluation

The articles in this special issue cover a range of practices in environmental education evaluation, from program evaluations to training and organizational impact. This article reflects on this collection and offers six recommendations on three broad themes that the issue's authors identify as shortcomings or opportunities for change: capacity building, program theory, and learning organizations.     

Systematic review of primary prevention human papillomavirus interventions targeting college students

ABSTRACT. Objectives: The purpose of this research was to systematically analyze human papillomavirus (HPV) interventions targeting college students. Methods: Completed interventions targeting general HPV prevention, HPV vaccine uptake, or antecedents of HPV vaccination uptake that used any quantitative design, published from January 2000 to January 2014 in the English language, were reviewed. Results: Twelve interventions met the criteria. Ten interventions produced 1 or more significant outcomes on primary or secondary variables of interest. Conclusions: Efficacy of interventions can be enhanced through tailoring materials and pretesting by sex, operationalizing theories, incorporating process evaluations, employing HPV vaccine uptake outcome measures, and utilizing Web-based delivery of programs.     

Child Sexual Abuse in Zimbabwe

Although child sexual abuse is a significant public health problem globally, its incidence, prevention, and management is less well described in resource-poor settings. In poorer settings prevention initiatives assume even more importance since resources for managing abused children are severely limited. This article examines the current status of policy and practice related to the prevention of child sexual abuse in Zimbabwe. It identifies implementation challenges and highlights opportunities that could be embraced to reduce CSA in Zimbabwe, based on evidence synthesized from recent work. Although Zimbabwe has a well-established legal and regulatory framework to protect children from child sexual abuse, implementation of existing policies is weak. Financial, human, and material resource constraints are frequently cited to explain limited prevention activity. Effective strategies for the prevention of child sexual abuse should focus on implementing existing legislation, targeting schoolchildren, and getting community involvement. A dedicated budget would help entrench these strategies, but gains can be achieved even in the absence of this.     

The Effectiveness of Educational Programs to Improve Recognition and Reporting of Elder Abuse and Neglect: A Systematic Review of the Literature

Health professionals often lack adequate protocols or knowledge to detect, manage, and prevent elder maltreatment. This systematic review describes and evaluates existing literature on the effectiveness of educational interventions to improve health professionals' recognition and reporting of elder abuse and neglect. Fourteen articles described 22 programs ranging from brief didactics to experiential learning and targeted a variety of health and social service audiences. Most evaluations were limited to satisfaction measures. These programs may result in increased awareness, collaboration, and improved case finding. However, using the published literature to guide new program planning is constrained by lack of details and limited evaluations.