An introduction to the Safe Schools/Healthy Students Initiative

The Safe Schools/Healthy Students (SS/HS) Initiative offers a unique opportunity to conduct large-scale, multisite, multilevel program evaluation in the context of a federal environment that places many requirements and constraints on how the grants are conducted and managed. Federal programs stress performance-based outcomes, valid and reliable data, addressing important problems, ensuring efficiency and fiscal responsibility, reducing burden on federal staff and grantees, and developing and disseminating useful solutions and recommendations. MANILA Consulting Group, Inc., (MANILA), in partnership with Battelle Centers for Public Health Research and Evaluation (Battelle) and RMC Research Corporation (RMC), has been conducting the SS/HS national cross-site evaluation, which involves the coordinated efforts of federal Project Officers, local education agencies, technical assistance providers, communication specialists, and national and local evaluators across a diverse set of socioeconomic and cultural contexts. To date, the national cross-site evaluation has provided data indicating that the SS/HS Initiative is, in fact, meeting these goals. Findings revealed that fewer students reported they had experienced violence and fewer students reported they had witnessed violence. Fully 96 percent of school staff said SS/HS had improved school safety. There was a 263 percent increase in the number of students who received school-based mental health services and a 519 percent increase in those receiving community-based mental health services. In addition, more than 80 percent of school staff reported that they saw reductions in alcohol and other drug use among their students. These encouraging results stress the need for ongoing coordination at all levels of the Initiative to continue to ensure safer schools and healthier students. This article provides an overview of the initiative and introduces four articles in this special issue.     

A national evaluation of Safe Schools/Healthy Students: outcomes and influences

The Safe Schools/Healthy Students (SS/HS) Initiative has awarded over $2 billion in grants to more than 350 school districts in partnership with local mental health, law enforcement, and juvenile justice agencies. To estimate the impact of grantee characteristics, grant operations, and near-term outcomes in reducing violence and substance use, promoting mental health, and enhancing school safety, logged odds ratios (LORs) were calculated contrasting Year 3 with Baseline performance from grantee-provided data on seven outcome measures. After comparing grantee performance across outcomes and outcomes across grantees, the LORs were entered as dependent variables in a series of meta-regressions in which grantee characteristics, grant operations, and near-term outcomes were tested after controlling for pre-grant characteristics. Findings indicate that the SS/HS Initiative significantly improved most outcomes, that within-grantee performance varied greatly by outcome, and that random-effects meta-regression appreciably decreased the variance available for modeling. The approach demonstrates that the SS/HS Initiative is effective and that locally collected performance data can be used to estimate grantee success in improving youth outcomes.     

A mixed-method exploration of functioning in Safe Schools/Healthy Students partnerships

This paper presents a mixed-method approach to measuring the functioning of Safe Schools/Healthy Students (SS/HS) Initiative partnerships. The SS/HS national evaluation team developed a survey to collect partners’ perceptions of functioning within SS/HS partnerships. Average partnership functioning scores were used to rank each site from lowest to highest. Sites with the most favorable perceptions of partnership functioning were defined as having average scores in the top 10% (n = 10) and sites with the least favorable perceptions of partnership functioning were defined as having average scores in the bottom 10% (n = 10). Qualitative data for these 20 sites were inductively open coded for emergent themes and analyzed for patterns using grounded theory approach. Six themes emerged that distinguished sites reporting the most favorable and least favorable perceptions of partnership functioning: partner engagement, facilitators, barriers, shared decision making, partnership structure, and sustainability. Sites reporting the most favorable perceptions of partnership functioning effectively utilized collaboration processes that facilitate coalition building, such as shared decision making, effective communication, and developing a clearly defined structure. Qualitative themes from this analysis provide evidence of validity for the partnership functioning scale used and illustrate distinguishing features between sites with the most favorable and least favorable perceptions of partnership functioning.     

A novel approach for evaluating programs designed to serve military veterans: Using an adapted common components analysis

Evaluators are challenged to keep pace with the vast array of Veteran support programs operating in the United States, resulting in a situation in which many programs lack any evidence of impact. Due to this lack of evidence, there is no efficient way to suggest which programs are most effective in helping Veterans in need of support. One potential solution to this dilemma is to reconceptualize program evaluation, by moving away from evaluating programs individually to evaluating what is common across programs. The Common Components Analysis (CCA) is one such technique that aggregates findings from programs that have undergone rigorous evaluation at the level of program components (e.g., content, process, barrier reduction). Given that many Veteran programs lack outcome evidence from rigorous studies, an adaptation to CCA is needed. This report examines cross-sectional data from a pilot study using an adapted CCA across five domains of well-being (i.e., employment, education, legal/financial/housing, mental/physical health, and social/personal relationships). The purpose of this preliminary study is to determine the feasibility of eliciting program nominations and program components from Veterans via an online survey. When coupled with a longitudinal research design, this adaptation to CCA will allow for stronger causal claims about the expected impact of different program components within and across a variety of domains.     

Striking a Balance between Program-Specific and Portfolio-Level Evaluation: Lessons Learned from a Multi-Site Evaluation on the Texas-Mexico Border

Funders are increasingly making strategic investments across multiple grantees, aiming for their portfolio to improve targeted outcomes in a specific issue area. To this end, funders might use multi-site evaluation (MSE) approaches to examine the impact of their collective investments. However, it is important to recognize that each program—and its own program evaluation—must be tailored to its setting, population, and local context to best meet the needs of its target population. Therefore, multi-site evaluations need to account for this complexity. This paper describes the Sí Texas project, a large initiative of eight grantees implementing different integrated behavioral health models to improve physical and mental health outcomes along the Texas-Mexico border. With over 4,200 MSE study participants, the evaluation for Sí Texas used a partnership-centered approach to both enhance the evidence base and build local organizational capacity. This paper describes this approach, the process of tailoring evaluation practices to the grantees’ context, and the challenge of balancing consistency at the grantee-level for the portfolio multi-site evaluation. Successes, challenges, and lessons learned related to study design, data collection, grantee partnership, and capacity building are discussed.     

Harmonizing databases? Developing a quasi-experimental design to evaluate a public mental health re-entry program

Our study is the first-ever initiative to merge administrative databases in Massachusetts to evaluate an important public mental health program. It examines post-incarceration outcomes of adults with serious mental illness (SMI) enrolled in the Massachusetts Department of Mental Health (DMH) Forensic Transition Team (FTT) program. The program began in 1998 with the goal of transitioning offenders with SMI released from state and local correctional facilities utilizing a core set of transition activities. In this study we evaluate the program's effectiveness using merged administrative data from various state agencies for the years 2007-2011, comparing FTT clients to released prisoners who, despite having serious mental health disorders, did not meet the criterion for DMH services. By systematically describing our original study design and the barriers we encountered, this report will inform future efforts to evaluate public programs using merged administrative databases and electronic health records.     

Harmonizing databases? Developing a quasi-experimental design to evaluate a public mental health re-entry program

Our study is the first-ever initiative to merge administrative databases in Massachusetts to evaluate an important public mental health program. It examines post-incarceration outcomes of adults with serious mental illness (SMI) enrolled in the Massachusetts Department of Mental Health (DMH) Forensic Transition Team (FTT) program. The program began in 1998 with the goal of transitioning offenders with SMI released from state and local correctional facilities utilizing a core set of transition activities. In this study we evaluate the program's effectiveness using merged administrative data from various state agencies for the years 2007–2011, comparing FTT clients to released prisoners who, despite having serious mental health disorders, did not meet the criterion for DMH services. By systematically describing our original study design and the barriers we encountered, this report will inform future efforts to evaluate public programs using merged administrative databases and electronic health records.     

The Father Friendly Initiative within Families: Using a logic model to develop program theory for a father support program

The transition to fatherhood, with its numerous challenges, has been well documented. Likewise, fathers’ relationships with health and social services have also begun to be explored. Yet despite the problems fathers experience in interactions with healthcare services, few programs have been developed for them. To explain this, some authors point to the difficulty practitioners encounter in developing and structuring the theory of programs they are trying to create to promote and support father involvement (Savaya, R., & Waysman, M. (2005). Administration in Social Work, 29(2), 85), even when such theory is key to a program's effectiveness (Chen, H.-T. (2005). Practical program evaluation. Thousand Oaks, CA: Sage Publications). The objective of the present paper is to present a tool, the logic model, to bridge this gap and to equip practitioners for structuring program theory. This paper addresses two questions: (1) What would be a useful instrument for structuring the development of program theory in interventions for fathers? (2) How would the concepts of a father involvement program best be organized? The case of the Father Friendly Initiative within Families (FFIF) program is used to present and illustrate six simple steps for developing a logic model that are based on program theory and demonstrate its relevance.     

Evaluating a community based mental health programme in West Bengal,India: Description of the methodology and lessons learned

Mental disorders impose an enormous burden on society. In developing countries like India, there is a lack of adequate number of trained mental health professionals to provide specialized care and 75–85 % of affected individuals do not have access to appropriate mental health services. The National Mental Health Programme (NMHP) is being implemented by the Government of India to support state governments in providing mental health services in the country. The Urban Mental Health Programme (UMHP) is a pilot initiative that has attempted the integration of mental health services in primary health care settings in two municipal wards in Kolkata, West Bengal, India. The overarching aim of this paper is to describe the methodology used for the evaluation of the community based mental health programme and to understand the processes of the programme in terms of barriers and facilitators. The current evaluation is based on a concurrent nested design, where qualitative and quantitative data are both collected at the same time but analysed separately and priority was given to qualitative data. This experience will contribute in helping other researchers to make some evaluations more effective, useful and manageable. Ethics approval was obtained from an institutional ethics committee of an organization (Ekjut) based in Ranchi, Jharkhand, India. The evaluation was undertaken by the George Institute for Global Health, New Delhi from February- June 2016.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

Evaluation of a cross-sector community initiative partnership: Delivering a local sport program

Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders’ satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.     

Developing a public health policy-research nexus: An evaluation of Nurse Practitioner models in aged care

A frustration often expressed by researchers and policy-makers in public health is an apparent mismatch between respective priorities and expectations for research. Academics bemoan an oversimplification of their work, a reticence for independent critique and the constant pressure to pursue evaluation funding. Meanwhile, policy-makers look for research reports written in plain language with clear application, which are attuned to current policy settings and produced quickly. In a context where there are calls in western nations for evidence based policy with stronger links to academic research, such a mismatch can present significant challenges to policy program evaluation. The purpose of this paper is to present one attempt to overcome these challenges. Specifically, the paper describes the development of a conceptual framework for a large-scale, multifaceted evaluation of an Australian Government health initiative to expand Nurse Practitioner models of practice in aged care service delivery. In doing so, the paper provides a brief review of key points for the facilitation of a strong research-policy nexus in public health evaluations, as well as describes how this particular evaluation embodies these key points. As such, the paper presents an evaluation approach which may be adopted and adapted by others undertaking public health policy program evaluations.     

The opportunities and challenges of multi-site evaluations: Lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

Defining,illustrating and reflecting on logic analysis with an example from a professional development program

Program designers and evaluators should make a point of testing the validity of a program's intervention theory before investing either in implementation or in any type of evaluation. In this context, logic analysis can be a particularly useful option, since it can be used to test the plausibility of a program's intervention theory using scientific knowledge. Professional development in public health is one field among several that would truly benefit from logic analysis, as it appears to be generally lacking in theorization and evaluation. This article presents the application of this analysis method to an innovative public health professional development program, the Health Promotion Laboratory. More specifically, this paper aims to (1) define the logic analysis approach and differentiate it from similar evaluative methods; (2) illustrate the application of this method by a concrete example (logic analysis of a professional development program); and (3) reflect on the requirements of each phase of logic analysis, as well as on the advantages and disadvantages of such an evaluation method. Using logic analysis to evaluate the Health Promotion Laboratory showed that, generally speaking, the program's intervention theory appeared to have been well designed. By testing and critically discussing logic analysis, this article also contributes to further improving and clarifying the method.     

Enabling careers, autonomy, and prosperity: Using community organizing and building approaches to improve the educational outcomes of people with mental illness

Objectives: The objective of this regional initiative was to develop access to educational opportunities for people with mental illness with a view to ultimately advancing their career prospects. Participants: The initiative engaged a broad range of community stakeholders including people with mental illness, their families, educators, mental health service providers and, policy analysts. Methods: The initiative used community organizing and development strategies to develop solutions to problems related to access to education. Results: The initiative was successful in mobilizing community participation, identifying priorities, and translating these priorities into action plans. Working groups of community stakeholders engaged in initiatives related to improving access to resources to support education, developing training for teachers in secondary schools, creating peer support systems, and developing a pilot supported education program as a partnership between a college and mental health service. Conclusion: Organized community building provided a foundation for a broad range of initiatives meant to improve access to educational opportunities for people with mental illness. Evaluation efforts will need to focus on the extent to which these initiatives ultimately ledto positive changes in the careers of people with mental illness.     

Evaluation of Yoga and Meditation Trainings with Adolescent Sex Offenders

This project was a program evaluation of an innovative approach to working with adolescent sex offenders at a human service agency in the Intermountain West. A literature review shows little content on the use of yoga and meditation instruction in general, and no content on the use of such methods with adolescent sex offenders. There is evidence, however, that such techniques can contribute to the improved physical and mental health of adults. Adolescent participants, their parents, and their trainers in the program were interviewed. Results of the interviews are summarized, and recommendations are forwarded.     

Identity and Boundary Tensions When Educators and Mental Health Professionals Collaborate: Between Wish and Reality

Schools are centers of collaboration in promoting the development and learning of children and adolescents. They seek to develop cultures of collaboration not only within their walls but also among constituencies with similar interests. These efforts, unfortunately, all too often prove frustrating and disappointing to all concerned. Yet, the need for interdisciplinary inquiry is greater today than ever before. This is particularly true considering the complex, diverse, and multifaceted nature of today's schools. No one professional can address all of the problems confronting education, particularly those of urban schools. For decades mental health and education professionals have joined together around their similar interest in promoting children and adolescent development. One such initiative is a program that fosters educators' development. This program, named the Teacher Education Program, began under the auspices of the Chicago Institute for Psychoanalysis and evolved into the Human Development and Learning Program when a partnership was formed with DePaul University's School of Education. From the program's 34-year history, the authors highlight specific tensions they experienced as directors of this collaborative initiative, and how these challenges either were or were not anticipated and/or addressed. The aims are to put a human face on the challenges and struggles encountered in the relationship between education and mental health professionals, and to identify common ground between educational and psychoanalytic theories and practices.