Social Service Research and Religion

Abstract

Development of accountability standards to demonstrate cause and effect relationships are gaining rapid advancement in the field of social sciences. Many governmental agencies, foundations, and other funders have developed approaches that require organizations to utilize science-based programs and incorporate evaluative methods to show improved outcomes and cost benefits to society. This article will examine the need for increased accountability in developing effective interventions by faith-based organizations in the delivery of social service interventions. Recently, there has been a strong movement toward governmental funding for faith-based institutions to provide social services, although there has been inadequate scientific data to demonstrate that approaches implemented are effective in meeting needs or yielding favorable outcomes. Similarly, many faith-based organizations provide innovative and effective programs that could serve as model programs if there was appropriate empirical evidence. This article will discuss how rigorous evaluative approaches such as randomized clinical control trials can produce scientific data on program effectiveness. We will use a case example in the field of drug and alcohol treatment to illustrate these points.     

Predicted outcomes in a national sample: A source of comparison for program evaluation

The lack of comparison groups is a common problem for researchers conducting evaluations of small, specialized programs. When conventional comparison groups are not available, the authors suggest a possible alternative approach which involves: (a) locating a sub-group in a national study for whom relevant variables have been measured; (b) generating a statistical model which predicts the outcome of interest in the national data; and (c) applying the predictive model to program data, in order to estimate outcomes in the absence of progam intervention. The authors illustrate this new application by presenting the results of an analysis of data from Project Talent, a national manpower study, and discuss its application to the evaluation of a medical training program in New York City.     

Evaluations of the special supplemental program for Women,infants, and children's (WIC's) effectiveness

The U.S. General Accounting Office (GAO) analyzed the evaluations of the Special Supplemental Program for Women, Infants, and Children (WIC) to determine whether they support assertions that have been made about the program's positive effects for five outcomes. Insufficient evidence was found for making any general or conclusive judgments about WIC's effectiveness overall, but in a limited way, the information does indicate the likelihood that WIC has positive effects in some areas. GAO also assessed problems in the state of evaluation efforts. That the program evaluations do not reveal whether WIC is having the effect intended by the legislation underscores the need to design and implement better studies. Generally speaking, GAO believes that if the lessons learned from past efforts are heeded, it should be possible to produce information on the overall effectiveness of WIC.     

Mentoring Policy Research Organizations: Project Evaluation Results

The international community has identified a positive link between good governance and economic development. There is an increasing appreciation of the effective role that local policy research organizations (PROs) in transition and developing countries can play in providing evidence-based policy recommendations as the basis for sound legislation and in assessing the efficacy of existing programs. Capacity building programs for PROs are seen as a key component of growth strategies. The donor community has responded with support for such efforts, which take many forms. On the other hand, the evaluation of the effectiveness of these capacity building and mentoring programs has been inadequate. This article presents the evaluation of one program, focusing on two important capacity building program outcomes: improved research capacity and greater effectiveness in the policy arena. A reflexive design is used. The results suggest the program??s approach is promising.     

How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.     

Looking Inside the “Black Box”

Abstract

The continued devolution of social welfare systems and services in the U.S. resultsin high stakes program evaluations in the field of family support and early intervention. Programs are expected to utilize evidence-based interventions and to demonstrate effectiveness. A look at implementation helps to differentiate between theories that do not work and programs that are not effective. Methods for identifying program implementation are needed. In a 17-site program evaluation, the author and her colleagues developed a methodology for measuring implementation and demonstrated the effects of differential implementation in understanding program outcomes.     

The Impact of Neighborhood Factors on Mental Health and Academic Outcomes for Adolescents Participating in the Prodigy Program

The Prodigy Cultural Arts program provides art classes embedded with three types of self-regulation skills (problem solving, social skills, and anger management) in an after school community setting to at-risk youths. Empirical evidence supports the effectiveness of self-regulation skills programs and the arts in programs for at-risk youths, yet the impact of the environment on the youth participants is often ignored. This study examines neighborhood structural characteristics along with previously identified individual, mental health variables in school outcomes for adolescents attending the Prodigy Program. Findings suggest that neighborhood structural characteristics have some influence in program outcomes. Limitations of the study and future research are discussed.     

Conflict of interest in the evaluation and dissemination of "model" school-based drug and violence prevention programs

Conflict of interest refers to a set of conditions in which professional judgment concerning the validity of research might be influenced by a secondary competing interest. The competing interest that has received most attention in the literature addressing the prevalence and effects of such conflicts on the practice of empirical research has been that of financial relationships between investigators and research sponsors. The potential for conflicts of interest to arise in the evaluation of drug prevention programs was raised by Moskowitz in this journal in 1993, but to date there has been no attempt made to estimate the scope of this problem. The present study addressed this issue using a sample of “model” school-based drug and violence prevention interventions by first, identifying the types or relationships that exist between program developers and program distributors, and, second, by assessing how many of the evaluations of these programs published in peer-reviewed journals had been conducted by the developers of the programs compared to independent evaluation teams. The data presented indicate that there are relatively few published evaluations that do not involve program developers and that there are few instances in which there is complete separation between the program developer and program distributor. Using the open systems model of the Institute of Medicine Committee on Research Integrity as a framework, it is argued that the culture and norms of the program developer and those of the program evaluator are fundamentally distinct and therefore failure to separate these roles produces high potential for conflict of interest to arise.     

Zero effects in substance abuse programs. Avoiding false positives and false negatives in the evaluation of community-based programs

The evaluation of community-based programs poses special design and analysis problems. The present article focuses on two major types of errors that can occur in such evaluations: false positive--incorrectly declaring a program to be effective--and false negatives--incorrectly declaring a program to be ineffective. The evaluation of a national demonstration of community-based programs to reduce substance abuse, Fighting Back, is used to illustrate several approaches to reduce the probability of errors. Both those errors that are affected by the design and those by analytic approaches are considered. Ways to assess multiple outcomes and to match the complexity of the program with design and analytic strategies are proposed. Community trials are complex interventions, and, although they can provide very useful information, their outcomes have to be understood in terms of the constructs they test and the contexts within which they are carried out.     

Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program

In 2005, state fish and wildlife agency wanted to examine how one of its conservation education programs was providing science-based understanding and outdoor experiences by evaluating students’ knowledge, skills, attitudes and intended behavioral outcomes related to fish, fishing and aquatic habitats in Montana. A key factor in this study was the acceptance by program stakeholders to conduct the evaluation using a utilization-focused evaluation approach to promote usability and accuracy of evaluation results. Using a quasi-experimental non-equivalent group design, more than 2000 students in participating classrooms throughout Montana received a pre-survey, post-survey and an extended post-survey; 114 teachers participated in an Internet survey and 16 program instructors took part in a structured open-ended telephone interview. The participatory approach and mixed methods enhanced abilities to interpret results of student surveys in particular. The user-focused approach was discovered to be personal and situational, allowed the facilitation of the evaluation process with consideration for increased application of evaluation findings and implementation of recommendations from beginning to end. Further development of evaluation pathways is needed to more effectively evaluate outcomes and implement practical and transferable measures to determine if environmental education activities produce desired participant outcomes.     

Puberty Education in a Global Context: Knowledge Gaps,Opportunities, and Implications for Policy

Youth programs and policies provide opportunities for institutions and societies to support healthy adolescent development. Puberty education programs are universally important, as they provide crucial knowledge and skills to help youth and their caregivers navigate the physical, emotional, and interpersonal changes of puberty with positive outcomes. However, few puberty programs have been rigorously evaluated, resulting in a lack of evidence‐based knowledge and practice in this area. This review examines the status of research on puberty education and related programs and draws on the broader intervention literature and recent research findings on adolescence to identify program features that might improve program effectiveness. Implications for policy are also discussed. The need for rigorous program evaluation is emphasized throughout.     

A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.     

Prevention interventions of alcohol problems in the workplace

The workplace offers advantages as a setting for interventions that result in primary prevention of alcohol abuse. Such programs have the potential to reach broad audiences and populations that would otherwise not receive prevention programs and, thereby, benefit both the employee and employer. Researchers have implemented and evaluated a variety of workplace alcohol problem prevention efforts in recent years, including programs focused on health promotion, social health promotion, brief interventions, and changing the work environment. Although some studies reported significant reductions in alcohol use outcomes, additional research with a stronger and integrated methodological approach is needed. The field of workplace alcohol prevention also might benefit from a guiding framework, such as the one proposed in this article.     

Interventions that Foster Healing Among Sexually Exploited Children and Adolescents: A Systematic Review

Sexual exploitation of children and adolescents is a pressing, yet largely under-recognized form of child abuse. The goals of this review were to identify interventions that have been implemented with sexually exploited children and adolescents and to better understand their effectiveness for fostering healing with this population. Our systematic search generated 4,358 publications of which 21 met our inclusion criteria. Based on their objectives and delivery, we organized the interventions into five categories: (a) focused health and/or social services, (b) intensive case management models, (c) psychoeducational therapy groups, (d) residential programs, and (e) other. Most programs were gender-specific, targeting girls and young women with just one being for boys and young men only. Studies reported on a range of outcomes including psychosocial outcomes, risky behaviors, trauma responses, mental health, protective factors, and public health outcomes. Despite differences in delivery, most of the interventions did, to some degree, appear to foster healing among sexually exploited children and adolescents. Findings from this review have implications for researchers, policy and program developers, and frontline practitioners who, through partnerships with one another, can advocate for the creation of evidence-informed, purpose-built, and thoughtfully delivered interventions.     

Effects of Africentric Socialization on Psychosocial Outcomes in Black Girls: The Critical Role of Gender

African American social work scholars recommend rigorous research and evaluation of Africentric interventions as one approach to building culturally appropriate, evidence-based treatments for the Black community. Following this approach, this pilot study evaluates the Imani Rites of Passage program, a 15-week Africentric curriculum with 10 Black adolescent females (mean age = 12.40), comparing their pre- and post-intervention data to that of a no-intervention group of 13 Black males (mean age = 12.08). The program evaluation is based on the degree of correspondence between theory and the measurement model spelled out by the cognitive-cultural view of African-American identity. The findings for Black females were consistent with the cognitive-cultural model in some ways but not in others, and they were more complex than the predictions. Future Africentric interventions guided by the cognitive-cultural model must consider gender differences in the outcomes. Implications for social work research and practice are discussed.     

Diversity training for the community aged care workers: A conceptual framework for evaluation

Older Australians are an increasingly diverse population, with variable characteristics such as culture, sexual orientation, socioeconomic status, and physical capabilities potentially influencing their participation in healthcare. In response, community aged care workers may need to increase skills and uptake of knowledge into practice regarding diversity through appropriate training interventions. Diversity training (DT) programs have traditionally existed in the realm of business, with little research attention devoted to scientifically evaluating the outcomes of training directed at community aged care workers. A DT workshop has been developed for community aged care workers, and this paper focuses on the construction of a formative evaluative framework for the workshop. Key evaluation concepts and measures relating to DT have been identified in the literature and integrated into the framework, focusing on five categories: Training needs analysis; Reactions; Learning outcomes, Behavioural outcomes and Results The use of a mixed methods approach in the framework provides an additional strength, by evaluating long-term behavioural change and improvements in service delivery. As little is known about the effectiveness of DT programs for community aged care workers, the proposed framework will provide an empirical and consistent method of evaluation, to assess their impact on enhancing older people’s experience of healthcare.     

A critique of progress in adolescent smoking prevention

Researchers have developed a number of programs for preventing adolescent smoking. This critique evaluated a cross-section of 12 of these programs. The majority demonstrated short-term effectiveness in reducing the onset of smoking. The studies stressed three common themes: information about smoking's physiological and social effects, skills to cope with peer and advertising pressures, and the modification of attitudes. Despite many positive outcomes in reducing smoking, the studies did not clearly establish whether the interventions did so by imparting coping skills or by modifying attitudes. This leaves in doubt the exact processes that influence smoking behavior. Carbon monoxide and thiocyanate analyses have not been established as alternatives to self-report, as these methods prove unreliable with adolescent populations. Due to the lack of longitudinal studies, the long-term effectiveness of these interventions remains undetermined.     

Pre-, post-, and longitudinal evaluation of juvenile justice education

This article describes two stages of the Juvenile Justice Educational Enhancement Program's pre-, post-, and longitudinal evaluation research. Pilot studies were used to explore how to design statewide research of pre- and postassessment scores and community reintegration outcomes. Preliminary findings suggest that higher performing educational programs produce greater educational gains as measured by academic achievement tests, credits earned, and pupil progression rates. The findings also indicate that these programs have more students returning to school and lower recidivism rates. Building on the pilot studies, refinements were made to the research designs to enable more comprehensive statewide evaluation. Current research includes collection of pre- and postassessment scores from official sources on approximately 16,000 juvenile justice youths. In addition, a research design has been developed to examine program effectiveness by measuring community reintegration variables. Multiple data sources, including official and self-reported data on family, school, employment, and subsequent crime involvement, will be used in the longitudinal study.     

Evaluation design for a community prevention trial. An environmental approach to reduce alcohol-involved trauma

The Community Prevention Trial was 5-year effort to reduce alcohol-involved injuries and death through a comprehensive program of community awareness and policy activities. The three experimental communities were of approximately 100,000 population each (one in Northern California, one in Southern California, and one in South Carolina). Matched comparison communities were used for each experimental community. This article describes the evaluation approach used in a program that sought to change environmental factors not a specific population or target group. This approach demanded unique evaluation approaches for determining overall community aggregate effects, that is, distal outcomes, as well as changes in key mediating variables, that is, process effects. The problem of trending and lagged effects of community prevention programs are discussed.     

The Effectiveness of Faith-Based Welfare-to-Work Programs

Abstract

This essay first highlights three major challenges of measuring program effectiveness. It concludes effectiveness can best be conceptualized in terms of program outcomes and that such outcomes-while difficult to measure-can be measured with sufficient accuracy to justify the effort. These considerations are then illustrated by data gathered from 17 Los Angeles welfare-to-work programs and their clients in order to assess the comparative effectiveness of faith-based and secular programs. The essay concludes that no one type of welfare-to-work program was more effective across the board than any other type of program. Instead, the different types of programs seemed to be especially effective in certain specialized areas. The faith-based programs were especially effective in providing welfare recipients with emotional support and a sense of having a sympathetic, understanding base of support. The for-profit providers were especially effective in providing needed training in marketable job skills and help in finding employment. The essay concludes by discussing two public policy implications that flow from this type of program specialization.