Systems engineering: A problem-solving approach to improving program performance

Evaluation methodologies have generally emphasized the measurement and assessment of program merit, as reflected by the achievement of program objectives. Less attention has been devoted to analyzing and explaining the reasons for deficient performance and, more importantly, what needs to be modified to improve performance. While evaluators do identify performance problems, the identified problems are not always analyzed. Consequently, the information reported to decision makers may be incomplete, inconclusive, and of limited utility in the decision-making process. This article describes an evaluation methodology based on problem solving techniques which can be effective and efficient in defining and analyzing problems which impair program performance. The model can be applied in evaluations where the purpose is to provide decision makers with information and recommendations to improve program performance and provide an alternative to those evaluation models which emphasize the assessment of program merit. The model also provides a potentially unique cost/benefit methodology for estimating the potential worth of improving program performance (i.e., solving the problem).     

The effects of Project SUCCESS on student academic performance: a quasi-experimental study

Project SUCCESS (PS) is a substance use prevention program that targets indicated high school students. We used archival data to explore the program's effects on students' academic achievement and disciplinary problems. It is essential to demonstrate such effects, if prevention curricula are to survive in schools that face multiple competing demands for class time. Using a quasi-experimental design that compared students exposed to the program to those not exposed in five schools, we examined the effects of PS on grades, absenteeism, and disciplinary problems. Using longitudinal analyses that linked individual students' program exposure data to their academic performance, we found that PS had a positive impact on both grades and absenteeism for two years following exposure. We note that these findings are particularly fortuitous, as PS was not designed to affect these outcomes. We recommend that future evaluations of school-based prevention curricula routinely include academic performance.     

Implementation and process evaluation of a school-based drug abuse prevention program: Project Towards No Drug Abuse

The present study provides an implementation, process, and immediate outcomes evaluation of the classroom component of Project Towards No Drug Abuse (TND). This project involves development and evaluation of a school-based drug abuse prevention curriculum for continuation high school youth, who are at relatively high risk for drug abuse. Three randomized conditions were evaluated: a standard care, classroom only, and classroom plus school-as-community. The latter condition was an enhanced school-based condition which involved outside-of-classroom meetings and activities. Implementation was high in both program conditions even though this was a higher risk context. Process evaluation data were favorable and did not vary between the two program conditions. Immediate outcomes data (knowledge) was higher in the two program conditions than in the standard care condition. Regarding the classroom program, addition of extra-classroom activities does not appear to alter the quality of delivery of the program.     

Predicted outcomes in a national sample: A source of comparison for program evaluation

The lack of comparison groups is a common problem for researchers conducting evaluations of small, specialized programs. When conventional comparison groups are not available, the authors suggest a possible alternative approach which involves: (a) locating a sub-group in a national study for whom relevant variables have been measured; (b) generating a statistical model which predicts the outcome of interest in the national data; and (c) applying the predictive model to program data, in order to estimate outcomes in the absence of progam intervention. The authors illustrate this new application by presenting the results of an analysis of data from Project Talent, a national manpower study, and discuss its application to the evaluation of a medical training program in New York City.     

A decision support tool to determine cost-to-benefit of a family-centered in-home program for at-risk adolescents

Family-centered program research has demonstrated its effectiveness in improving adolescent outcomes. However, given current fiscal constraints faced by governmental agencies, a recent report from the Institute of Medicine and National Research Council highlighted the need for cost-benefit analyses to inform decision making by policymakers. Furthermore, performance management tools such as balanced scorecards and dashboards do not generally include cost-benefit analyses. In this paper, we describe the development of an Excel-based decision support tool that can be used to evaluate a selected family-based program for at-risk children and adolescents relative to a comparison program or the status quo. This tool incorporates the use of an efficient, user-friendly interface with results provided in concise tabular and graphical formats that may be interpreted without need for substantial training in economic evaluation. To illustrate, we present an application of this tool to evaluate use of Boys Town’s In-Home Family Services (IHFS) relative to detention and out-of-home placement in New York City. Use of the decision support tool can help mitigate the need for programs to contract experts in economic evaluation, especially when there are financial or time constraints.     

How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.     

An evaluation of substance abuse aftercare program for homeless women with children using confounding variable-control design

An outcome evaluation of a substance abuse aftercare program for homeless women with children was conducted using confounding variable-control evaluation design. The confounding variables are chosen from pre-treatment and other contextual variables of the clients that are known to have significant influence on the program outcome, but those that could not have been influenced a priori by the client involvement in in-treatment program activity at Transition House (TH). The latter is the independent variable of this evaluation design. The pre-treatment variables are measured by severity of alcohol and other drug (AOD) problems of the clients, their mental health status, age, and their job status before enrollment in the program. The contextual confounding variables are composed of family and social support available to the clients before and during recovery. While applying multiple regression analysis, we were able to explain 50.8% of the total variance in program outcome by four pre-treatment variables. By adding two contextual variables of family and social support, the total variance in program outcome explained is increased to 64.1%. Finally, by adding the degree of client involvement in in-treatment program activity, we were able to augment the total variance of the program outcome to 69.7%. By estimating the changed variance of program outcome by the in-treatment program activity during the final step, controlling for all other variables previously entered, we were able to establish that client involvement in in-treatment program had unique and positive impact on the program outcome distinct from those explained by the confounding variables. The additional variance uniquely added by in-treatment program activity is 5.6% (p < .001). It has been determined that the degree of client involvement in in-treatment program had positive and systematic impact on the program outcome.     

Evaluating a management and organization development effort in mental health agencies.

Although the use of management and organization development techniques in health and human service settings is increasing, there are few systematic studies of their effects. Those which do exist are problematic in many respects. An 18-month program incorporating both management and organization development is described along with the results of its evaluation, which were positive and well-utilized. A discussion of the methods employed, the problems encountered, and the outcomes obtained illustrates that these evaluation efforts are needed and can be useful to program evaluators involved in planned-change efforts.     

The truncation-by-death problem: what to do in an experimental evaluation when the outcome is not always defined

Although experiments are viewed as the gold standard for evaluation, some of their benefits may be lost when, as is common, outcomes are not defined for some sample members. In evaluations of marriage interventions, for example, a key outcome--relationship quality--is undefined when a couple splits up. This article shows how treatment-control differences in mean outcomes can be misleading when outcomes are not defined for everyone and discusses ways to identify the seriousness of the problem. Potential solutions to the problem are described, including approaches that rely on simple treatment-control differences-in-means as well as more complex modeling approaches.     

Evaluating unintended program outcomes through Ripple Effects Mapping (REM): Application of REM using grounded theory

Several evaluation models exist for investigating unintended outcomes, including goal-free and systems evaluation. Yet methods for collecting and analyzing data on unintended outcomes remain under-utilized. Ripple Effects Mapping (REM) is a promising qualitative evaluation method with a wide range of program planning and evaluation applications. In situations where program results are likely to occur over time within complex settings, this method is useful for uncovering both intended and unintended outcomes. REM applies an Appreciative Inquiry facilitation technique to engage stakeholders in visually mapping sequences of program outcomes. Although it has been used to evaluate community development and health promotion initiatives, further methodological guidance for applying REM is still needed. The purpose of this paper is to contribute to the methodological development of evaluating unintended outcomes and extend the foundations of REM by describing steps for integrating it with grounded theory.     

Social Return On Investment (SROI): Problems,solutions … and is SROI a good investment?

The conclusion of this special issue on Social Return On Investment (SROI) begins with a summary of both advantages and problems of SROI, many of which were identified in preceding articles. We also offer potential solutions for some of these problems that can be derived from standard evaluation practices and that are becoming expected in SROIs that follow guidances from international SROI networks. A remaining concern about SROI is that we do not yet know if SROI itself adds sufficient benefit to programs to justify its cost. Two frameworks for this proposed metaevaluation of SROI are suggested, the first comparing benefits to costs summatively (the resource → outcome model). The second framework evaluates costs and benefits according to how much they contribute to or are caused by the different activities of SROI. This resource → activity → outcome model could enable outcomes of SROI to be maximized within resource constraints (such as budget and time limits) on SROI. Alternatively, information from this model could help minimize the costs of achieving a specific level of return on investment from conducting SROI. Possible problems with this metaevaluation of SROI are discussed.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

Inspiring Hope,Envisioning Alternatives: The Importance of Peer Role Models in a Mandated Treatment Program for Women

ABSTRACT

This article presents evidence from a program evaluation effort conducted with clients in a mandated treatment setting for drug-involved women charged with felony-level offenses. We present themes from qualitative interviews that stress the importance of identifying with a peer who is further along in recovery as a crucial mechanism for instilling hope and fostering program engagement. We also provide survey data that shows that a client's ability to imagine a life without using drugs, at intake, is a significant predictor of program success. Finally, the Stages of Change Readiness and Treatment Eagerness scale did not show any significant correlations with program success in this population. All women entered this program from very challenging situations, but we conclude that those who enter with the least hope and smallest (or absent) vision of any alternative future are especially vulnerable for a bad outcome. We urge researchers and practitioners to further explore the ways in which connections with peer role models facilitate engagement and successful outcomes; and to systematically evaluate the effect of interventions designed to foster such connections on program outcomes.     

Monitoring and evaluation design of Malawi’s Right Foods at the Right Time nutrition program

Child stunting is a public health problem in Malawi. In 2014, the Government of Malawi launched the Right Foods at the Right Time (RFRT) program in Ntchisi district delivering nutrition social and behavior change communication, a small-quantity lipid-based nutrient supplement to children 6–23 months, and nutrition sensitive activities. Monitoring and evaluation (M&E) systems are key aspects of successful program implementation. We describe these and the methodology for an impact evaluation that was conducted for this program. Two monitoring systems using traditional and electronic platforms were established to register and track program delivery and processes including number of eligible beneficiaries, worker performance, program participation, and to monitor input, output, and outcome indicators. The impact evaluation used comparative cross-sectional and longitudinal designs to assess impact on anthropometric and infant and young child feeding outcomes. Three cross-sectional surveys (base-, mid-, and end-line) and two longitudinal cohorts of children followed in 6-month intervals from 6 to 24 months of age, were conducted in sampled households in the program and a neighboring comparison district. Additional M&E included qualitative studies, a process evaluation, and a cost-effectiveness study. The current paper describes lessons from this program’s M&E, and demonstrates how multiple implementation research activities can inform course-correction and program scale-up.     

Towards Program Evaluation for Practitioner Learning: Human Service Practitioners' Perceptions of Evaluation

Abstract

While human service practice improvement is acknowledged as an important outcome of program evaluation, much evaluation is narrowly focused on accountability requirements. This type of evaluation often has limited use and relevance to human service practitioners. In exploring forms of evaluation that do have practice relevance, this article presents the experiences and perceptions of human service practitioners subject to evaluations that set out to foster learning. Across the cases participants expressed concerns about data quality and credibility, the relevance of evaluation data to the program context, the invasiveness of the evaluation, management of the process (particularly limited consultation with practitioners), and the lack of effective dissemination. These cases illustrate the challenges for evaluation to be useful to practice, and to foster meaningful improvements to services.     

基于层次关联矩阵的风电工程招标研究

针对风电工程招标中常见的招标方案评价问题,提出了基于层次分析法和关联矩阵的招标方案评价模型,通过该模型很好地将招标方案评价过程中的一些定性的指标,转化为定量的数据,并用于招标方案的评价,很好地克服了招标方案评价中一些定性指标的计算问题,同时将层次分析法与关联矩阵相结合,结合了这两种方法的优点,从而保证能更为准确地确定最终的招标方案。     

Jane Addams and the Campaign of Theodore Roosevelt

Abstract

Built on the assumption that both rational and nonrational thought go into problem solving, but only rational approaches have been presented as appropriate technology for program development, this article offers an alterative way of designing social programs. We call this approach “emergent planning.” Our aim is to provide a conceptualization of planning processes that will be useful regardless of culture, mission or goals of the human service organization within which planning and service occur. Our hope is to deconstruct what constitutes rational thinking in program planning in order to critically consider alternative ways of thinking and planning within the context of multiculturalism and globalization. The goal is recognition of a subjugated model of planning so that those who do not plan rationally are not automatically assumed to lack competence to engage in “real” planning and effective problem solving.     

Evaluating theory-based evaluation: information, norms, and adherence

Programmatic social interventions attempt to produce appropriate social-norm-guided behavior in an open environment. A marriage of applicable psychological theory, appropriate program evaluation theory, and outcome of evaluations of specific social interventions assures the acquisition of cumulative theory and the production of successful social interventions - the marriage permits us to advance knowledge by making use of both success and failures. We briefly review well-established principles within the field of program evaluation, well-established processes involved in changing social norms and social-norm adherence, the outcome of several program evaluations focusing on smoking prevention, pro-environmental behavior, and rape prevention and, using the principle of learning from our failures, examine why these programs often do not perform as expected. Finally, we discuss the promise of learning from our collective experiences to develop a cumulative science of program evaluation and to improve the performance of extant and future interventions.     

Welcome to the Neighborhood? Long‐Term Impacts of Moving to Low‐Poverty Neighborhoods on Poor Children's and Adolescents' Outcomes

This study examined 7‐year follow‐up data from the Yonkers Project, a study of a 1985 court‐ordered neighborhood desegregation program in Yonkers, NY. Low‐income Black and Latino families residing in impoverished neighborhoods who were randomly selected to relocate to publicly funded townhouses in middle‐class communities and demographically similar families who were not selected to move were interviewed. Self‐ and parent‐report data on 8–18‐year‐old children and youth's educational outcomes, problem behavior, and parent–child relations were examined (N=221). Youth 15–18 years of age who relocated to more advantaged neighborhoods reported lower school performance and more hyperactive behavior problems and substance use than peers who remained in impoverished neighborhoods. Program effects on parenting also emerged, with mover parents reporting less stringent monitoring and harsh disciplinary techniques than stayers. Family economic resources and neighborhood conditions, assessed approximately 5 years before, accounted for some of the program effects at 7 years.     

The relationship between implementation fidelity and educational outcomes in a school-based family support program: development of a model for evaluating multidimensional full-service programs

There is increasing recognition of the need for evaluations that identify program processes or mediators and assess degree of program implementation rather than focusing solely on outcome evaluation. This paper describes the application of complementary qualitative and quantitative evaluation procedures to assess the degree of implementation of multi-component family support programs for improving educational outcomes for at risk youth, and to assess the relationship between program implementation and outcomes. The qualitative evaluation involved prolonged engagement to identify common program domains or mediators. Using a method called Innovation Configuration Analysis, levels of implementation of program domains were explicated as well as an overall Implementation Fidelity Index. Strong positive relationships were found between overall program implementation and program-level outcomes achieved by student participants.