An organization-environment framework for assessing program implementation

Evaluations of health programs are typically conducted from a rational bureaucratic framework in which client change and client outcome are seen as logical outcomes of program activity. This paper proposes a theoretical framework for examining organizational consequences of program implementation. Factors in the internal environment (e.g., the hospital) and the external environment (e.g., the community) which contribute to the program's success or failure, are considered. Observational and structured interview data are used to compare two state funded perinatal projects, one an organizational "success" and the other a costly failure. Attributes of external environments which may be important to successful implementation include demand for services, provider competition, access to information, social beliefs, and professional commitment. Important features of the internal environment include host organization commitment and conflict, and structural attributes: complexity, formalization, centralization, and coordination. The paper concludes with theoretically-based questions which serve as guidelines for process evaluation.     

Strengths, uses and problems of global scales as an evaluation instrument

A global scale can provided means for integrating a variety of client/patient assessment techniques to complement each other in a useful manner. Global scales have been applied to service planning and evaluation as well as being useful for the on-going communication of clients' clinical/functioning status. Global scales are attractive to many service programs because of their apparent ease of implementation and their apparent face validity. Furthermore, there is an extensive body of literature describing global scales as hightly reliable and valid when properly implemented and maintained. The literature also suggests that there are some serious pitfalls. While initial implementation of a global scale is easy, the maintenance of a reliable, valid and useful scale appears to require their active use in treatment planning, treatment review and clinical supervision processes. It also requires that more extensive multidimensional ratings be made at intake, review and termination. Furthermore, staff training and development sessions two or three times a year are needed to surface and deal with differences in clinician ratings. Given these sorts of supports, then, a global scale has been demonstrated as a useful tool in service program management as well as clinical process and outcome studies, including studies of cost effectiveness. Since the major use of a global scale is as an integrating construct, a quantitative model is offered to describe the relationship between global scale ratings and multidimensional facets influencing the global ratings.     

Skill acquisition and client movement indices. Implementing cost-effective analysis in rehabilitation programs.

This article outlines the development and implementation of a cost-effective approach to client and program evaluation. The indices presented summarize a client's acquisition of behavioral skills (Skill Acquisition Index) and progression along a continuum of becoming less dependent and more productive (Client Movement Index). Data are presented summarizing how the two non-monetary outcome measures have been used in formative program evaluation. The paper also discusses some of the problems involved in implementing cost-effective analyses in rehabilitation programs and suggests specific strategies to overcome the problems.     

Global scales : Strengths, uses and problems of global scales as an evaluation instrument

A global scale can provided means for integrating a variety of client/patient assessment techniques to complement each other in a useful manner. Global scales have been applied to service planning and evaluation as well as being useful for the on-going communication of clients' clinical/functioning status. Global scales are attractive to many service programs because of their apparent ease of implementation and their apparent face validity. Furthermore, there is an extensive body of literature describing global scales as hightly reliable and valid when properly implemented and maintained. The literature also suggests that there are some serious pitfalls. While initial implementation of a global scale is easy, the maintenance of a reliable, valid and useful scale appears to require their active use in treatment planning, treatment review and clinical supervision processes. It also requires that more extensive multidimensional ratings be made at intake, review and termination. Furthermore, staff training and development sessions two or three times a year are needed to surface and deal with differences in clinician ratings. Given these sorts of supports, then, a global scale has been demonstrated as a useful tool in service program management as well as clinical process and outcome studies, including studies of cost effectiveness. Since the major use of a global scale is as an integrating construct, a quantitative model is offered to describe the relationship between global scale ratings and multidimensional facets influencing the global ratings.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

Theory and method in treatment evaluation

We outline a conceptual model of evaluation research that has both guided and flowed from our research on the process and outcome of treatment for alcohol abuse and for depression. The conceptual model encompasses three basic processes: the process of treatment selection, the process of treatment, and the process underlying the onset and course of the disorder or problem being treated. A theory guiding a program evaluation can focus on one or more of these processes, and findings from evaluations can contribute to the development of better theories of these processes. We discuss how theories can guide both experimental and naturalistic evaluations, as well as aid in the estimation of treatment effects with either type of design. We then consider the role of theory and method in exploring patient-treatment matching. Finally, we discuss the implications of theory-guidance for the utility of evaluations—particularly the utility for evaluation researchers.     

An evaluation of substance abuse aftercare program for homeless women with children using confounding variable-control design

An outcome evaluation of a substance abuse aftercare program for homeless women with children was conducted using confounding variable-control evaluation design. The confounding variables are chosen from pre-treatment and other contextual variables of the clients that are known to have significant influence on the program outcome, but those that could not have been influenced a priori by the client involvement in in-treatment program activity at Transition House (TH). The latter is the independent variable of this evaluation design. The pre-treatment variables are measured by severity of alcohol and other drug (AOD) problems of the clients, their mental health status, age, and their job status before enrollment in the program. The contextual confounding variables are composed of family and social support available to the clients before and during recovery. While applying multiple regression analysis, we were able to explain 50.8% of the total variance in program outcome by four pre-treatment variables. By adding two contextual variables of family and social support, the total variance in program outcome explained is increased to 64.1%. Finally, by adding the degree of client involvement in in-treatment program activity, we were able to augment the total variance of the program outcome to 69.7%. By estimating the changed variance of program outcome by the in-treatment program activity during the final step, controlling for all other variables previously entered, we were able to establish that client involvement in in-treatment program had unique and positive impact on the program outcome distinct from those explained by the confounding variables. The additional variance uniquely added by in-treatment program activity is 5.6% (p < .001). It has been determined that the degree of client involvement in in-treatment program had positive and systematic impact on the program outcome.     

The relationship between implementation fidelity and educational outcomes in a school-based family support program: development of a model for evaluating multidimensional full-service programs

There is increasing recognition of the need for evaluations that identify program processes or mediators and assess degree of program implementation rather than focusing solely on outcome evaluation. This paper describes the application of complementary qualitative and quantitative evaluation procedures to assess the degree of implementation of multi-component family support programs for improving educational outcomes for at risk youth, and to assess the relationship between program implementation and outcomes. The qualitative evaluation involved prolonged engagement to identify common program domains or mediators. Using a method called Innovation Configuration Analysis, levels of implementation of program domains were explicated as well as an overall Implementation Fidelity Index. Strong positive relationships were found between overall program implementation and program-level outcomes achieved by student participants.     

Evaluation of a cross-sector community initiative partnership: Delivering a local sport program

Corporate community initiatives (CCI) are often established via cross-sector partnerships with nonprofit agencies to address critical social problems. While there is a growing body of literature exploring the effectiveness and social impact of these partnerships, there is a limited evaluative research on the implementation and execution processes of CCIs. In this paper, we examined the implementation and operational processes in the delivery of a professional sport organization's CCI initiative using program theory evaluation. The findings showed discrepancies between the associate organization and the implementers regarding understanding and fulfilling responsibilities with performing certain aspects (maintaining accurate records and program marketing) of the service delivery protocol. Despite program stakeholders being satisfied overall with the program delivery, contradictions between program stakeholders’ satisfaction in the quality of program delivery was found in critical components (marketing and communications) of the service delivery. We conclude that ongoing evaluations are necessary to pinpoint the catalyst of the discrepancies along with all partners valuing process evaluation in addition to outcome evaluation.     

Explaining Client Satisfaction with an Employee Assistance Program

ABSTRACT

The first objective of this paper is to explore the psychometric qualities as well as the usefulness of an employee assistance program client satisfaction scale. The responses obtained from 82 clients of an external service provider permitted a validation of the satisfaction scale and ultimately the selection of five items showing stronger convergent validity as well as high reliability (alpha = .89). The second objective is to explore the possible relationships between demographic (gender, age), treatment (referral-type, type of problem, urgency, degree of resolution) and organizational variables (unionization, job category, sector, size), and variations in client satisfaction levels. The results of bivariate analysis indicate that the type of referral, the type of problem, and the degree of resolution of the problem are related to client satisfaction. This study provides some support for the inclusion of treatment variables in a comprehensive explanatory model of client satisfaction. It is further argued that client satisfaction is an adequate measure of the success of an employee assistance program. Finally, some methodological issues surrounding the use of such surveys are discussed.     

A Long-Term Family-Oriented Treatment for Adolescents with Substance-Related Disorders: An Outcome Study

Treatment programs for substance-disordered adolescents increasingly use family and Twelve-Step orientations. Pathway Family Center adds to that model a long term stay (9 to 15 months), a day treatment setting, and the unique feature of temporarily placing the client with the family of another client who has progressed further in the program. This outcome study surveyed the 100 clients and their parents from the first five years of the program, using mailed questionnaires and telephone interviews. Information was collected from 51% of clients and 61% of parents, and client-parent agreement was high. Ninety-eight percent of adolescents reported that they did not return to prior drug severity, 51% reported no use since discharge, 23% reported one or two relapses, 88.2% reported improved family relations, and 90.2% reported improved quality of life. In comparing those who remained completely abstinent with those who relapsed, the non-relapsers were twice as likely to have a diagnosis of uni-polar depression at admission. We hypothesize that the depression is associated with hitting bottom, an indication that the client may be motivated to commit to major change. These outcomes compare very favorably with those of other adolescent treatment programs.     

Lapses,infidelities, and creative adaptations: Lessons from evaluation of a participatory market development approach in the Andes

Participatory approaches are frequently recommended for international development programs, but few have been evaluated. From 2007 to 2010 the Andean Change Alliance evaluated an agricultural research and development approach known as the “Participatory Market Chain Approach” (PMCA). Based on a study of four cases, this paper examines the fidelity of implementation, the factors that influenced implementation and results, and the PMCA change model. We identify three types of deviation from the intervention protocol (lapses, creative adaptations, and true infidelities) and five groups of variables that influenced PMCA implementation and results (attributes of the macro context, the market chain, the key actors, rules in use, and the capacity development strategy). There was insufficient information to test the validity of the PMCA change model, but results were greatest where the PMCA was implemented with highest fidelity. Our analysis suggests that the single most critical component of the PMCA is engagement of market agents – not just farmers – throughout the exercise. We present four lessons for planning and evaluating participatory approaches related to the use of action and change models, the importance of monitoring implementation fidelity, the limits of baseline survey data for outcome evaluation, and the importance of capacity development for implementers.     

Using Online Databases to Guide Practice and Research

Abstract

As tools for human service practitioners and researchers, online databases to locate evaluation literature are becoming increasingly useful. This article uses a clinical problem (concern about the effectiveness of electroconvuisive shock therapy for depressed persons), and a researcher's problem (planning a study to evaluate group treatment for child abusers), to illustrate logic of online searches for evaluation literature. New techniques for synthesizing many studies may require sweeping changes in how abstracts are formulated for bibliographic databases. Study synthesis techniques suggest ways to code studies to describe treatment method, client type, outcome measures, indices of study quality, and indices of treatment effect size. By replacing abstracts with such coded information, evaluation studies could be synthesized continuously for their program and policy implications.     

A participatory evaluation framework in the establishment and implementation of transdisciplinary collaborative centers for health disparities research

ObjectiveWe describe the formulation and implementation of a participatory evaluation plan for three Transdisciplinary Collaborative Centers for Health Disparities Research funded by the National Institute of Minority Health and Health Disparities.MethodsAlthough different in scope of work, all three centers share a common goal of establishing sustainable centers in health disparities science in three priority areas – social determinants of health, men’s health research, and health policy research.ResultsThe logic model guides the process, impact, and outcome evaluation. Emphasis is placed on process evaluation in order to establish a “blue print” that can guide other efforts as well as assure that activities are being implemented as planned.ConclusionWe have learned three major lessons in this process: (1) Significant engagement, participation, and commitment of all involved is critical for the evaluation process; (2) Having a “roadmap” (logic model) and “directions” (evaluation worksheets) are instrumental in getting members from different backgrounds to follow the same path; and (3) Participation of the evaluator in the leadership and core meetings facilitates continuous feedback.     

The evolving relationship between casework skills,engagement, and positive case outcomes in child protection: A structural equation model

This paper reports a study that aims to increase the understanding of the relationship between workers and their child protection clients through the development of a client engagement model. A conceptual exploratory structural equation model (SEM) of caseworker skills, engagement and client outcomes is presented. Results indicate that including parents in the planning, not ignoring problems parent identify as important, not asking them to do something they would not find helpful, and locating appropriate services, leads to higher client engagement. Although not directly related, having workers return clients phone calls acted as mediator between other worker casework skills and client engagement. Client engagement was also influenced by satisfaction with the case outcome. Worker engagement was also influenced by whether they included parents in the planning, which in turn was influenced by worker experience. Workers who were kind and considerate were more likely to be engaged. Worker engagement predicted satisfaction with the case outcome and whether workers' believed families would contact the agency in the future. Lastly, client engagement predicted worker engagement although it is our belief that this is an iterative reciprocal process.     

Staff and client perspectives on the Journey Mapping online evaluation tool in a drug court program

The purpose of this study was to assess staff and client perspectives on the Internet-based Journey Mapping program evaluation tool. A drug court program was chosen for a case study research design. Six staff and 10 clients participated in interviews and observations, and also responded to a questionnaire. A staff survey provided additional data. The study revealed evidence that (1) Journey Mapping was user-friendly, saved time, promoted positive thinking, facilitated communication, increased learning, and created client voice; (2) Journey Mapping, as a program evaluation tool, supported both administrative and treatment functions of the drug court; and (3) Journey Mapping's positive orientation supplied momentum for clients’ treatment progress. Journey Mapping's distinctive technology offered a viable option for including narrative methods along with quantitative methods in program evaluation.     

Accountability revisited: the arrival of client outcome evaluation

In the past few years, "accountability" for public mental health programs has become differentiated in the minds of not only program evaluators, but also program managers and funders, including state and local-level legislators. Increasingly, these officials are becoming concerned with more than just the numbers and targets of services delivered, and the cost involved, and are looking for evidence of positive outcome or impact on clients to justify program implementation and maintenance. This represents a significant move beyond the two accountability models that most recently seemed to be the focus of most formal accountability efforts--performance measurement and quality assurance. Pressures for implementing these two alternatives seem to have been reduced somewhat by the new federal Administration, but even prior to its advent there had been a rapid escalation in awareness of and concern for client outcome measurement among important audiences, including state and local mental health policy-makers and the U.S. Congress. This presents a major new opportunity and challenge for program evaluators at this new accountability focus continues to gather momentum.     

Gender differences in client–provider relationship as active ingredient in substance abuse treatment

The client–provider relationship is increasingly evaluated as an active ingredient in the delivery of substance abuse treatment services. This study examines gender differences in client–provider relationship as an important treatment ingredient affecting retention in treatment and reduced post-treatment substance use. The study uses data collected for the National Treatment Improvement Evaluation Study (NTIES), a prospective, cohort study of U.S. substance abuse treatment programs and their clients. Data on individual characteristics were collected at the pre-treatment interview; on client–provider relationship and services received at treatment exit; and on post-treatment drug use at 12 months post-treatment. The analytic sample consists of 3027 clients from 59 service delivery units (1922 men and 1105 women). Structural equation modeling (SEM) was used to assess the structural relations and causal connections between relationship and service variables and treatment outcome variables. Results indicate that a positive client–provider relationship is related directly to longer duration and reduced post-treatment drug use for the total sample and for men analyzed separately. For women, a positive client–provider relationship was related directly to treatment duration and only indirectly to reduced post-treatment drug use. The findings point to the significance of including client–provider relationship in service delivery models – both as a therapeutic element as well as an element facilitative of matching services to specific client needs.     

Religiousness and Adolescent Behaviors

Abstract

This study examined the relative influence of religiousness on a sub-sample of adolescents (N = 1911) from a nationally representative sample of youth. Outcome measures included delinquency, substance abuse, physical health, mental health, and education. The study controlled for family processes and peer influences, as well as a variety of sociodemographic factors.

Parent religiosity and family processes, especially parental styles, are found to influence girls more so than boys, while exposure to peer influences affects both boys and girls on all outcome measures except delinquency. Findings highlight gender related similarities and differences that should be taken into account when providing interventions to adolescents and their families.