Ethical dilemmas in evaluating programs with family court related clients

Two ethical dilemmas are discussed; breaching client confidentiality in order to intervene in the client's illegal behavior, and defining the role of evaluator as expert witness. The evaluation context concerns a combined program development and evaluation project with clients involved in family court proceedings for child abuse or neglect. Different perspectives, approaches, and experiences in resolving each dilemma are reviewed. The conclusion argues that approaches to ethical dilemmas in evaluation projects must go beyond techniques to disguise data, to include an assessment of the values and value commitments connected to the different roles an evaluator must perform.     

Staff and client perspectives on the Journey Mapping online evaluation tool in a drug court program

The purpose of this study was to assess staff and client perspectives on the Internet-based Journey Mapping program evaluation tool. A drug court program was chosen for a case study research design. Six staff and 10 clients participated in interviews and observations, and also responded to a questionnaire. A staff survey provided additional data. The study revealed evidence that (1) Journey Mapping was user-friendly, saved time, promoted positive thinking, facilitated communication, increased learning, and created client voice; (2) Journey Mapping, as a program evaluation tool, supported both administrative and treatment functions of the drug court; and (3) Journey Mapping's positive orientation supplied momentum for clients’ treatment progress. Journey Mapping's distinctive technology offered a viable option for including narrative methods along with quantitative methods in program evaluation.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.     

Case management: lessons from earlier efforts

Case management is a major, new initiative in the provision of community support services to implement public policy of maintaining mental patients in the community. A review of the case management function as used in the anti-poverty program in the 1960s, shows that a case manager works in a task environment consisting of a multitude of independent service agencies, works within the culture of a core service agency, and relates to chronic clients. The case manager's position is shaped by each of these relationships. Each provides dimensions for a formative evaluation of the program. Case management evaluation can identify issues, and document problems to contribute to the rational redesign of our service system.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Methodological strategies for evaluating youth gang prevention programs

This article reflects on common challenges and lessons learned during the evaluation of gang prevention programs based on case studies of three federally funded Canadian programs. Elements of evaluation design, implementation, data analysis and reporting of results are discussed. More specifically, the article highlights issues that occur when evaluating community projects focused on preventing extreme risks for violence and the complexity of working in potentially dangerous and/or unstable work environments. Topics covered include the problem with quasi-experimental designs, model fidelity adherence, program documentation, client recruitment and retention, and data collection. Recommendations are provided to improve evaluations of youth gang prevention programs and similar community-based interventions that focus on the prevention of youth violence.     

An organization-environment framework for assessing program implementation

Evaluations of health programs are typically conducted from a rational bureaucratic framework in which client change and client outcome are seen as logical outcomes of program activity. This paper proposes a theoretical framework for examining organizational consequences of program implementation. Factors in the internal environment (e.g., the hospital) and the external environment (e.g., the community) which contribute to the program's success or failure, are considered. Observational and structured interview data are used to compare two state funded perinatal projects, one an organizational "success" and the other a costly failure. Attributes of external environments which may be important to successful implementation include demand for services, provider competition, access to information, social beliefs, and professional commitment. Important features of the internal environment include host organization commitment and conflict, and structural attributes: complexity, formalization, centralization, and coordination. The paper concludes with theoretically-based questions which serve as guidelines for process evaluation.     

The evolution of Rare Pride: Using evaluation to drive adaptive management in a biodiversity conservation organization

Rare Pride is a social marketing program that stimulates human behavior change in order to promote biodiversity conservation in critically threatened regions in developing countries. A series of formal evaluation studies, networking strategies, and evaluative inquiries have driven a 20-year process of adaptive management that has resulted in extensive programmatic changes within Pride. This paper describes the types of evaluation that Rare used to drive adaptive management and the changes it caused in Pride's theory-of-change and programmatic structure. We argue that (a) qualitative data gathered from partners and staff through structured interviews is most effective at identifying problems with current programs and procedures, (b) networking with other organizations is the most effective strategy for learning of new management strategies, and (c) quantitative data gathered through surveys is effective at measuring program impact and quality. Adaptive management has allowed Rare to increase its Pride program from implementing about two campaigns per year in 2001 to more than 40 per year in 2009 while improving program quality and maintaining program impact.     

The operation was a success but the patient died: A critique of “the implementation and evaluation of a problem-solving training program for adolescents”

This critique examines George S. Tellado's account of his evaluation of a problem-solving program for adolescents in a public junior high school in Pennsylvania. The program was apparently successful (i.e., it was shown by the evaluation to improve students' problem-solving capacities), and the evaluation was, for the most part, technically sound, but the program did not survive. This critique suggests that by examining the problem-solving program as an organizational change intervention, several problems can be uncovered regarding the program's design, implementation, evaluation, and institutionalization. In particular, it appears that the school system's staff did not have sufficient ownership of the program to understand it or to wish to carry it on after the program's originator and evaluator were no longer present. This article addresses these problems in detail and recommends techniques from the field of organizational change consultation to increase the likelihood that such programs will live on into the future.     

The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.     

Ethical problems in program evaluation: Advice for trapped evaluators

This paper argues that ethical problems in program evaluation are increased when conflicting or incompatible evaluation models are applied concurrently. Three models are illustrated: an Amelioration Model, generating better information for a program's own decision makers; an Accountability Model, focusing on public data disclosure and citizen participation in evaluation; and an Advocacy Model, in which the evaluation is designed to advance the program's interests in the competition for resources. Ethical problems inherent in each model, and arising from combining models are presented, along with case examples and discussion. The authors suggest several directions that program evaluators can consider to reduce their vulnerability to ethical problems.     

Evaluation of a complex, multisite, multilevel grants initiative

The Safe Schools/Healthy Students (SS/HS) national evaluation seeks to assess both the implementation process and the results of the SS/HS initiative, exploring factors that have contributed to or detracted from grantee success. Each site is required to forge partnerships with representatives from education, mental health, juvenile justice, and law enforcement, coordinating and integrating their efforts and working together to contribute to comparable outcomes (e.g., reduced violence and alcohol and drug use, improved mental health services). The evaluation uses multiple data collection techniques (archival data, surveys, site visits, interviews, and focus groups) from a variety of sources (project directors, community partners, schools, and students) over several years. Certain characteristics of the SS/HS initiative represent unique challenges for the evaluation, including the absence of common metrics for baseline, outcome data, and lack of comparison group. A unifying program theory was required to address these challenges and synthesize the large amounts of qualitative and quantitative information collected. This article stresses the role of program theory in guiding the evaluation.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

A Comparative Analysis and Evaluation of a University Employee Assistance Program

Abstract

The major goal of this presentation is to provide overview of evaluation results of the University of Pittsburgh (Pitt) Faculty and Staff Assistance Program (FSAP). The major goal of the Pitt FSAP evaluation was to generate information pertaining to the program's service adequacy, program effectiveness, client satisfaction, cost-efficiency and cost-savings. Such an evaluation effort is becoming increasingly important due to the recent trend of organizational downsizing and coping with resource constraints. As Universities downsize, it can be challenging to justify allocating scarce funds to employee assistance programming. This study also includes review of EAPs among eighteen universities and colleges from around the country in order to compare program performance of the Pitt FSAP system.     

The Dutch Client Databank in Public Social Work

ABSTRACT

The history of the Dutch National Databank for Public Social Work began in 1960 in the province of Limburg. Social workers in that province met at a seminar and decided to set up a registration system concerning client characteristics and the counselling process which all the institutes of public social work in Limburg could be required to register in a uniform way. Thirty-three years later Dutch Public Social Work agencies are registering data according to a national system which was wanted, designed and developed by the united local authorities and all the institutes of Public Social Work. As a result, the National Databank for Public Social Work administers the data of all of the 166 institutes in the Netherlands and consequently is privileged to deal with the client characteristics and other specific data of the 250,000 client units that are involved every year with Public Social Work.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

A Two-Level Community Intervention Model for Homeless Mothers with Mental Health or Substance Abuse Disorders

ABSTRACT

This paper describes a two-tiered community intervention model in response to a call from the Center for Mental Health Services for the development of best-practice strategies for homeless mothers with mental health and/or substance use disorders and their children. This model incorporates best practice principles gained from previous research projects pitched at the system and client levels. The model has been conceptualized, presented, and funded by CMHS and is currently being implemented in a local community. This model is unique in that it develops an intervention that addresses both levels of service delivery (i.e., system and client) in a single project. Blending systems integration/wraparound service philosophy and consumer-driven/strength-based development approaches provides a model that encompasses interventions at the individual client level via supported case management, and improved coordination and collaboration at the service system level. While this model was developed for a specific community population, the approach is general inits applicability to other areas of practice.     

Gamesmanship for internal evaluators: Knowing when to “hold 'em” and when to “fold 'em”

This article discusses the ethical conflicts between an internal evaluator's altruistic social reform motivation and an organization's practical survival motivation. The internal evaluator might encounter (a) pressure to downplay negative and emphasize positive findings; (b) reinforcement for nonthreatening, routine evaluation activities; (c) greater interest in making the evaluation unit visible than in using evaluation results; or (d) reduced access to privileged information.