Evaluation in the community mental health centers program: A bold new reproach?

The Federal Community Mental Health Centers Program (CMHC)-from 1963 to 1981-was heralded as a revolution in mental health care. Championed by many, and severely criticized by others, the actual impact of the program on the nation's mental health remains unclear. The authorization to evaluate the CMHC Program came originally from congressional legislation (PL 90-174), and later from the policies and regulations of NIMH under a series of Federal laws, notably PL 94-63. From 1976-1980, two dominant evaluation strategies were prevalent: funds expended by NIMH each year for studies of CMHC services or program-wide evaluations, and a much larger expenditure by CMHCs to conduct their own, independent evaluations following federal guidelines. As the Center's Program was turned over to the states in the form of block grants (PL 97-35), a group of professionals involved with setting and carrying out federal CMHC evaluation policy of both varieties met in public forum to debate the impact of these two evaluation approaches. While some participants cited gains in evaluation technology and impact upon local management of CMHCs, others found the lack of a coordinated and systematic approach to evaluating the CMHC Program to have been an opportunity missed. The impact of CMHC evaluation efforts are also discussed in terms of their major contribution to the field of evaluation research as a whole.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

An evaluator of the future

For the evaluator who is part of the management of a human service organization, future expectations will be more demanding than at present. Operating staff, managers, and funders will already be familiar with the methods and the usefulness of capable program evaluation. Improved information tools will help the evaluator meet this challenge, by keeping the organization visible and manageable. A more fundamental change will be an improved “language of accountability”. Program standards, accreditation, and review now focus on structure and process characteristics. What is emerging is a growing consensus on definitions of client problems and program objectives in relation to these problems. This consensus spurs psychometric research on measures of problems and of the attainment of program objectives. Dependable, standard measures and meaningful comparative data will make the evaluation of program effectiveness possible.     

Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Pacific Northwest utilities have sponsored the nation's earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility-sponsored program evaluations. This article critically reviews the methods and findings of four major Northwest utility program evaluations. Recommendations for future evaluation and program management efforts also are discussed.     

Systemic Pathways for Social Transformation: School Change,Multicultural Organization Development,Multicultural Education,and LGBT Youth

This essay reviews key themes in the research on school change, multicultural organization development, and multicultural education initiatives to suggest a new perspective from which to understand efforts to meet the needs of lesbian, gay, bisexual, and transgender (LGBT) youth in school settings. A model is proposed, which indicates that the range of current program efforts are best understood when seen in their relationship to each other and as parts of a continuum of school change strategies. By providing a systemic overview based on multicultural organization development principles, the strengths and applicability of a range of current change efforts may offer useful suggestions of practical, developmentally appropriate next steps for educators, parents, and community advocates working to change schools to better serve LGBT youth.     

Organizational capacity building: addressing a research and practice gap

The purpose of this article is to address the gap between evaluation research, and the practice of capacity building with nonprofits. This study describes a 5-year capacity building initiative with grassroots organizations including a longitudinal evaluation of the implementation and outcomes achieved. Formative processes yielded many lessons that were used to improve the capacity building model of services. The results show that the majority of groups met a priori expectations for participation success. Organizational staff valued technology, consultants, and program funding the most. Increases were found in board membership and perceptions of visibility of the organization were enhanced. Executive directors reported greater awareness of needs and improved management knowledge. These small organizations fill many unmet needs and more capacity building evaluation studies are needed to understand the mechanisms that support their efforts and the impact on their sustainability.     

The impact of general accounting office program evaluations on government

General Accounting Office (GAO) audit reports have an impact on government operations by encouraging growth in the size and scope of the state and by encouraging the Congress and executive agencies to attend to the auditor's dominant values of economy, efficiency, and effectiveness in government. Through its financial, management, and program audits, the GAO program evaluators seek to rationalize agency management and programs so that more resources are required. The GAO's program evaluation efforts also serve Congress by providing for oversight of public policy, a bludgeon to coax agencies into pursuing activities preferred by congressmen, and a tool for electoral activities. In the end, the evaluation auditors are doomed to frustration in achieving their ideals for effective democratic government. This is so because “practical business policy” is irrelevant to government operations and inconsistent with the core values of a democratic state.     

An industrial application of meta-evaluation

This article describes the efforts of an industrial training organization to increase the number and to improve the quality of “follow-up” evaluations (i.e., evaluations in terms of post-training job performance). Efforts included an analysis of the problems encountered in evaluating training, development and implementation of corrective actions, and an evaluation of the corrective actions. The project illustrates the process of “meta-evaluations.”     

Open systems evaluation and the logic model: Program planning and evaluation tools

This paper describes two case management projects and efforts to evaluate outcomes. Lessons learned from the first effort suggest that a tool referred to as the logic model and a particular approach to evaluation called the “open systems” model might be useful to planners and program evaluators. The logic model provides a means of presenting conditions a program is intended to address, activities that constitute a program, short term outcomes resulting from program activities and long term impacts. The open systems perspective emphasizes evaluation as a tool to be used to achieve program objectives as opposed to establishing cause and effect relationships. This paper also describes how the logic model and open systems evaluation were used to facilitate the development of an evaluation plan for a project designed to assist homeless, substance abusing, pregnant women.     

Organizing evaluation to serve the needs of program planners and managers

Evaluation is often both conceptually and functionally separate from program planning and operations management. Though program managers may not see themselves as served well by the work of evaluation staffs, there are a number of potentially beneficial side effects to be realized from close collaboration between program administrators and evaluators. These flow only when the evaluation task is broadly defined and when both administrators and evaluators recognize their interdependence. Particular functional benefits to management from evaluation are described. Characteristics of interorganizational relations which might increase the probability of their attainment are then outlined and discussed.     

A remedial strategy for data analysis

Professional evaluators are often called upon to analyze data produced by a catastrophically inadequate evaluation design. This problem is occurring more frequently as accountability pressures force program experts into evaluation activities for which they are not trained. A remedial strategy involving diagnosis of error, application of a corrective procedure and sensitization of program personnel of the need for a more sophisticated stance, is proposed as a solution. A case study is described and the contribution of a remedial strategy to improved evaluation is outlined.     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

Enhancing program review for evaluation and management

Program review has not received the attention it warrants as a program evaluation tool despite its wide use for evaluation and management purposes. The use of the program review will probably endure on the strength of its face validity and irrespective of other developments in the field of program evaluation. Evaluators should realize this and, accordingly, attempt to improve its effectiveness. This paper presents one organization's approach to achieving this objective through the explication of development principles, implementation guidelines and review items. This paper also discusses benefits that can be expected from a systematic development of this tool and presents various research directions and potentials in this area.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

Evaluating employee assistance policy in an HMO-based alcoholism project.

One aspect of successful employee assistance program (EAP) implementation is the adoption of a formal, written policy, reflecting company commitment to EAP guidelines and goals. This study of criteria predictive of such policy adoption was conducted at the occupational alcoholism project of a New England health maintenance organization (HMO). Data on nearly 400 organizations contacted by occupational program consultants (OPCs) over a 20-month period were collected by questionnaire and interview. One third of these organizations adopted employee assistance policies and set about establishing formal programs. Stepwise multiple regression is the principal method used to pinpoint the correlates of policy adoption. Two of the most important of these are the attitudes of contact persons within the organization toward alcoholism and employee assistance programs, particularly their admission of alcohol problems within their social networks; and the consultants' persistence and marketing skills. The adopting organizations also had reputations for being progressive, and actively concerned about employee welfare; they tended to be large, their executives autonomous, and their union membership rates high. Inhibiting policy acceptance were fears that a written policy would jeopardize the reputation and image of the organization, and that an employee assistance program would remove internal control of personal procedures. The adequacy of the evaluative data and methods are discussed, and recommendations are offered in the interests of streamlining the efforts of OPCs and of achieving greater penetration of targeted organizations.     

Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

A response to the BPA evaluation plan: Some modest but utilitarian considerations

In response to the complex and expensive Bonneville Power Administration (BPA) evaluation plan, this article attempts to identify some pragmatic issues and alternative approaches to conservation program evaluation. In particular, the need for simpler, lower-cost evaluation options and the importance of process evaluation to program management are emphasised. This is followed by the delineation of several key principles to be observed in conducting a more modest, but still methodologically sound and managerially useful, evaluation of energy conservation programs.