Comparative influence of political, cost/benefit, and statistical information on administrative decision making.

The influence of three types of evaluation information was studied in simulated decisionmaking situations within a community mental health center setting. Twelve administrators reviewed one of two decision problems (direct services or indirect services funding) and were then presented political, cost/benefit and statistical information. After each presentation, the administrators rated the importance of each type of information for decision making. Only the cost/benefit data were rated as significantly influencing their decisions. The influence of each type was not dependent upon the decision problem, nor to length of time in administrative positions. The results were interpreted in terms of “who says what, how, and to whom. ” If evaluators desire a greater impact on the decision making behavior of community health administrators, they should consider employing a pay off-based evaluation strategy and providing cost/benefit information.     

Local response to planning mandates. The prevalence and utilization of needs assessment by human service agencies.

Federal and state needs assessment mandates have traditionally been criticized both by their originators and by the local human service agencies that are required to conduct these studies as part of the planning process. This paper presents some preliminary evidence that local opposition to assessment may be shrinking as agencies gain a new appreciation for the varied uses of assessment information. Using data from a sample of Wisconsin agencies the prevalence of assessments between 1979 and 1980 is described and the conditions surrounding the practice of needs assessment are outlined. Particular attention is given to factors that affect the utilization of needs assessment results, including the characteristics of the sponsoring agency, the attributes of the study, and the structure of the assessment process.     

Referral to treatment: improving client-treatment match through process analysis.

This paper presents an expedient procedure for assessing referral counselor knowledge of the effectiveness of various treatment programs for different client types. Among the advantages of this technique are timely results, ease of application, and usefulness for process and formative evaluation. An empirical example using data from a drug abuse referral program is included to demonstrate one application of the technique. The discussion includes additional examples where the technique would have potential utility.     

The evaluator as program consultant

This paper presents a model for the role of the evaluator in which the evaluator acts as a Program Consultant who operates in three domains: Program Development, Planned Change, and Evaluation Technology. This permits the performance of evaluations that have greater validity and utility. The model is presented both from the theoretical perspective of linkage and from the context of the evaluation of a Case Management program operated by a Community Action Agency. Each of the domains is discussed with examples from the Case Management Evaluation. The paper concludes with a discussion of the contexts in which the model is best applied.     

The development of evaluation as a concession

The organization of evaluation activities takes place within the already existing programs and power arrangements of complex organizations. In this context, evaluation is likely to develop along the lines of a licensed concession operating within a particular program area and as an expression of a concession made by complex organizations to the ideal of rational decision-making. In addition, the formation of professional career paths in evaluation may lead to career immobility. The “evaluation as a profession” issue itself could be resolved not by evaluation becoming a profession but by evaluation becoming a specialty of the professional manager.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Private thoughts and public actions:: A commentary on articles dealing with ethical issues in evaluation

This commentary represents a personal reaction to seven articles on ethical issues in evaluation. The author of the commentary begins with two assumptions: (1) Professional evaluators might not have the vocabulary and constructs to analyze their thoughts and actions in ethical terms, and (2) professional evaluators have often confronted troubling situations that probably contain ethical issues embedded within them. Consequently, she queries the seven articles individually and collectively in terms of the following questions: 1. What is the range of ethical problems I might encounter as an evaluator; and how do I recognize the ethical characteristics of professional dilemmas? 2. How do I describe troubling evaluation problems I have encountered that seem to have an ethical component? 3. What are some approaches I have used or might use in thinking about a proper course of action? 4. How do I know if, in a given situation, I have behaved ethically? 5. Where do we go from here?     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

The role of the father in the family: Toward a gender-neutral family policy

A review of the literature suggests that: (a) Women are not the exclusive potential contributors to quality child care; (b) men can provide care equally well; (c) a growing number of men want to increase their involvement in child care. A number of obstacles impede, however, increased paternal involvement in child care; these include the existing societal value system, the present structure of the labor market, and the nature of many existing social services. Therefore, various aspects of social and family policy are examined and a variety of options are proposed.     

Client progress toward referral for discharge to the community: an illustration of an evaluation technique.

Data on referral and discharge from a short-term habilitative facility are used to illustrate the "survival analysis" of the rate at which institutionalized mentally retarded persons are released to the community and regression analysis of factors which help predict time to release. These techniques can be used to assess the degree to which habilitative facilities and community agencies achieve their stated goals, to help community agencies plan placement requirements, to compare the progress of various subgroups of clients, and to assess the overall benefit of deinstitutionalization efforts for the entire population at which it is aimed.     

Strengths, uses and problems of global scales as an evaluation instrument

A global scale can provided means for integrating a variety of client/patient assessment techniques to complement each other in a useful manner. Global scales have been applied to service planning and evaluation as well as being useful for the on-going communication of clients' clinical/functioning status. Global scales are attractive to many service programs because of their apparent ease of implementation and their apparent face validity. Furthermore, there is an extensive body of literature describing global scales as hightly reliable and valid when properly implemented and maintained. The literature also suggests that there are some serious pitfalls. While initial implementation of a global scale is easy, the maintenance of a reliable, valid and useful scale appears to require their active use in treatment planning, treatment review and clinical supervision processes. It also requires that more extensive multidimensional ratings be made at intake, review and termination. Furthermore, staff training and development sessions two or three times a year are needed to surface and deal with differences in clinician ratings. Given these sorts of supports, then, a global scale has been demonstrated as a useful tool in service program management as well as clinical process and outcome studies, including studies of cost effectiveness. Since the major use of a global scale is as an integrating construct, a quantitative model is offered to describe the relationship between global scale ratings and multidimensional facets influencing the global ratings.     

Evaluating Reagan's block grants. Informed opinions on essential issues and needed information resources

In this paper a set of common issues has been proposed to provide a framework for evaluating the effects of current and future block grant programs. By using these issues to identify current information needs and resources, the author attempts to promote the wider use of the available evaluation information on the current block grants and budget reductions, and to identify information gaps in these data bases.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

Behavioral evaluation of a state program of deinstitutionalization of the developmentally disabled

The purpose of this study was to evaluate one phase of a statewide program of deinstitutionalizing developmentally disabled individuals. Forty adults who had spent a minimum of six months in community placements were evaluated with respect to changes in behavior associated with movement out of state institutions and into community settings such as group homes and day activity centers. Pre- and post-deinstitutionalization measures were taken with the Behavior Development Survey. Post-deinstitutionalization measures on Cataldo and Risley's Resident Activity Manifest were compared to the same measures obtained on 159 developmentally disabled individuals scheduled for deinstitutionalization. A consistent pattern of positive changes on both instruments favored deinstitutionalization, but changes could not be unequivocally attributed to movement out of the institutions per se.     

Problem characteristics, decision processes and evaluation activity: a preliminary study of mental health center directors

This article reports on the results of a preliminary study on the characteristics of problems and problem solving in mental health centers. The major findings were that directors were generally quite satisfied both with the available level of information intended to help them make decisions, and with their ability to make good, even optimal, decisions. Much of this confidence resulted from the use of assumptions about the problems which reduced the need for empirical information. The problem solving process was highly dynamic and interactive, with changes over time in the characteristics of problems, constraints and alternative solutions. The directors played an active role in attempting to foster and influence the direction of many of these instabilities. Some implications for evaluation and research are discussed.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.