Some characteristics of moral problems in evaluation practice

In order to understand better the nature of moral problems in evaluation, an actual incident from evaluation practice is examined. Based on the incident, six characteristics of moral problems are identified and briefly discussed. The types of research needed on moral aspects of evaluation practice are presented, and ways in which the evaluation profession can aid practitioners in dealing with moral problems are outlined.     

Classic 1960s articles in educational evaluation

The primary thrust of the interest in educational evaluation which persists today began with the passage of the Elementary and Secondary Education Act of 1965. A study of journal citations was used to identify which of the early 1960s writings have continued to influence evaluation thought in education throughout the 1970s. Five classic articles were identified and reasons for their continued influence are suggested.     

Insuring equity in compensatory education student selection

Student selection strategies, because they determine who will receive special services, must be evaluated in terms of equity and validity. Yet those responsible for designing compensatory education student selection in the local school district lack methods that allow them to appraise their selection strategies in terms of equity. This paper describes a decision-theory-based procedure that, if followed, will increase the informational value of data often available from program evaluation data collection and district information systems. The information produced by the decision theory approach bears directly on the basic question of who gets served.     

Techniques for the analysis of geographic data in evaluation

Although geographic relationships are often an important aspect of a program or system under evaluation, these relationships have frequently been ignored by evaluators. Standard techniques do exist, however, for the analysis of geographic data. This paper presents three such techniques, Geocode Analysis, Trend Surface Analysis, and Social Area Analysis, and discusses their use in evaluation. With these techniques, evaluators can more adequately deal with the questions of geographic relationships which arise in many studies.     

Post-evaluation determination of a program's generalizability

The literature on the generalizability of program effects focuses exclusively on the a priori development of evaluation designs which enable certain generalizations to be made. Due to a variety of factors, new questions of a program's generalizability often arise only after the completion of the formal evaluation study. This paper suggests secondary analysis procedures which can be employed using existing evaluation data to estimate a program's generalizability when followup field studies are not feasible.     

Management consulting and evaluation

The basic functions and procedures of management consulting are contrasted here with formal program evaluation. A case example further illustrates the differing approaches used in programmatic problem solving. The similarities and differences between these two professional specializations are highlighted in order to understand better the unique but complementary contribution each makes to the improvement of program operations.     

The development of evaluation as a concession

The organization of evaluation activities takes place within the already existing programs and power arrangements of complex organizations. In this context, evaluation is likely to develop along the lines of a licensed concession operating within a particular program area and as an expression of a concession made by complex organizations to the ideal of rational decision-making. In addition, the formation of professional career paths in evaluation may lead to career immobility. The “evaluation as a profession” issue itself could be resolved not by evaluation becoming a profession but by evaluation becoming a specialty of the professional manager.     

Strengths, uses and problems of global scales as an evaluation instrument

A global scale can provided means for integrating a variety of client/patient assessment techniques to complement each other in a useful manner. Global scales have been applied to service planning and evaluation as well as being useful for the on-going communication of clients' clinical/functioning status. Global scales are attractive to many service programs because of their apparent ease of implementation and their apparent face validity. Furthermore, there is an extensive body of literature describing global scales as hightly reliable and valid when properly implemented and maintained. The literature also suggests that there are some serious pitfalls. While initial implementation of a global scale is easy, the maintenance of a reliable, valid and useful scale appears to require their active use in treatment planning, treatment review and clinical supervision processes. It also requires that more extensive multidimensional ratings be made at intake, review and termination. Furthermore, staff training and development sessions two or three times a year are needed to surface and deal with differences in clinician ratings. Given these sorts of supports, then, a global scale has been demonstrated as a useful tool in service program management as well as clinical process and outcome studies, including studies of cost effectiveness. Since the major use of a global scale is as an integrating construct, a quantitative model is offered to describe the relationship between global scale ratings and multidimensional facets influencing the global ratings.     

Experimental evaluation of a humanities program

This research sought to determine the effects of a one-hour humanities lecture concerning midwest architectural preservation on a group of college undergraduates. Effects were assessed by a semantic differential procedure which revealed two factors in a pre-treatment trial. A four-group design required administration of the instrument to two groups one week prior to the lecture, and to all groups one week following the lecture. ANOVA and post hoc analyses revealed significant changes in the experimental group's “semantic space” on three of eight architectural concepts which were assessed. The results provide an empirical basis for the success of the program which the humanities people had “known” all along.     

Metaphysics and professional resistance to program evaluation

One of the most common and most difficult problems confronting program evaluators is the resistance they encounter when attempting to implement evaluation efforts. The philosophical and psychological foundations of this resistance are explored. Problems in the evaluative paradigm are presented. Value differences between evaluators and practitioners are highlighted. Emotional bases for practitioner behavior are clarified and types and loci of resistance are outlined. Suggestions for coping strategies are discussed.     

Dealing with unknown variables in policy/program evaluation

A leading objection to the feasibility of applying benefit-cost analysis in evaluation is that one often cannot know the benefits, the costs, or the probabilities of their occurring for a given proposal. Threshold analysis involves handling unknown variables by converting relevant evaluation problems into questions as to whether a given benefit, cost, probability, or combination of these elements is more or less than a threshold. Above the threshold, the proposed program would be relatively profitable; and below the threshold, it would be relatively unprofitable. Applying that analysis may require the drawing of simple graphs to stimulate the minds of knowledgeable persons as to the range of the actual benefits, costs or probabilities. The approach can be applied when the benefits or the costs are either monetary or nonmonetary, and when the problems involve either go/no-go decisions or conflicting-choice decisions.     

Putting schools as socio/economic institutions back into educational evaluation: A response to Kean and McNamara

A number of individuals have proposed the gradual refinement of existing programs, rather than their continual closing down, as a safeguard against educational fads and political pressures. The gradual refinement thesis is extended, refined, and criticized in this article. A different unit of analysis is proposed, one which takes into account the role of educational institutions in assisting in the reproduction of both cultural and economic inequality. It is argued that decisions about the gradual refinement of programs and institutions are not only technical but ideological and politico/economic in nature. Criteria for deciding on gradual refinement, based on a strategy of nonreformist reforms, are suggested.     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.