Some characteristics of moral problems in evaluation practice

In order to understand better the nature of moral problems in evaluation, an actual incident from evaluation practice is examined. Based on the incident, six characteristics of moral problems are identified and briefly discussed. The types of research needed on moral aspects of evaluation practice are presented, and ways in which the evaluation profession can aid practitioners in dealing with moral problems are outlined.     

Legislative history and evaluation

Legislative histories can be done with those evaluative studies which are based on programs which have a history in the legislature. ESEA Title I, Title IV, and the Title I Technical Assistance Centers are examples of programs which have their direct source in federal legislation. The evaluator could ask whether the intent (aims and objectives) of a program is the same as the intent of the legislation or could ask whether the outcomes of a program were the same as those intended by the legislature. The purpose of this paper is to introduce the reader to the doing of legislative history in an evaluation study. The paper introduces the reader to five major sources for legislative history. The paper then discusses seven acceptable sources of information. Following this is a five-step guide to doing a legislative history.     

Classic 1960s articles in educational evaluation

The primary thrust of the interest in educational evaluation which persists today began with the passage of the Elementary and Secondary Education Act of 1965. A study of journal citations was used to identify which of the early 1960s writings have continued to influence evaluation thought in education throughout the 1970s. Five classic articles were identified and reasons for their continued influence are suggested.     

Techniques for the analysis of geographic data in evaluation

Although geographic relationships are often an important aspect of a program or system under evaluation, these relationships have frequently been ignored by evaluators. Standard techniques do exist, however, for the analysis of geographic data. This paper presents three such techniques, Geocode Analysis, Trend Surface Analysis, and Social Area Analysis, and discusses their use in evaluation. With these techniques, evaluators can more adequately deal with the questions of geographic relationships which arise in many studies.     

Post-evaluation determination of a program's generalizability

The literature on the generalizability of program effects focuses exclusively on the a priori development of evaluation designs which enable certain generalizations to be made. Due to a variety of factors, new questions of a program's generalizability often arise only after the completion of the formal evaluation study. This paper suggests secondary analysis procedures which can be employed using existing evaluation data to estimate a program's generalizability when followup field studies are not feasible.     

Management consulting and evaluation

The basic functions and procedures of management consulting are contrasted here with formal program evaluation. A case example further illustrates the differing approaches used in programmatic problem solving. The similarities and differences between these two professional specializations are highlighted in order to understand better the unique but complementary contribution each makes to the improvement of program operations.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

Differential validity: Another threat to compensatory education evaluations

The effects of the changing nature of intelligence test items during the follow-up period for compensatory education program evaluations on the judgements about the efficacy of such programs are examined. Data generated from Stanford-Binet test items, assuming known “true” underlying motor and verbal scores, as well as data from three compensatory education evaluations, are examined. The changing nature of the IQ test items, i.e., the differential validity of the tests, is put forth as an explanation for the often disheartening results of such program evaluations. Methods for dealing with the problem of differential validity are suggested.     

Ethical dilemmas in evaluating programs with family court related clients

Two ethical dilemmas are discussed; breaching client confidentiality in order to intervene in the client's illegal behavior, and defining the role of evaluator as expert witness. The evaluation context concerns a combined program development and evaluation project with clients involved in family court proceedings for child abuse or neglect. Different perspectives, approaches, and experiences in resolving each dilemma are reviewed. The conclusion argues that approaches to ethical dilemmas in evaluation projects must go beyond techniques to disguise data, to include an assessment of the values and value commitments connected to the different roles an evaluator must perform.     

Strengths, uses and problems of global scales as an evaluation instrument

A global scale can provided means for integrating a variety of client/patient assessment techniques to complement each other in a useful manner. Global scales have been applied to service planning and evaluation as well as being useful for the on-going communication of clients' clinical/functioning status. Global scales are attractive to many service programs because of their apparent ease of implementation and their apparent face validity. Furthermore, there is an extensive body of literature describing global scales as hightly reliable and valid when properly implemented and maintained. The literature also suggests that there are some serious pitfalls. While initial implementation of a global scale is easy, the maintenance of a reliable, valid and useful scale appears to require their active use in treatment planning, treatment review and clinical supervision processes. It also requires that more extensive multidimensional ratings be made at intake, review and termination. Furthermore, staff training and development sessions two or three times a year are needed to surface and deal with differences in clinician ratings. Given these sorts of supports, then, a global scale has been demonstrated as a useful tool in service program management as well as clinical process and outcome studies, including studies of cost effectiveness. Since the major use of a global scale is as an integrating construct, a quantitative model is offered to describe the relationship between global scale ratings and multidimensional facets influencing the global ratings.     

Harnessing the reliability of outcome measures

A method is presented for engineering the necessary levels of measurement reliability for evaluating ongoing programs. Two studies of levels of client functioning at a community mental health center (CMHC), in which an outcome measure, the Global Assessment Scale (GAS), did not perform as expected, drew attention to the need for better control of outcome measure reliabilities. Drawing from generalizability theory, a study was conducted of three sources of GAS score variance — clients, raters, and training in the use of the scale. Several estimates of reliability (ERs) were developed, depending on the manner in which the GAS ratings were, or would be, obtained in the CMHC. The differences among these ERs clarified why the GAS had lower reliability when used in our setting. Finally, two hypothetical examples are described to illustrate the utility of applying generalizability theory to achieve higher reliabilities for outcome measures.     

A research design for generalizing from multiple case studies

Many evaluations would be more useful to decision makers if their results were both explanatory and generalizable. In view of the limitations of traditional methods in meeting both these goals at the same time, the authors describe a research design explicitly intended for generalizing about the range of applicability of “explanatory patterns” (within a designated “target” population of cases). The paper presents an overview of the main features of a “multiple case study design,” shorthand for a multiple site, structured case study design. It then discusses the nature of explanatory patterns, how case study investigators pursue and recognize valid patterns, and how an analyst can apply the same logic to cross-site analysis in order to make valid inferences about the limiting conditions under which particular explanatory patterns apply to particular populations.     

Accountability to whom?

Social scientists are participating evermore frequently in the evaluation of social action programs. Participation, however, does not always imply social action. In keeping with the tradition of social sciences as critical and debunking activities, practitioners might consider the merits of conceiving themselves as agents of social change rather than technicians responding to emissaries of social control. This position also implies a view of applied research as an active enterprise through which reflective practitioners participate in the definition and direction of applied research efforts. After reviewing implications of non-reflective and reflective stances, the conclusion is that applied researchers adopting a social action perspective actually maximize their obligations as scholar-scientists even though a non-neutral posture is assumed.     

The efficacy of early intervention programs for children with organic impairment: A quantitative review

The results of studies examining the effectiveness of early intervention for infants and children with organic impairment and developmental delay were reviewed using recently developed quantitative methods that treat the literature review process as a unique type of scientific inquiry. Thirty-eight studies meeting certain predetermined criteria were included in the review. The 38 studies contained a total of 118 statistical hypothesis tests that evaluated the effectiveness of early intervention. An analysis of these tests based on the calculation of effect sizes revealed that subjects receiving early intervention performed better on a wide range of dependent measures than subjects not receiving intervention. The outcomes were found to be related to several design and study characteristics. Larger effect sizes were associated with preexperimental designs, and also with studies in which the internal validity was rated as poor. Several other design variables such as how subjects were assigned to conditions and how the dependent measure was recorded were related to study outcome as measured by effect size. The conclusion was made that an accurate interpretation of the early intervention research literature cannot be made without consideration of specific design variables and study characteristics.     

Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Pacific Northwest utilities have sponsored the nation's earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility-sponsored program evaluations. This article critically reviews the methods and findings of four major Northwest utility program evaluations. Recommendations for future evaluation and program management efforts also are discussed.