Making a case for an interface between quasi-experimental designs and social impact analyses

This paper describes the community impact assessment conducted after Concorde operations were initiated at Dulles International Airport. Problems in the design and conduct of this study that make causal interpretations implausible are discussed. The present authors report an improved design to eliminate these problems as well as increase the external validity of the results. A discussion of how the careful implementation of quasiexperimental designs allows for treatment-related causal interpretations to be made is also included. Finally, it is emphasized that practical application of these data make it possible to improve the implementation of the treatment in communities receiving it at a later date.     

Differential validity: Another threat to compensatory education evaluations

The effects of the changing nature of intelligence test items during the follow-up period for compensatory education program evaluations on the judgements about the efficacy of such programs are examined. Data generated from Stanford-Binet test items, assuming known “true” underlying motor and verbal scores, as well as data from three compensatory education evaluations, are examined. The changing nature of the IQ test items, i.e., the differential validity of the tests, is put forth as an explanation for the often disheartening results of such program evaluations. Methods for dealing with the problem of differential validity are suggested.     

Dealing with unknown variables in policy/program evaluation

A leading objection to the feasibility of applying benefit-cost analysis in evaluation is that one often cannot know the benefits, the costs, or the probabilities of their occurring for a given proposal. Threshold analysis involves handling unknown variables by converting relevant evaluation problems into questions as to whether a given benefit, cost, probability, or combination of these elements is more or less than a threshold. Above the threshold, the proposed program would be relatively profitable; and below the threshold, it would be relatively unprofitable. Applying that analysis may require the drawing of simple graphs to stimulate the minds of knowledgeable persons as to the range of the actual benefits, costs or probabilities. The approach can be applied when the benefits or the costs are either monetary or nonmonetary, and when the problems involve either go/no-go decisions or conflicting-choice decisions.     

Experimental evaluation of a humanities program

This research sought to determine the effects of a one-hour humanities lecture concerning midwest architectural preservation on a group of college undergraduates. Effects were assessed by a semantic differential procedure which revealed two factors in a pre-treatment trial. A four-group design required administration of the instrument to two groups one week prior to the lecture, and to all groups one week following the lecture. ANOVA and post hoc analyses revealed significant changes in the experimental group's “semantic space” on three of eight architectural concepts which were assessed. The results provide an empirical basis for the success of the program which the humanities people had “known” all along.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Using macro variables in program evaluation

This paper discusses the importance of using macro-level variables for program analysis. It is argued that three types of macro variables--cultural, economic, and demographic--provide a context for evaluation, and offer alternative explanations for the success or failure of a program. The problems of causality and availability of data are discussed. Macro variables are seen to be especially useful in "meta evaluation," and in integrating the results of specific program evaluations into general conclusions.     

The evaluator as program consultant

This paper presents a model for the role of the evaluator in which the evaluator acts as a Program Consultant who operates in three domains: Program Development, Planned Change, and Evaluation Technology. This permits the performance of evaluations that have greater validity and utility. The model is presented both from the theoretical perspective of linkage and from the context of the evaluation of a Case Management program operated by a Community Action Agency. Each of the domains is discussed with examples from the Case Management Evaluation. The paper concludes with a discussion of the contexts in which the model is best applied.     

The conduct of program evaluation reviews in the general accounting office

Professional behavior during the General Accounting Office's (GAO's) present era of oversight can best be understood by reviewing the audit types that populate village life in the GAO and the repertoire the office uses in conducting program evaluation audits. Four types of auditors are present, each of which has a role to play in the audit repertoire. This repertoire focuses on finding faults in government programs in order to catch or sustain congressional interest in GAO's work. The “audit trail” of each report may last up to 3 years and involves considerable organizational strife and conflict with outside groups. To sustain themselves through this arduous auditing process, auditors have developed a creed through which they conceptualize themselves as an elite with a mission to insure good government.     

Problems in being responsive: Reflections on an evaluation of a program for training motorcycle riders

This paper discusses some methodological issues arising from an evaluation of the Queensland Road Safety Council Motorcycle Training Program. The basic orientation of the evaluation was responsive/illuminative; responsive in the sense that the evaluation proceeded interactively, with appropriate issues and procedures identified and modified progressively through direct observation of the program and discussion with the participants; illuminative in the sense that the final report attempted to provide a detailed account of the program operation and of its strengths and weaknesses as perceived by various stakeholders. An account is given of the main considerations in designing, implementing, and reporting the evaluation. Implications for future evaluations are discussed.     

Quality of life indicators and program evaluation

Social indicators and program evaluation research have much in common. Quality of life work specifically raises the question of objective versus subjective variables, aggregate data versus distributional and individual issues, and the reference standards used in judgment and evaluation decisions. This article presents perspectives on these issues from the orientation of quality of life research and proposes that these approaches have relevance for program evaluation work.     

Behavioral evaluation of a state program of deinstitutionalization of the developmentally disabled

The purpose of this study was to evaluate one phase of a statewide program of deinstitutionalizing developmentally disabled individuals. Forty adults who had spent a minimum of six months in community placements were evaluated with respect to changes in behavior associated with movement out of state institutions and into community settings such as group homes and day activity centers. Pre- and post-deinstitutionalization measures were taken with the Behavior Development Survey. Post-deinstitutionalization measures on Cataldo and Risley's Resident Activity Manifest were compared to the same measures obtained on 159 developmentally disabled individuals scheduled for deinstitutionalization. A consistent pattern of positive changes on both instruments favored deinstitutionalization, but changes could not be unequivocally attributed to movement out of the institutions per se.     

Comparative influence of political, cost/benefit, and statistical information on administrative decision making.

The influence of three types of evaluation information was studied in simulated decisionmaking situations within a community mental health center setting. Twelve administrators reviewed one of two decision problems (direct services or indirect services funding) and were then presented political, cost/benefit and statistical information. After each presentation, the administrators rated the importance of each type of information for decision making. Only the cost/benefit data were rated as significantly influencing their decisions. The influence of each type was not dependent upon the decision problem, nor to length of time in administrative positions. The results were interpreted in terms of “who says what, how, and to whom. ” If evaluators desire a greater impact on the decision making behavior of community health administrators, they should consider employing a pay off-based evaluation strategy and providing cost/benefit information.     

Part/whole percentaging as a useful method in policy/program evaluation

Part/whole percentaging analysis is a simple and meaningful way of evaluating alternative policies, decisions, or other options with multiple goals measured on different dimensions. That kind of analysis involves converting the raw scores of each alternative policy on each goal into a part/whole percentage by summing the raw scores for each goal and dividing each raw score by the total. One can then sum the part/whole percentages across the goals for each policy after assigning at least tentative relative weights to each goal. Part/whole percentages are added for goals that one would like to maximize and are subtracted for goals that one would like to minimize. Where the relative weight of a goal or score is too difficult to determine, one can determine the threshold value at which there is a tie between the tentative first choice and the second choice, and then just determine whether the actual value is above or below the threshold value. The aggregated part/whole percentages for each policy can be used to choose the best policy, the best subset of policies, or to allocate a budget across the policies in proportion to their aggregated part/whole percentages.     

Developmental trends in program evaluation

It is suggested that the ontogeny of program evaluators replicates the phytogeny of the profession of program evaluation. A trend across the purposes of program evaluation from program amelioration to program advocacy and ultimately to program accountability is projected.     

A systematic approach to increasing use of management-oriented program evaluation data

With the implementation of Management Information Systems (MISs) in human service delivery settings, came the expectation that managers would make data-based decisions. There is substantial evidence, however, that this has not occurred. The present authors describe a systematic approach to correcting this deficiency. The approach involves a behavioral definition of management information use, the identification of disruptions in the behaviors comprising information use, and an intervention designed to reduce disruptions and promote use. Conclusions about the effectiveness of this approach gained from its implementation in a community mental health center are described.     

“Paradigms lost”: Some thoughts on choosing methods in evaluation research

Patton agrees with us concerning the need for a catholic approach to method choices in evaluation. However, Patton criticizes us for setting up a straw man when we argued that quantitative and qualitative methods are not inevitably linked to different “paradigms” and that allegiance to a particular paradigm does not force the evaluator into a choice between mutually exclusive qualitative and quantitative methods. The first part of this article shows how some of the past literature creates the impression both of an inevitable method-paradigm linkage and of a forcedchoice between qualitative and quantitative paradigms. It is argued that such impressions could have pernicious consequences for evaluation practice by artificially restricting method choices. In the final section, we consider some of the difficulties of making method choices and suggest that the purposefully developed strengths of different methods should not be completely ignored.