Using macro variables in program evaluation

This paper discusses the importance of using macro-level variables for program analysis. It is argued that three types of macro variables--cultural, economic, and demographic--provide a context for evaluation, and offer alternative explanations for the success or failure of a program. The problems of causality and availability of data are discussed. Macro variables are seen to be especially useful in "meta evaluation," and in integrating the results of specific program evaluations into general conclusions.     

The impact of general accounting office program evaluations on government

General Accounting Office (GAO) audit reports have an impact on government operations by encouraging growth in the size and scope of the state and by encouraging the Congress and executive agencies to attend to the auditor's dominant values of economy, efficiency, and effectiveness in government. Through its financial, management, and program audits, the GAO program evaluators seek to rationalize agency management and programs so that more resources are required. The GAO's program evaluation efforts also serve Congress by providing for oversight of public policy, a bludgeon to coax agencies into pursuing activities preferred by congressmen, and a tool for electoral activities. In the end, the evaluation auditors are doomed to frustration in achieving their ideals for effective democratic government. This is so because “practical business policy” is irrelevant to government operations and inconsistent with the core values of a democratic state.     

The conduct of program evaluation reviews in the general accounting office

Professional behavior during the General Accounting Office's (GAO's) present era of oversight can best be understood by reviewing the audit types that populate village life in the GAO and the repertoire the office uses in conducting program evaluation audits. Four types of auditors are present, each of which has a role to play in the audit repertoire. This repertoire focuses on finding faults in government programs in order to catch or sustain congressional interest in GAO's work. The “audit trail” of each report may last up to 3 years and involves considerable organizational strife and conflict with outside groups. To sustain themselves through this arduous auditing process, auditors have developed a creed through which they conceptualize themselves as an elite with a mission to insure good government.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Making a case for an interface between quasi-experimental designs and social impact analyses

This paper describes the community impact assessment conducted after Concorde operations were initiated at Dulles International Airport. Problems in the design and conduct of this study that make causal interpretations implausible are discussed. The present authors report an improved design to eliminate these problems as well as increase the external validity of the results. A discussion of how the careful implementation of quasiexperimental designs allows for treatment-related causal interpretations to be made is also included. Finally, it is emphasized that practical application of these data make it possible to improve the implementation of the treatment in communities receiving it at a later date.     

Dealing with unknown variables in policy/program evaluation

A leading objection to the feasibility of applying benefit-cost analysis in evaluation is that one often cannot know the benefits, the costs, or the probabilities of their occurring for a given proposal. Threshold analysis involves handling unknown variables by converting relevant evaluation problems into questions as to whether a given benefit, cost, probability, or combination of these elements is more or less than a threshold. Above the threshold, the proposed program would be relatively profitable; and below the threshold, it would be relatively unprofitable. Applying that analysis may require the drawing of simple graphs to stimulate the minds of knowledgeable persons as to the range of the actual benefits, costs or probabilities. The approach can be applied when the benefits or the costs are either monetary or nonmonetary, and when the problems involve either go/no-go decisions or conflicting-choice decisions.     

The evaluator as program consultant

This paper presents a model for the role of the evaluator in which the evaluator acts as a Program Consultant who operates in three domains: Program Development, Planned Change, and Evaluation Technology. This permits the performance of evaluations that have greater validity and utility. The model is presented both from the theoretical perspective of linkage and from the context of the evaluation of a Case Management program operated by a Community Action Agency. Each of the domains is discussed with examples from the Case Management Evaluation. The paper concludes with a discussion of the contexts in which the model is best applied.     

Behavioral evaluation of a state program of deinstitutionalization of the developmentally disabled

The purpose of this study was to evaluate one phase of a statewide program of deinstitutionalizing developmentally disabled individuals. Forty adults who had spent a minimum of six months in community placements were evaluated with respect to changes in behavior associated with movement out of state institutions and into community settings such as group homes and day activity centers. Pre- and post-deinstitutionalization measures were taken with the Behavior Development Survey. Post-deinstitutionalization measures on Cataldo and Risley's Resident Activity Manifest were compared to the same measures obtained on 159 developmentally disabled individuals scheduled for deinstitutionalization. A consistent pattern of positive changes on both instruments favored deinstitutionalization, but changes could not be unequivocally attributed to movement out of the institutions per se.     

“Paradigms lost”: Some thoughts on choosing methods in evaluation research

Patton agrees with us concerning the need for a catholic approach to method choices in evaluation. However, Patton criticizes us for setting up a straw man when we argued that quantitative and qualitative methods are not inevitably linked to different “paradigms” and that allegiance to a particular paradigm does not force the evaluator into a choice between mutually exclusive qualitative and quantitative methods. The first part of this article shows how some of the past literature creates the impression both of an inevitable method-paradigm linkage and of a forcedchoice between qualitative and quantitative paradigms. It is argued that such impressions could have pernicious consequences for evaluation practice by artificially restricting method choices. In the final section, we consider some of the difficulties of making method choices and suggest that the purposefully developed strengths of different methods should not be completely ignored.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

Handling problems of ecological fallacy in program planning and evaluation

The issue of ecological fallacy is highlighted in the use of social area analysis as a method of program planning and evaluation. Data on psychiatric admissions from one catchment area in New York City show misleading cross level generalization to be a real and ever present danger that planners and evaluators should not ignore. A simple method of assessing the degree and major sources of difference between higher (geographic) and lower (sociological) level relationships is illustrated. Suggestions are made regarding ways of clarifying cross level generalizations.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Pitfalls in planning: Contrasting perspectives on mental health and corporate planning

Planning is a management activity that has been linked to high quality services, organizational growth, and in austere times, to organizational survival. The present study was undertaken to identify the critical barriers that inhibit or derail effective planning. It was predicted that human service and corporate managers would generally agree about the most and least important planning pitfalls in their systems. Discrepancies would be related to differences in the structural characteristics of organizations in the two sectors. Rankings made by 57 senior mental health administrators of the most and least important pitfalls in planning were contrasted with those of 159 corporate executives. Substantial agreement between the two groups of executives was found.     

Models and procedures for evaluating government provided leisure services

This paper briefly describes an experimental project in which government attempted to set up a viable management information and feedback system for evaluating accountability in services delivery. Conceptual models for agency goals and services delivery were designed and measures of program performance were developed in the provision of leisure and recreational services. Two citizen surveys are described which contributed data for service provider feedback. A discussion of perspectives on the definition of equity is presented.     

Legal standards for evaluation research: A reply to Smith

Smith (1981), asserting that not all evaluative studies require the same degree of evidence for decision-making, suggests a legal paradigm for determining how much certainty is necessary for informed decision-making. Specifically, he argues for the use of three levels of certainty, i.e., suggestive, preponderant, or conclusive evidence, modeled after the legal standards of proof. Unfortunately, such an approach is untenable for several reasons. Smith's suggestion that decision-makers adopt legal terminology, for example, is really a recommendation of form rather than substance and, furthermore, his analogy from the law to the evaluation context is based upon a misunderstanding of the trial process. These and other problems with Smith's suggestions are discussed.     

Experimental evaluation of a humanities program

This research sought to determine the effects of a one-hour humanities lecture concerning midwest architectural preservation on a group of college undergraduates. Effects were assessed by a semantic differential procedure which revealed two factors in a pre-treatment trial. A four-group design required administration of the instrument to two groups one week prior to the lecture, and to all groups one week following the lecture. ANOVA and post hoc analyses revealed significant changes in the experimental group's “semantic space” on three of eight architectural concepts which were assessed. The results provide an empirical basis for the success of the program which the humanities people had “known” all along.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.     

Problems in being responsive: Reflections on an evaluation of a program for training motorcycle riders

This paper discusses some methodological issues arising from an evaluation of the Queensland Road Safety Council Motorcycle Training Program. The basic orientation of the evaluation was responsive/illuminative; responsive in the sense that the evaluation proceeded interactively, with appropriate issues and procedures identified and modified progressively through direct observation of the program and discussion with the participants; illuminative in the sense that the final report attempted to provide a detailed account of the program operation and of its strengths and weaknesses as perceived by various stakeholders. An account is given of the main considerations in designing, implementing, and reporting the evaluation. Implications for future evaluations are discussed.