Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Differential validity: Another threat to compensatory education evaluations

The effects of the changing nature of intelligence test items during the follow-up period for compensatory education program evaluations on the judgements about the efficacy of such programs are examined. Data generated from Stanford-Binet test items, assuming known “true” underlying motor and verbal scores, as well as data from three compensatory education evaluations, are examined. The changing nature of the IQ test items, i.e., the differential validity of the tests, is put forth as an explanation for the often disheartening results of such program evaluations. Methods for dealing with the problem of differential validity are suggested.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

Mental health needs assessment of children and adolescents

Needs assessment is essential input for program planning. Little data are, however, available about the various approaches and whether strategies for needs assessment are viable. It was found by an inexpensive methodology that opinions about the importance of various services, opinions about the potential acceptance of services, services provided, and the total need for services are interrelated. A hybrid approach that utilized data from teachers and the people that provide mental health services was also a practical means to obtain information about the unmet needs of children and adolescents in the country. Data supported the value of this technique for the assessment of the unmet needs of children and adolescents. Thus, an inexpensive methodology can provide useful information from multiple perspectives.     

“Paradigms lost”: Some thoughts on choosing methods in evaluation research

Patton agrees with us concerning the need for a catholic approach to method choices in evaluation. However, Patton criticizes us for setting up a straw man when we argued that quantitative and qualitative methods are not inevitably linked to different “paradigms” and that allegiance to a particular paradigm does not force the evaluator into a choice between mutually exclusive qualitative and quantitative methods. The first part of this article shows how some of the past literature creates the impression both of an inevitable method-paradigm linkage and of a forcedchoice between qualitative and quantitative paradigms. It is argued that such impressions could have pernicious consequences for evaluation practice by artificially restricting method choices. In the final section, we consider some of the difficulties of making method choices and suggest that the purposefully developed strengths of different methods should not be completely ignored.     

Dealing with unknown variables in policy/program evaluation

A leading objection to the feasibility of applying benefit-cost analysis in evaluation is that one often cannot know the benefits, the costs, or the probabilities of their occurring for a given proposal. Threshold analysis involves handling unknown variables by converting relevant evaluation problems into questions as to whether a given benefit, cost, probability, or combination of these elements is more or less than a threshold. Above the threshold, the proposed program would be relatively profitable; and below the threshold, it would be relatively unprofitable. Applying that analysis may require the drawing of simple graphs to stimulate the minds of knowledgeable persons as to the range of the actual benefits, costs or probabilities. The approach can be applied when the benefits or the costs are either monetary or nonmonetary, and when the problems involve either go/no-go decisions or conflicting-choice decisions.     

Comparative influence of political, cost/benefit, and statistical information on administrative decision making.

The influence of three types of evaluation information was studied in simulated decisionmaking situations within a community mental health center setting. Twelve administrators reviewed one of two decision problems (direct services or indirect services funding) and were then presented political, cost/benefit and statistical information. After each presentation, the administrators rated the importance of each type of information for decision making. Only the cost/benefit data were rated as significantly influencing their decisions. The influence of each type was not dependent upon the decision problem, nor to length of time in administrative positions. The results were interpreted in terms of “who says what, how, and to whom. ” If evaluators desire a greater impact on the decision making behavior of community health administrators, they should consider employing a pay off-based evaluation strategy and providing cost/benefit information.     

Models and procedures for evaluating government provided leisure services

This paper briefly describes an experimental project in which government attempted to set up a viable management information and feedback system for evaluating accountability in services delivery. Conceptual models for agency goals and services delivery were designed and measures of program performance were developed in the provision of leisure and recreational services. Two citizen surveys are described which contributed data for service provider feedback. A discussion of perspectives on the definition of equity is presented.     

The evaluator as program consultant

This paper presents a model for the role of the evaluator in which the evaluator acts as a Program Consultant who operates in three domains: Program Development, Planned Change, and Evaluation Technology. This permits the performance of evaluations that have greater validity and utility. The model is presented both from the theoretical perspective of linkage and from the context of the evaluation of a Case Management program operated by a Community Action Agency. Each of the domains is discussed with examples from the Case Management Evaluation. The paper concludes with a discussion of the contexts in which the model is best applied.     

Applicability of standards for evaluations of educational programs, projects and materials in an international setting

This paper starts with a short discussion of the book Standards for Evaluations of Educational Programs, Projects and Materials by the Joint Committee on Standards for Educational Evaluation (McGraw-Hill, 1981). It then goes on to examine the usefulness in countries other than the United States of these standards and argues that evaluation of “programs” is more complicated and more politically biased than evaluation of “projects” and “materials.” In Europe, evaluation of educational programs (school systems) has proved to be highly correlated with the overriding objectives of different political parties, e.g., conservative, liberal, socialist, whereas educational policy in the United States is less bound to political parties, varying rather from case to case. In order to show how generally agreed upon educational goals can be interpreted according to different values, the author has reanalysed some studies of mean achievement in reading comprehension and science for 10- and 14-year-old students in 15 countries (IEA Project, 1973). The rank order of these countries is changed if, instead of national mean achievement, one takes proportion of students under a minimum level or above a prescribed top level as the criterion. Also the national (program) standard is evaluated in terms of between-school variance in results and in proportion of single-school units with exceptionally low or exceptionally high mean results. The conclusion is that the book on standards for evaluations of educational programs provides a good checklist of prerequisites for a reliable and valid evaluation. However, it is also concluded that—due to differences in values of program outcomes—such standards do not guarantee that the result of the evaluation will be indisputable.     

Problems in being responsive: Reflections on an evaluation of a program for training motorcycle riders

This paper discusses some methodological issues arising from an evaluation of the Queensland Road Safety Council Motorcycle Training Program. The basic orientation of the evaluation was responsive/illuminative; responsive in the sense that the evaluation proceeded interactively, with appropriate issues and procedures identified and modified progressively through direct observation of the program and discussion with the participants; illuminative in the sense that the final report attempted to provide a detailed account of the program operation and of its strengths and weaknesses as perceived by various stakeholders. An account is given of the main considerations in designing, implementing, and reporting the evaluation. Implications for future evaluations are discussed.     

Evaluation research in social work

Evaluation research in social work is becoming more varied and is showing signs of greater sophistication. It is taking on new forms and functions in such innovations as single subject designs and developmental research. Field experiments have begun to look more like scientific undertakings aimed at advancing knowledge and less like bad marriages between tight designs and loose programs. Agencies have made a start toward making evaluation a part of routine operations through the introduction of accountability systems and other evaluation mechanisms. In so doing they may be turning away from occasional, splashy projects to a more humdrum, but probably more efficient and useful approach to evaluation.     

Institutional child abuse

Although various forms of child abuse have been extensively investigated, other forms of psychological and institutional abuse continue to exist. This paper will focus on a variety of experiences occurring in institutions and residential centers that, in effect, cause a variety of detrimental effects and harm to those in those agencies who are supposedly “in treatment.” The various forms of institutional abuse will be examined and ramifications and repercussions explored.