The foster care system simulation: Evaluation of a training innovation

Although foster care is often spoken of as a system with many interactive facets, foster care personnel often carry out their specific roles with little understanding or consideration of the ultimate impact of that system. A simulation of public foster care service delivery designed to sensitize participants to these interactions is described. Evaluation of the simulation's accuracy in portrayal of actual service delivery and a study of the kinds of learning demonstrated by participants at three points (pretest, posttest, and follow-up) are reported. Participants found the simulation an attractive training method, indicated a greater appreciation of system-wide foster care problems, and reported changes in foster care thinking and work, but other indicators of learning were inconsistent.     

The role of the father in the family: Toward a gender-neutral family policy

A review of the literature suggests that: (a) Women are not the exclusive potential contributors to quality child care; (b) men can provide care equally well; (c) a growing number of men want to increase their involvement in child care. A number of obstacles impede, however, increased paternal involvement in child care; these include the existing societal value system, the present structure of the labor market, and the nature of many existing social services. Therefore, various aspects of social and family policy are examined and a variety of options are proposed.     

Treatment participation and outcome in a program for problem drinker-drivers

Relationships were examined between 13 predictor variables and two criteria of improvement in a program for problem drinker-drivers. Four measures of social background (income, educations, age, marital status), four measures of psychopathology based on the Minnesota Multiphasic Personality Inventory (MMPI), one measure of general alcoholism, and four indicators of treatment involvement served as predictor variables. Two factorially derived criteria served as outcome indicators, one reflecting general improvement, the other reflecting improved social adjustment, but a failure to reduce drinking. This second factor was interpreted as indicating resistance. Analysis included partial correlation and multiple regression, from which explained variance was completely partitioned into sources unique to each predictor, and sources shared by combinations of predictors. Results suggested that with other factors controlled, treatment involvement was linked both to improvement and to resistance. Similar conclusions were suggested for social background, but slightly different ones for psychopathology and alcoholism. Psychopathology was uniquely related to improvement; alcoholism, to resistance.     

Social policy evaluation: An example from drinking age legislation

This article represents an exploration into issues relevant to the application of evaluation methodologies and theories to the social policy arena. A number of evaluation approaches and issues are explored in the context of an illustrative policy example — that of the legislative changes in minimum legal drinking age in the state of Michigan. This effort is directed towards the goal of furthering the solution of social problems through effective involvement of evaluation strategies in social policy development.     

Documentation in evaluation research: Managerial and scientific requirements

Documentation in evaluation research consists of written material, in human- or machine-readable form, pertaining to the plans, activities, and results of the project. It is argued here that good documentation is essential for effective management of evaluations, and for responsible reporting of the research procedures and findings. Documentation relating to electronic data processing activity is especially important. The purpose of this paper is to stimulate consideration and discussion of documentation, and to emphasize its importance in evaluation research. The role of documentation in the planning and control functions of project management is reviewed, and the importance of documentation in the assessment of research quality with respect to objectivity, validity, and replicability is discussed. Reasons for poor documentation are considered. An outline of documentation required in different phases of research projects is provided, and recommendations for improving the quality of documentation are presented.     

The operation was a success but the patient died: A critique of “the implementation and evaluation of a problem-solving training program for adolescents”

This critique examines George S. Tellado's account of his evaluation of a problem-solving program for adolescents in a public junior high school in Pennsylvania. The program was apparently successful (i.e., it was shown by the evaluation to improve students' problem-solving capacities), and the evaluation was, for the most part, technically sound, but the program did not survive. This critique suggests that by examining the problem-solving program as an organizational change intervention, several problems can be uncovered regarding the program's design, implementation, evaluation, and institutionalization. In particular, it appears that the school system's staff did not have sufficient ownership of the program to understand it or to wish to carry it on after the program's originator and evaluator were no longer present. This article addresses these problems in detail and recommends techniques from the field of organizational change consultation to increase the likelihood that such programs will live on into the future.     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

An organization-environment framework for assessing program implementation

Evaluations of health programs are typically conducted from a rational bureaucratic framework in which client change and client outcome are seen as logical outcomes of program activity. This paper proposes a theoretical framework for examining organizational consequences of program implementation. Factors in the internal environment (e.g., the hospital) and the external environment (e.g., the community) which contribute to the program's success or failure, are considered. Observational and structured interview data are used to compare two state funded perinatal projects, one an organizational "success" and the other a costly failure. Attributes of external environments which may be important to successful implementation include demand for services, provider competition, access to information, social beliefs, and professional commitment. Important features of the internal environment include host organization commitment and conflict, and structural attributes: complexity, formalization, centralization, and coordination. The paper concludes with theoretically-based questions which serve as guidelines for process evaluation.     

A cross-cultural test of the situational bias hypothesis : The israeli scene

The major aim of this study was to test the commonly held assumption, within the Israeli educational context, that the relatively poor mean performance of disadvantaged students on conventional ability tests is due, in part, to extraneous situational factors, systematically disadvantageous to their test performance. To that end, two controlled field experiments were conducted on independent samples of N = 288 and N = 48 elementary school pupils, respectively. The first experiment centered on the interaction between sociocultural group characteristics and each of two situational factors, namely, test atmosphere and examiner status, in affecting ability test performance. The second experiment explored the effects of test atmosphere per se on the nonverbal ability test scores of disadvantaged pupils. On the whole, this research provides evidence inconsistent with the situational bias hypothesis and does not support the assumption that disadvantaged children's ability test scores can be substantially improved by manipulating test atmosphere conditions. In view of the evidence showing that situational variables do not significantly bias testing results in favor of one group over another, it follows that current educational evaluation policies in Israel, relying heavily on the standardized ability tests results of disadvantaged groups, need not be changed.     

Problems in being responsive: Reflections on an evaluation of a program for training motorcycle riders

This paper discusses some methodological issues arising from an evaluation of the Queensland Road Safety Council Motorcycle Training Program. The basic orientation of the evaluation was responsive/illuminative; responsive in the sense that the evaluation proceeded interactively, with appropriate issues and procedures identified and modified progressively through direct observation of the program and discussion with the participants; illuminative in the sense that the final report attempted to provide a detailed account of the program operation and of its strengths and weaknesses as perceived by various stakeholders. An account is given of the main considerations in designing, implementing, and reporting the evaluation. Implications for future evaluations are discussed.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.