The foster care system simulation: Evaluation of a training innovation

Although foster care is often spoken of as a system with many interactive facets, foster care personnel often carry out their specific roles with little understanding or consideration of the ultimate impact of that system. A simulation of public foster care service delivery designed to sensitize participants to these interactions is described. Evaluation of the simulation's accuracy in portrayal of actual service delivery and a study of the kinds of learning demonstrated by participants at three points (pretest, posttest, and follow-up) are reported. Participants found the simulation an attractive training method, indicated a greater appreciation of system-wide foster care problems, and reported changes in foster care thinking and work, but other indicators of learning were inconsistent.     

Applicability of multiple matrix sampling to estimating effectiveness of educational programs

When evaluating the effectiveness of an educational program, a sampling strategy known as “multiple matrix sampling” is particularly effective and efficient when the goal of the evaluation is estimating group (as opposed to individual) performance. Although this technique involves sampling both program participants and items and, subsequently, different participants being assessed over different sets of items, the results may be used to estimate those which would have been obtained had all participants been administered all items. The technique is here described in some detail, with advantages and disadvantages discussed, and examples of its application are given.     

A Study of An Alternative School for Pregnant and/or Parenting Teens: Quantitative and Qualitative Evidence

In response to multiple problems faced by pregnant and parenting teens, many alternative school-based comprehensive programs have been established throughout the United States. By collecting data from a sample of enrollees of such a school-based comprehensive program and comparing them with data collected from a sample of non-enrollees, the present study systematically evaluates the program. The results of the study seem to suggest that the program participants were more likely to have higher educational aspiration, better reproductive health outcomes, higher contraceptive use, and more breast-feeding practice and intention than those of their non-participating counterparts. The implications of the study findings for comprehensive school-based program for pregnant and/or parenting teens are discussed.Ruhul Amin is a Senior Fellow at the Center for Health Disparities Solutions and Senior Research Scientist in the Institute for Urban Research, Morgan State University, Baltimore, MD 21239. Dorothy C. Browne is the Director of the Institute for Urban Research, Jamir Ahmed is a research assistant at the Center for Health Disparities Solutions and Takanori Sato is research assistants at the Institute for Urban Research, Morgan State University, Baltimore, MD 21239.     

Behavioral evaluation of a state program of deinstitutionalization of the developmentally disabled

The purpose of this study was to evaluate one phase of a statewide program of deinstitutionalizing developmentally disabled individuals. Forty adults who had spent a minimum of six months in community placements were evaluated with respect to changes in behavior associated with movement out of state institutions and into community settings such as group homes and day activity centers. Pre- and post-deinstitutionalization measures were taken with the Behavior Development Survey. Post-deinstitutionalization measures on Cataldo and Risley's Resident Activity Manifest were compared to the same measures obtained on 159 developmentally disabled individuals scheduled for deinstitutionalization. A consistent pattern of positive changes on both instruments favored deinstitutionalization, but changes could not be unequivocally attributed to movement out of the institutions per se.     

A cross-cultural test of the situational bias hypothesis : The israeli scene

The major aim of this study was to test the commonly held assumption, within the Israeli educational context, that the relatively poor mean performance of disadvantaged students on conventional ability tests is due, in part, to extraneous situational factors, systematically disadvantageous to their test performance. To that end, two controlled field experiments were conducted on independent samples of N = 288 and N = 48 elementary school pupils, respectively. The first experiment centered on the interaction between sociocultural group characteristics and each of two situational factors, namely, test atmosphere and examiner status, in affecting ability test performance. The second experiment explored the effects of test atmosphere per se on the nonverbal ability test scores of disadvantaged pupils. On the whole, this research provides evidence inconsistent with the situational bias hypothesis and does not support the assumption that disadvantaged children's ability test scores can be substantially improved by manipulating test atmosphere conditions. In view of the evidence showing that situational variables do not significantly bias testing results in favor of one group over another, it follows that current educational evaluation policies in Israel, relying heavily on the standardized ability tests results of disadvantaged groups, need not be changed.     

Problems in being responsive: Reflections on an evaluation of a program for training motorcycle riders

This paper discusses some methodological issues arising from an evaluation of the Queensland Road Safety Council Motorcycle Training Program. The basic orientation of the evaluation was responsive/illuminative; responsive in the sense that the evaluation proceeded interactively, with appropriate issues and procedures identified and modified progressively through direct observation of the program and discussion with the participants; illuminative in the sense that the final report attempted to provide a detailed account of the program operation and of its strengths and weaknesses as perceived by various stakeholders. An account is given of the main considerations in designing, implementing, and reporting the evaluation. Implications for future evaluations are discussed.     

The goal-oriented approach to evaluation: Critique and case study from drug abuse treatment

The specification and measurement of program goals remains central to most evaluation research strategies, yet procedures for implementing this approach are not well-articulated. It is the purpose of this discussion to describe a stepwise procedure for programmatic goal setting and monitoring used in a demonstration drug treatment program for women. Three implementation steps are described: (a) goal setting, (b) checking for consistency, (c) monitoring and feedback. The advantages and limitations of this approach are discussed and useful complementary measurement strategies are suggested.     

Artifact in client satisfaction assessment: experience in community mental health settings

Artifact in client satisfaction assessment is discussed and the results of a study of three factors thought to mediate client satisfaction ratings; (a) general life satisfaction, (b) mode of administration, and (c) psychological symptomatology, are reported. A standard client satisfaction questionnaire (CSQ) was modified to yield parallel forms and was administered orally and in writing to 92 clients in two mental health day treatment programs. Satisfaction ratings obtained from these clients were quite similar to out-patient ratings obtained in previous studies conducted in this setting and using the same measures. Oral administration of the CSQ produced 10% higher satisfaction ratings than written administration (p less than .05) and less missing data (p less than .01). Satisfaction ratings were also obtained using a simple graphic instrument. Graphic ratings were comparable to CSQ ratings. Satisfaction with life in general and level of psychiatric symptoms together accounted for 25% of CSQ variance. The implication of these findings for future client satisfaction research is discussed.     

Developing and transferring behavioral technologies for children and youth

Behavioral researchers have developed a variety of technologies for improving the capacities of children and their caregivers. Despite the availability of behavioral technologies for children and youth, their widespread adoption and sustained use has been limited. Characteristics for appropriate (e.g., sustainable, effective) technologies for children and youth are suggested here, and strategies for research and development of these methods are described. This paper outlines the process of transferring behaviorial technologies including the phases of awareness, interest, assessment, trial, implementation, maintenance, and reinvention. Finally, issues implicit in the process of transferring behavioral technologies for children and youth are discussed.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

The evaluation of treatment in therapeutic communities: Problems of design and implementation

The evaluation of therapeutic communities provides an opportunity to explore the relationship between evaluation methodologies and the constraints imposed by the nature of the treatment setting. In most therapeutic communities the environment represents the primary vehicle of treatment and, as a result, virtually every aspect of the program's operations are, directly or indirectly, related to the treatment goals. This paper examines some of the problems this situation creates for the evaluation of such programs. It suggests that the need to develop an effective working relationship with the program becomes the primary concern of the evaluation process, and it analyzes the use of specific techniques involved in establishing this type of relationship. Such an approach may be useful in other settings where the unique characteristics of the program become the primary constraint in evaluation design and implementation.     

The marginal high school student: defining the problem and searching for policy

The central issue explored in this paper concerns those principles and objectives that should guide the formulation of policy aimed at the marginal high school student. Recent efforts to make these youth more employable through specialized and remedial programs are examined. After the general ineffectiveness of such programs is noted, an analysis is offered which links delinquency and adolescent development theory. The concept of adolescent social development is explored along with the effects of experiential education programs as a context for stimulating this development. Finally, to promote the use of programs which provide for adolescent social development, a policy of modified vouchers is advocated.