Evaluating correctional health care. The lessons of New York City.

New York City's correctional health care system is a complicated one, with many component parts. Correspondingly, the task of developing an evaluation process to assess this system turned out to be complex as well. The major responsibility for developing such a process and for keeping it in motion lies with the Contract Evaluation Unit of the Department of Health. Our experiences in meeting these challenges, as well as the historical and institutional factors that have influenced our efforts, should be of interest to professional groups facing similar tasks in other localities and to readers curious about how particular health service evaluation approaches take form. This article describes our method, how it emerged, and that it entails, with an eye toward identifying what might usefully be applied to the study of other correctional health care services and settings.     

Handling problems of ecological fallacy in program planning and evaluation

The issue of ecological fallacy is highlighted in the use of social area analysis as a method of program planning and evaluation. Data on psychiatric admissions from one catchment area in New York City show misleading cross level generalization to be a real and ever present danger that planners and evaluators should not ignore. A simple method of assessing the degree and major sources of difference between higher (geographic) and lower (sociological) level relationships is illustrated. Suggestions are made regarding ways of clarifying cross level generalizations.     

To be or not to be: Control and balancing of type I and type II errors

The various costs of Type I and Type II errors of inference from data are discussed. Unfortunately, other things equal, Types I and II errors are inversely related. Six methods of minimizing Type I error in studies are presented, each of which may be employed even after data are collected. Type II errors may also be minimized by a combination of study design and analytic means. Although some of these techniques must be built into the study others can be undertaken when beginning work on a data set. Only one of these techniques will increase Type I error, and those involving data reduction may decrease both types of error.     

Leverage and evaluation effectiveness

Weaknesses in evaluations often can be traced to structural limitations in the positions of evaluation researchers. Conventional human relations techniques often are an insufficient basis for securing strong support for evaluation research. Strategies for increasing evaluation research leverage are reviewed. Alignment of evaluation research with regulatory bodies with authority to suspend public program expenditures is advocated. Several likely obstacles in the development of the regulatory evaluation model are anticipated and addressed.     

Assessing the efficacy of ombudsman services for the aged in long term care institutions

Long term care ombudsman programs are a recently established component within the gerontological services continuum. Public policy in this area is not yet well defined and there is a conspicuous absence of systematic studies assessing the effectiveness of such program efforts. This research reports on the accomplishments of an urban based, volunteer nursing home ombudsman program. Survey findings show moderate levels of goal attainment in the policy and social relations domains and more promising results in the organizational/managerial sphere. The relationship between perceptions of program success and varying ombudsman role behaviors is underscored.     

Predicted outcomes in a national sample: A source of comparison for program evaluation

The lack of comparison groups is a common problem for researchers conducting evaluations of small, specialized programs. When conventional comparison groups are not available, the authors suggest a possible alternative approach which involves: (a) locating a sub-group in a national study for whom relevant variables have been measured; (b) generating a statistical model which predicts the outcome of interest in the national data; and (c) applying the predictive model to program data, in order to estimate outcomes in the absence of progam intervention. The authors illustrate this new application by presenting the results of an analysis of data from Project Talent, a national manpower study, and discuss its application to the evaluation of a medical training program in New York City.     

Social work ethics and children: Protection versus empowerment

Evidence indicates that a growing proportion of children and adolescents experience traditional child protective services as intrusive, oppressive, and even exploitive. Professionals tend to deny the reality that youngsters experience, project their own perceptions and vulnerabilities onto youngsters, and refuse to recognize the ability and right of youngsters to control their own lives. Empowerment of minors is an ethical obligation of the social work profession and the only true protection from abuse and neglect.     

Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Pacific Northwest utilities have sponsored the nation's earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility-sponsored program evaluations. This article critically reviews the methods and findings of four major Northwest utility program evaluations. Recommendations for future evaluation and program management efforts also are discussed.     

Behavioral epidemiology and social area analysis: The study of blackout looting

This article describes the use of social area analytic techniques in the study of the looting which occurred during the Con Edison power failure of July 13 and 14, 1977. Detailed procedures are outlined of the systematic collection and analysis of data which were developed from available data. Analyses contrasting competing explanatory theoretical models for blackout looting are presented. The central dependent variable is ratio of looted to retail stores. Both the numerator and denominator are aggregated to the census tract level.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Making a case for an interface between quasi-experimental designs and social impact analyses

This paper describes the community impact assessment conducted after Concorde operations were initiated at Dulles International Airport. Problems in the design and conduct of this study that make causal interpretations implausible are discussed. The present authors report an improved design to eliminate these problems as well as increase the external validity of the results. A discussion of how the careful implementation of quasiexperimental designs allows for treatment-related causal interpretations to be made is also included. Finally, it is emphasized that practical application of these data make it possible to improve the implementation of the treatment in communities receiving it at a later date.     

Mental health needs assessment of children and adolescents

Needs assessment is essential input for program planning. Little data are, however, available about the various approaches and whether strategies for needs assessment are viable. It was found by an inexpensive methodology that opinions about the importance of various services, opinions about the potential acceptance of services, services provided, and the total need for services are interrelated. A hybrid approach that utilized data from teachers and the people that provide mental health services was also a practical means to obtain information about the unmet needs of children and adolescents in the country. Data supported the value of this technique for the assessment of the unmet needs of children and adolescents. Thus, an inexpensive methodology can provide useful information from multiple perspectives.     

Adapting the jury trial for program evaluation: A report of an experience

A portion of the graduate program in clinical community psychology at SUNY Buffalo was subjected to a jury trial as a form of program evaluation. The theory of the trial as evaluation, the problems of implementation, and a posttrial evaluation are discussed. The trial, while time-consuming, especially in its pretrial phases, has the potential for presenting a dramatic picture of a program through the medium of human testimony. The jury was able to arrive at clear decisions on questions put to it, with a high degree of confidence. Decision makers accepted some of the jury's conclusions, and subjective evidence suggests that many of the controversial issues which generated the trial were resolved for the group by the procedure. The posttrial evaluation revealed limitations, such as evidence which was not presented at the trial. The experience proved useful for purposes of interdisciplinary education, providing another lens through which the evaluation problem could be viewed.     

Evaluating a human service program: Employing the judicial evaluation model

The emergence of adversarial models as an approach to formative and summative evaluations is gaining recognition among educational research professionals. The implementation of the Judicial Evaluation Model (JEM), as described in this article, is the first application to a human service employment and training program. Evaluation questions raised within the study were designed to assess the efficacy of linkage arrangements between the Comprehensive Employment and Training Act (CETA) prime sponsors and education service providers in the Commonwealth of Virginia. The four stages of the JEM and its application to CETA are discussed, the panel findings are reported along with noted pitfalls and strengths, suggested guidelines for implementation, and a few recommendations.     

Evaluating and improving prevention programs: a strategy from drug abuse.

This paper describes a methodology used to overcome the difficulties in evaluating and improving prevention programs. The methodology was applied in this case to a drug abuse prevention program. The methodology includes using health-oriented rather than pathology-oriented outcome measures, random assignment to groups, multiple settings, multiple outcome measures, goal-free evaluation, and the recording of process variables. The first year results showed program effectiveness in one setting and not in the other. Process analyses pointed toward improvements that could be made. Those improvements were implemented and led to greater program effectiveness in the second year.     

Evaluating Reagan's block grants. Informed opinions on essential issues and needed information resources

In this paper a set of common issues has been proposed to provide a framework for evaluating the effects of current and future block grant programs. By using these issues to identify current information needs and resources, the author attempts to promote the wider use of the available evaluation information on the current block grants and budget reductions, and to identify information gaps in these data bases.     

Case management: lessons from earlier efforts

Case management is a major, new initiative in the provision of community support services to implement public policy of maintaining mental patients in the community. A review of the case management function as used in the anti-poverty program in the 1960s, shows that a case manager works in a task environment consisting of a multitude of independent service agencies, works within the culture of a core service agency, and relates to chronic clients. The case manager's position is shaped by each of these relationships. Each provides dimensions for a formative evaluation of the program. Case management evaluation can identify issues, and document problems to contribute to the rational redesign of our service system.     

Putting schools as socio/economic institutions back into educational evaluation: A response to Kean and McNamara

A number of individuals have proposed the gradual refinement of existing programs, rather than their continual closing down, as a safeguard against educational fads and political pressures. The gradual refinement thesis is extended, refined, and criticized in this article. A different unit of analysis is proposed, one which takes into account the role of educational institutions in assisting in the reproduction of both cultural and economic inequality. It is argued that decisions about the gradual refinement of programs and institutions are not only technical but ideological and politico/economic in nature. Criteria for deciding on gradual refinement, based on a strategy of nonreformist reforms, are suggested.