To be or not to be: Control and balancing of type I and type II errors

The various costs of Type I and Type II errors of inference from data are discussed. Unfortunately, other things equal, Types I and II errors are inversely related. Six methods of minimizing Type I error in studies are presented, each of which may be employed even after data are collected. Type II errors may also be minimized by a combination of study design and analytic means. Although some of these techniques must be built into the study others can be undertaken when beginning work on a data set. Only one of these techniques will increase Type I error, and those involving data reduction may decrease both types of error.     

Handling problems of ecological fallacy in program planning and evaluation

The issue of ecological fallacy is highlighted in the use of social area analysis as a method of program planning and evaluation. Data on psychiatric admissions from one catchment area in New York City show misleading cross level generalization to be a real and ever present danger that planners and evaluators should not ignore. A simple method of assessing the degree and major sources of difference between higher (geographic) and lower (sociological) level relationships is illustrated. Suggestions are made regarding ways of clarifying cross level generalizations.     

Evaluating the impact of utility company billing plans on residential energy consumption

Many utility companies offer their customers the choice of participation in an average payment plan, which enables them to pay a fixed sum for their utility bill each month (with final settlement at the end of the billing year), instead of the conventional “pay as you go” billing procedure. Because customers on average payment plans are protected from paying large bills during peak energy-use seasons and because the information about monthly energy use and its cost is perhaps less salient to them, it was hypothesized that customers on the average payment plan would use more electricity than customers not on the plan. Using a nonequivalent control group design, the electricity consumption of a selection of customers of two utility companies (Ns = 475 and 74) was examined. The results showed that there was no evidence to support the hypothesis. Since the logic of hypothesis testing does not permit the ready acceptance of the null hypothesis, several procedural, methodological, and statistical points were made to buttress the conclusion that the average payment plans had no effect on electricity consumption.     

Behavioral epidemiology and social area analysis: The study of blackout looting

This article describes the use of social area analytic techniques in the study of the looting which occurred during the Con Edison power failure of July 13 and 14, 1977. Detailed procedures are outlined of the systematic collection and analysis of data which were developed from available data. Analyses contrasting competing explanatory theoretical models for blackout looting are presented. The central dependent variable is ratio of looted to retail stores. Both the numerator and denominator are aggregated to the census tract level.     

Assessing the efficacy of ombudsman services for the aged in long term care institutions

Long term care ombudsman programs are a recently established component within the gerontological services continuum. Public policy in this area is not yet well defined and there is a conspicuous absence of systematic studies assessing the effectiveness of such program efforts. This research reports on the accomplishments of an urban based, volunteer nursing home ombudsman program. Survey findings show moderate levels of goal attainment in the policy and social relations domains and more promising results in the organizational/managerial sphere. The relationship between perceptions of program success and varying ombudsman role behaviors is underscored.     

Evaluating correctional health care. The lessons of New York City.

New York City's correctional health care system is a complicated one, with many component parts. Correspondingly, the task of developing an evaluation process to assess this system turned out to be complex as well. The major responsibility for developing such a process and for keeping it in motion lies with the Contract Evaluation Unit of the Department of Health. Our experiences in meeting these challenges, as well as the historical and institutional factors that have influenced our efforts, should be of interest to professional groups facing similar tasks in other localities and to readers curious about how particular health service evaluation approaches take form. This article describes our method, how it emerged, and that it entails, with an eye toward identifying what might usefully be applied to the study of other correctional health care services and settings.     

Referral to treatment: improving client-treatment match through process analysis.

This paper presents an expedient procedure for assessing referral counselor knowledge of the effectiveness of various treatment programs for different client types. Among the advantages of this technique are timely results, ease of application, and usefulness for process and formative evaluation. An empirical example using data from a drug abuse referral program is included to demonstrate one application of the technique. The discussion includes additional examples where the technique would have potential utility.     

How experiential education relates to college goals and objectives

Goals and undergraduate education objectives related to experiential learning were identified in college brochures and through interviews with selected college personnel. All college faculty were mailed questionnaires asking the extent of their agreement with the goals and objectives and their opinion as to the learning environment most conducive to achieving these objectives.Lack of faculty agreement with experiential learning goals and faculty unwillingness to endorse off-campus learning experiences conflicted with the published philosophy of the college. A Guttman Scale constructed from faculty responses to goals showed that as the degree of abstraction of each goal increased, so did the degree of faculty agreement.Based on college goals, valid and reliable instruments were developed to determine students' attitudes about field experiences, and to assess performance in field placements. As amounts of fieldwork increased, students' attitudes about self became more positive, performances at field sites showed higher achievement, and understanding of college goal-related concepts increased. Any amount of fieldwork appeared to help focus students' professional goals.     

Predicted outcomes in a national sample: A source of comparison for program evaluation

The lack of comparison groups is a common problem for researchers conducting evaluations of small, specialized programs. When conventional comparison groups are not available, the authors suggest a possible alternative approach which involves: (a) locating a sub-group in a national study for whom relevant variables have been measured; (b) generating a statistical model which predicts the outcome of interest in the national data; and (c) applying the predictive model to program data, in order to estimate outcomes in the absence of progam intervention. The authors illustrate this new application by presenting the results of an analysis of data from Project Talent, a national manpower study, and discuss its application to the evaluation of a medical training program in New York City.     

Leverage and evaluation effectiveness

Weaknesses in evaluations often can be traced to structural limitations in the positions of evaluation researchers. Conventional human relations techniques often are an insufficient basis for securing strong support for evaluation research. Strategies for increasing evaluation research leverage are reviewed. Alignment of evaluation research with regulatory bodies with authority to suspend public program expenditures is advocated. Several likely obstacles in the development of the regulatory evaluation model are anticipated and addressed.     

The conduct of program evaluation reviews in the general accounting office

Professional behavior during the General Accounting Office's (GAO's) present era of oversight can best be understood by reviewing the audit types that populate village life in the GAO and the repertoire the office uses in conducting program evaluation audits. Four types of auditors are present, each of which has a role to play in the audit repertoire. This repertoire focuses on finding faults in government programs in order to catch or sustain congressional interest in GAO's work. The “audit trail” of each report may last up to 3 years and involves considerable organizational strife and conflict with outside groups. To sustain themselves through this arduous auditing process, auditors have developed a creed through which they conceptualize themselves as an elite with a mission to insure good government.     

Adapting the jury trial for program evaluation: A report of an experience

A portion of the graduate program in clinical community psychology at SUNY Buffalo was subjected to a jury trial as a form of program evaluation. The theory of the trial as evaluation, the problems of implementation, and a posttrial evaluation are discussed. The trial, while time-consuming, especially in its pretrial phases, has the potential for presenting a dramatic picture of a program through the medium of human testimony. The jury was able to arrive at clear decisions on questions put to it, with a high degree of confidence. Decision makers accepted some of the jury's conclusions, and subjective evidence suggests that many of the controversial issues which generated the trial were resolved for the group by the procedure. The posttrial evaluation revealed limitations, such as evidence which was not presented at the trial. The experience proved useful for purposes of interdisciplinary education, providing another lens through which the evaluation problem could be viewed.     

The dysfunctional side effects of quantitative indicator production: illustration from mental health care (a message from Chicken Little)

In the current context of economic conservation, accountability and retrenchment from social programs, justification is required in order to maintain human service capacity. In this effort, evaluation is a key component. Yet desire to base action upon information resulting from evaluation must be tempered by increased attention to the effects of the evaluation process. All too often premature application of quantitative indicators formerly used for individual assessment and research to bureaucratic decision making produces side effects that are dysfunctional in nature. Frequently these side effects influence service delivery directly. More insidious, because they are less easily discerned, are distortions introduced into the data by evaluation pressure. These not only delay effects on service but also impair understanding of the very process they are meant to illuminate. In order to illustrate this phenomenon, the author reviews a general literature and utilized examples from mental health care. Common themes are identified and a tentative theory of side effect generation proposed.     

Case management: lessons from earlier efforts

Case management is a major, new initiative in the provision of community support services to implement public policy of maintaining mental patients in the community. A review of the case management function as used in the anti-poverty program in the 1960s, shows that a case manager works in a task environment consisting of a multitude of independent service agencies, works within the culture of a core service agency, and relates to chronic clients. The case manager's position is shaped by each of these relationships. Each provides dimensions for a formative evaluation of the program. Case management evaluation can identify issues, and document problems to contribute to the rational redesign of our service system.     

Putting schools as socio/economic institutions back into educational evaluation: A response to Kean and McNamara

A number of individuals have proposed the gradual refinement of existing programs, rather than their continual closing down, as a safeguard against educational fads and political pressures. The gradual refinement thesis is extended, refined, and criticized in this article. A different unit of analysis is proposed, one which takes into account the role of educational institutions in assisting in the reproduction of both cultural and economic inequality. It is argued that decisions about the gradual refinement of programs and institutions are not only technical but ideological and politico/economic in nature. Criteria for deciding on gradual refinement, based on a strategy of nonreformist reforms, are suggested.     

Disappointing and deviant youth and the rage of the elders

A generalized adult hostility toward youth derives from the complexities of the parent/child relationship, often mirrored in the helper/client and therapist/patient dyads. Parents derive considerable or little narcissistic gain from the accomplishments of their offspring, and conversely, are disappointed or even mortified by their failures. This is particularly true of narcissistically vulnerable parents—helpers and therapists as well—and accounts for widespread attitudes of competition, disparagement, and outright hostility toward young people. In the extreme, the pathological situations of emotional exploitation, incest, and physical abuse occur, but lesser forms of exploitation, such as personally gratifying advocacy, political gain, cultism, and enhancement of professional status, point to similar, but seemingly less destructive, forms of narcissistic gain. Youth may respond by conforming, defying, disappointing, and deviating. Adults may be either pleased, or enraged.     

Small area indicators and social change

This article examines the relevance of measures of social change for small areas and how these can be used in need assessment. Specifically, this article explicates the problems of selecting a limited group of indicators from among the broad panoply of indicators available, and of determining the different kinds of information provided by indicators controlling for the level of aggregation. The use of small area social indicators to do need assessment is illustrated using data from the 1970 and 1980 Health Demographic Profile System.     

Inpatient psychiatric treatment of the difficult child: Common practices and their ethical implications

This article reviews and discusses the available research literature pertaining to the prevalence of seclusion and restraint procedures in inpatient psychiatric treatment facilities for children and adolescence. The rationales for use, potential effects, and factors found to influence the application of these more extreme clinical interventions are considered. The author suggests that the supposed therapeutic function of these types of practices has not been substantiated and their implementation can vary widely from one setting to another. Results from several studies suggest that clearly defined criteria for use and more observational studies of prevalence and precipitant factors could considerably refine these procedures and reduce potential abuses. The psychopharmacologic approach to the treatment of difficult children is examined as a sophisticated counteraggression strategy which has yet to demonstrate therapeutic efficacy. These clinical procedures, which are utilized in scores of inpatient settings, generate serious ethical questions for the health professional concerned with the legal rights of those minors labeled as deviant members of families or the society at large.