Evaluation research in social work

Evaluation research in social work is becoming more varied and is showing signs of greater sophistication. It is taking on new forms and functions in such innovations as single subject designs and developmental research. Field experiments have begun to look more like scientific undertakings aimed at advancing knowledge and less like bad marriages between tight designs and loose programs. Agencies have made a start toward making evaluation a part of routine operations through the introduction of accountability systems and other evaluation mechanisms. In so doing they may be turning away from occasional, splashy projects to a more humdrum, but probably more efficient and useful approach to evaluation.     

Evaluating sociodemographic data: implications for program planning and development

Sociodemographic data can have broad implications for program planning and development. Yet, such data are often obtained and analyzed in a non-rigorous methodological fashion. Four of these methodological issues are discussed in this paper. Their relevance for program planning and development are clarified, using examples from the utilization, needs assessment, illness behavior, and social functioning literature.     

The value of unattractive characteristics in social entitlements.

Social programs legislated as entitlements are attractive to promoters of health and welfare services because of the programmatic security and stability such a status offers. Yet existing entitlements and other publicly funded services routinely fail to exhibit the high performance standards that an entitlement status requires. Such "flaws" as confusion over purpose, obstructive procedures, inefficient administration, and cost overruns can also serve important functional and political purposes. Therefore, administrators and evaluators who are in a position to witness these discrepancies will sometimes find their roles as manager and critic compromised. Instead of pursuing policies that would satisfy entitlement principles, they may only be in a position to implement and defend the "flaws" as reasonable compromises. There is the risk, however, that the public will remain skeptical about overall programmatic results despite optimistic evaluation reports. More careful legislation, more disciplined administration, and more neutral evaluation will be required if human services are to retain the public's confidence.     

The development of evaluation as a profession: Current status and some predictions

Sociological models of professionalization are applied to recent events in program evaluation in order to understand the development of the field and to predict future events. The focus is on problems that might arise and on the necessity of ameliorating them. The possibility of eliminating such problems is discounted since they appear to be integral elements of the process of professionalization. Major elements of the analysis include discussions of the consequences of the development of a specialized body of evaluation knowledge, the definition of evaluation tasks, exclusivity in the performance of those tasks, the development of professional associations, and problems in the training of evaluators. The reasons for problems in these areas are presented and the importance of being aware of the origin of the problems is discussed.     

The impact of citizen evaluation review on community mental health center programs

A field study of techniques for involving local citizens in the evaluation of Community Mental Health Center (CMHC) services shows that lay citizens can make recommendations that service agencies will use to improve services. Seventeen citizen groups received evaluation information from participating CMHCs and developed 310 recommendations. Seventy-one percent of these suggestions were used by the CMHCs. Staff and governing board attitudes, fiscal pressures, and other organizational characteristics affected the success of the citizen groups.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Cybernetic appropriateness review: Does it change physician hospital utilization patterns?

All hospital care can be characterized as either appropriate (i.e., efficatious and necessary) or inappropriate (i.e., services without benefit or services that could be provided in a less costly setting). A demonstration project combining an appropriateness determination methodology with a cybernetic control system was implemented in a study area that included six hospitals and approximately 400 practicing physicians. Project results to date indicate substantial improvements in the ratio of appropriate to inappropriate care for both hospitals and individual physicians. Evaluation research design problems and areas of future research are also discussed.     

The behavior of the battered child: A behavioral deficiency syndrome resulting from developmental interferences and reactive learning

Behavioral observation of battered children reveals a generalized syndrome consisting of apathy, social isolation, emotional constriction, aggression, and intellectual, motor and language retardation. This syndrome is considered to be a ‘behavioral deficiency syndrome.’ Developmental interference imposed by abusive parents upon the child, and subsequent reactive learning by the child are discussed as etiological factors in the development of this syndrome. Implication of these etiological factors for therapy of the abusive family are also discussed.     

Accountability revisited: the arrival of client outcome evaluation

In the past few years, "accountability" for public mental health programs has become differentiated in the minds of not only program evaluators, but also program managers and funders, including state and local-level legislators. Increasingly, these officials are becoming concerned with more than just the numbers and targets of services delivered, and the cost involved, and are looking for evidence of positive outcome or impact on clients to justify program implementation and maintenance. This represents a significant move beyond the two accountability models that most recently seemed to be the focus of most formal accountability efforts--performance measurement and quality assurance. Pressures for implementing these two alternatives seem to have been reduced somewhat by the new federal Administration, but even prior to its advent there had been a rapid escalation in awareness of and concern for client outcome measurement among important audiences, including state and local mental health policy-makers and the U.S. Congress. This presents a major new opportunity and challenge for program evaluators at this new accountability focus continues to gather momentum.     

Employee counseling services evaluation system: design, issues and conclusions

Employee Counseling Services (ECS) programs have emerged as an effective method for dealing with employees whose job performance has deteriorated due to personal problems such as alcoholism or drug abuse. It is estimated that 18% of any work force is effected by such problems at any given time and that these employees cost industry billions of dollars annually in lost productivity, abuse of sick leave, etc. One of the critical concerns of this emerging field is the need for comprehensive evaluation that can demonstrate the cost-effectiveness and cost-benefits of ECS programs. This paper will describe the model Federal ECS developed at the U.S. Department of Health and Human Services as well as the components of the evaluation system designed to determine its effectiveness. The system is the first of its kind and will be a valuable contribution to the field. The paper then raises issues and develops conclusions about designing such an evaluation within the context of a large bureaucracy.     

The application of PERT in evaluation of human service programs

In this study a strategy well known to industry. Project Evaluation and Review Technique (PERT), was used to evaluate the community placement procedures in one institution for retarded. Case file review of two samples of residents was used to develop PERT networks showing movements between various living and vocational training settings. Analysis of these networks provided data and pinpointed needed procedural changes to meet the placement objective. The large number of recommendations derived from PERT and supplemental data provided evidence that PERT can be a valuable evaluation package component in institutions for the retarded. Evaluation applications in other human service systems were suggested.     

A cost-benefit analysis of a smoking cessation program

Smoking cessation programs are usually evaluated in terms of the percentage of participants who have stopped smoking at least 6 months after the program's completion. This paper shows how a relatively low rate of behavior change may actually result in a high ratio of benefits to costs. Cost-benefit analysis is done for a specific program but the procedures and cost estimates are generalizable to other smoking cessation programs. Benefits to the firm included reduced costs of insurance and the savings due to employee absenteeism and disability. Costs included the program costs as well as the opportunity costs of the participants. Data from the literature and from the specific program are combined to form conclusions about the effectiveness of the program.