Evaluating employee assistance programs. A review of methods, outcomes, and future directions

Renewed interest in assisting troubled employees has led to an upsurge in the development of employee assistance programs, coupled with demands for demonstrable effectiveness. This review examines the nature and scope of these programs, their administrative and methodological context, and the types and outcomes of evaluation studies conducted thus far. Proposals for improving future investigations through a number of different approaches and strategies are then made.     

The Tellado case study exercise: A metacritique

This paper provides an overview and critique of the papers in the Tellado Case Study Exercise. Two main themes are found in the critiques: the belief that Tellado lacked a clear audience for his evaluation and that he lacked an appropriate change orientation that would insure utilization. The sources of these criticisms are discussed from the perspective of evaluation as a linkage activity. The use of this perspective leads to the recognition that all parties to the case study may have focused on the wrong stakeholders in the utilization process. Thus, a major problem with utilization-focused and stakeholder based approaches to evaluation is revealed. Possible ways to avoid this problem are discussed. The paper concludes with a request for additional case studies to be contributed to the journal.     

Using an alternative evaluation measure for assessing juvenile diversion programs

This paper describes the effectiveness of six juvenile diversion programs in the metropolitan area of Denver, Colorado. Program success was measured by recidivism reduction. However, by using a one-for-one statistical client-baseline matching method and a variable definition of recidivism, we managed to overcome drawbacks that often occur when using recidivism as an outcome indicator. Together, program participants showed a 26% reduction in recidivism rate over that of a matched baseline sample. Although four of the six programs studied accounted for the overall success rate, we explain why two of the programs showed no significant reductions in recidivism rates.     

Response from a policymakers' perspective

The case in point (Tellado, 1984) needs to be defined in terms of a client for the research. The research may also have implications for early intervention to mitigate the problems uncovered in the case study. An important element in the development of problem-solving skills may be linguistic skills. Further research about this variable seems warranted.     

Identification of mother-child interaction patterns:: A longitudinal evaluation of early age intervention effects on early adolescents

Evidence is offered that meaningful patterns of mother-child interaction could be detected in families more than 10 years after a family-centered, early age, preventive intervention. The interactions of mother-child dyads in a situational test during the child's early adolescence were examined for three experimental conditions: a lower socioeconomic experimental group that participated in the intervention for approximately 2 years, an untreated lower socioeconomic, and an untreated upper middle socioeconomic control group.Factor analyses of family interaction measures revealed three factors, which were called Democratic-Participatory Interaction, Mixed Dominance Interaction, and Mother Controlled Interaction. Separate ANOVAS based on factor scores were performed for each factor. Both the intervention group and the middle socioeconomic comparison group were significantly higher than in the lower socioeconomic comparison group in the first factor.     

Legal standards for evaluation research: A reply to Smith

Smith (1981), asserting that not all evaluative studies require the same degree of evidence for decision-making, suggests a legal paradigm for determining how much certainty is necessary for informed decision-making. Specifically, he argues for the use of three levels of certainty, i.e., suggestive, preponderant, or conclusive evidence, modeled after the legal standards of proof. Unfortunately, such an approach is untenable for several reasons. Smith's suggestion that decision-makers adopt legal terminology, for example, is really a recommendation of form rather than substance and, furthermore, his analogy from the law to the evaluation context is based upon a misunderstanding of the trial process. These and other problems with Smith's suggestions are discussed.     

An alternative evaluation approach for the problem-solving training program: A utilization-focused evaluation process

A utilization-focused approach in evaluating this program would have placed more emphasis on clearly identifying evaluation users, their information needs, and likely use of findings. With the advantages of hindsight, it appears that such an approach might have led to greater focus on critical implementation issues that affected the eventual decision not to continue the program despite seemingly positive outcomes evaluation findings. Other methods options are also discussed, including the potential usefulness of critical case studies. Finally, there is discussion of how to prepare decision makers for utilization and work with them to interpret and apply findings.     

A systems method for the identification of variable service needs of the elderly

Current trends in the aging sector make critical the development of more precision in the policy and decision-making processes. This study explores a strategy for improving service allocation decisions in the aging services area. Two research tools were combined—Delphi and magnitude estimation—to identify the perceptions of key decision makers about the variable service needs of different aging risk groups. The results indicate that such distinctions can be made and can provide the basis of improving several aspects of the allocation process.     

Correlates of honor ratings in a clinical clerkship employing a faculty forum evaluation system

5he purpose of this study was to determine whether or not suspected variables affected a surgery clerk's chances of being awarded an honor rating. Findings indicated a significant relationship between a student receiving an honor rating and his or her preceptor's predetermined level of student advocacy, the number of completed patient interview and physical examination write-ups, and final examination scores. There was no significant relationship found between honor ratings and the preceptor's status, the sequence in which the student was discussed or the length of time spent discussing the student at the final evaluation meeting, or the number of clinical faculty present at the meeting. These results lend support to the faculty forum evaluation approach, but suggest a need for further scrutiny of some influencing variables to ensure all students are fairly considered and honor ratings judiciously awarded.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

Political participation: A comparative view of Israeli and American youth

This study explored the ways in which adolescents in Israel and the United States perceived government, politics, and public affairs. A series of open-ended interviews were conducted with approximately 300 high school students in each country immediately prior to the national elections in 1980–1981. Differences were found in regard to the extent to which the young people felt affected by and capable of affecting the larger political system, the extent to which voting was viewed as an important rite of passage to adulthood, and the extent to which they perceived having real choices in candidates and parties, with Israeli youth generally demonstrating more engagement in national governmental affairs. Finally, the conditions and factors that were likely to lead adolescents toward concern, interest, and participation in public affairs were offered.     

Road signs in ethical quicksand

This commentary highlights some of the major moral and ethical dilemmas illustrated by authors in this special issue. From an ethical viewpoint, the complexity of evaluation activities is likened to “quicksand” terrain in which directional or warning signs should be erected to aid personal choices when such issues are encountered. However, even previous training and personal introspection are inadequate to completely avoid, circumvent, or handle a specific situation because of its complexity and the competing values inherent in ethical choices.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

Toward overcoming problems in evaluation research: A beginning perspective on power

An information feedback model and an ideology of evaluation is posed. The authors' experiences in actually doing evaluation research are then contrasted with the model and ideology. Suggestions are made for closing the gap between reality and the model, primarily by altering the structural constraints that present obstacles to achieving the model. The authors assume that impact on policy is the appropriate criterion for the success of an evlauation, and they conclude that political constraints are primary in preventing evaluation research from having this impact. It is proposed that evaluators need to develop more explicitly political ways of acquiring power of their own in order to maximize such impact. Some ways social researchers might conceptualize and construct an appropriate base of power are considered.     

A user-focused model for the utilization of evaluation

It is proposed that the present low level of utilization of evaluation findings is traceable in part to their failure to address directly the information needs of a clearly specified decision maker. An alternative model proceeding from such specification is proposed here, with evaluation closely interwoven with the on-going innovation process. The model suggests a number of implications for the organizational role of evalutors, for the design of evaluations, and for directions for methodological development. In particular, we suggest that evaluation designs be assessed against possible “threats to utility” as well as against the traditional “threads to validity.”