Artifact in client satisfaction assessment: experience in community mental health settings

Artifact in client satisfaction assessment is discussed and the results of a study of three factors thought to mediate client satisfaction ratings; (a) general life satisfaction, (b) mode of administration, and (c) psychological symptomatology, are reported. A standard client satisfaction questionnaire (CSQ) was modified to yield parallel forms and was administered orally and in writing to 92 clients in two mental health day treatment programs. Satisfaction ratings obtained from these clients were quite similar to out-patient ratings obtained in previous studies conducted in this setting and using the same measures. Oral administration of the CSQ produced 10% higher satisfaction ratings than written administration (p less than .05) and less missing data (p less than .01). Satisfaction ratings were also obtained using a simple graphic instrument. Graphic ratings were comparable to CSQ ratings. Satisfaction with life in general and level of psychiatric symptoms together accounted for 25% of CSQ variance. The implication of these findings for future client satisfaction research is discussed.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Documentation in evaluation research: Managerial and scientific requirements

Documentation in evaluation research consists of written material, in human- or machine-readable form, pertaining to the plans, activities, and results of the project. It is argued here that good documentation is essential for effective management of evaluations, and for responsible reporting of the research procedures and findings. Documentation relating to electronic data processing activity is especially important. The purpose of this paper is to stimulate consideration and discussion of documentation, and to emphasize its importance in evaluation research. The role of documentation in the planning and control functions of project management is reviewed, and the importance of documentation in the assessment of research quality with respect to objectivity, validity, and replicability is discussed. Reasons for poor documentation are considered. An outline of documentation required in different phases of research projects is provided, and recommendations for improving the quality of documentation are presented.     

A validity scale for the Sharp consumer satisfaction scales.

Because of the apparently inflated scores commonly reported for consumer satisfaction with mental health services, we developed a 5-item validity scale for the SHARP questionnaire. The scale discriminated significantly (P less than .001) between those former clients who offered suggestions for improvement of services and those who did not (N = 38). In another experiment, the scale significantly improved the ability to predict the 46 clients' true scores from their obtained scores via stepwise regression (p less than .01). Weights were obtained from that sample of former clients, which significantly reduced the deviation of the obtained satisfaction scores from the true scores, with all p less than .05. In two successive experiments (N = 49 and 26), weights were applied to new samples, significantly reducing deviation from the criterion scores for 5 of the 6 scales (p less than .05), and new weights were then derived. A final set of weights was derived from the combined samples of the three experiments, (N = 121) which significantly improved prediction of the criterion and lowered the mean on all scales, with all p less than .05.     

The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome

An 18-item version of the Client Satisfaction Questionnaire (CSQ-18) was included in an experimental study of the effects of pretherapy orientation on psychotherapy outcome. The psychometric properties of the CSQ-18 in this study were compared with earlier findings. In addition, the correlations of the CSQ-18 with service utilization and psychotherapy outcome measures were examined. Results indicated that the CSQ-18 had high internal consistency (coefficient alpha = .91) and was substantially correlated with remainer-terminator status (rs = .61) and with number of therapy sessions attended in one month (r = .54). The CSQ-18 was also correlated with change in client-reported symptoms (r = -.35), indicating that greater satisfaction was associated with greater symptom reduction. Results also demonstrated that a subset of items from the scale (the CSQ-8) performed as well as the CSQ-18 and often better. The excellent performance of the CSQ-8, coupled with its brevity, suggests that it may be especially useful as a brief global measure of client satisfaction.     

Problem characteristics, decision processes and evaluation activity: a preliminary study of mental health center directors

This article reports on the results of a preliminary study on the characteristics of problems and problem solving in mental health centers. The major findings were that directors were generally quite satisfied both with the available level of information intended to help them make decisions, and with their ability to make good, even optimal, decisions. Much of this confidence resulted from the use of assumptions about the problems which reduced the need for empirical information. The problem solving process was highly dynamic and interactive, with changes over time in the characteristics of problems, constraints and alternative solutions. The directors played an active role in attempting to foster and influence the direction of many of these instabilities. Some implications for evaluation and research are discussed.     

Handling problems of ecological fallacy in program planning and evaluation

The issue of ecological fallacy is highlighted in the use of social area analysis as a method of program planning and evaluation. Data on psychiatric admissions from one catchment area in New York City show misleading cross level generalization to be a real and ever present danger that planners and evaluators should not ignore. A simple method of assessing the degree and major sources of difference between higher (geographic) and lower (sociological) level relationships is illustrated. Suggestions are made regarding ways of clarifying cross level generalizations.     

Evaluating a management and organization development effort in mental health agencies.

Although the use of management and organization development techniques in health and human service settings is increasing, there are few systematic studies of their effects. Those which do exist are problematic in many respects. An 18-month program incorporating both management and organization development is described along with the results of its evaluation, which were positive and well-utilized. A discussion of the methods employed, the problems encountered, and the outcomes obtained illustrates that these evaluation efforts are needed and can be useful to program evaluators involved in planned-change efforts.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.     

To be or not to be: Control and balancing of type I and type II errors

The various costs of Type I and Type II errors of inference from data are discussed. Unfortunately, other things equal, Types I and II errors are inversely related. Six methods of minimizing Type I error in studies are presented, each of which may be employed even after data are collected. Type II errors may also be minimized by a combination of study design and analytic means. Although some of these techniques must be built into the study others can be undertaken when beginning work on a data set. Only one of these techniques will increase Type I error, and those involving data reduction may decrease both types of error.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

A user-focused model for the utilization of evaluation

It is proposed that the present low level of utilization of evaluation findings is traceable in part to their failure to address directly the information needs of a clearly specified decision maker. An alternative model proceeding from such specification is proposed here, with evaluation closely interwoven with the on-going innovation process. The model suggests a number of implications for the organizational role of evalutors, for the design of evaluations, and for directions for methodological development. In particular, we suggest that evaluation designs be assessed against possible “threats to utility” as well as against the traditional “threads to validity.”     

A survey of mental health district structure and effectiveness

A national survey using a mailed questionnaire was undertaken to clarify issues of mental health district governance. CMHC, state hospital and district management and governance staff from 49 identified districts classified their districts on 20 types of decision authority and 14 measures of district effectiveness. Each respondent type thought their districts: (1) dealt with state money issues, seeking regional money, and coordinating/changing programs and services, and (2) effectively enhanced planning capabilities. Service providers did not find planning capabilities to be systematically related to specific decision authorities. CMHC management perceived less district control than district officials and, with state hospital management, perceived less effectiveness. District and CMHC officials see a direct relationship between district control and effectiveness. In a subsample of 11 districts each described by multiple respondent types, no item of authority or effectiveness was rated universally positive by all groups. District management subjectivity and legislative implications are discussed.     

Quality of life assessment and drug abuse treatment program evaluation

Quality of life assessment is used as an outcome measure technique to evaluate a central medical intake (CMI) unit for drug treatment programs. While significant improvements across time are evident in both perceived well-being and in the standard federal funding criteria, no differences are observed between those referred into treatment by the CMI and those entering directly. Moreover, the improvements observed are the same regardless of whether the patient stayed in treatment or not.     

Dealing with unknown variables in policy/program evaluation

A leading objection to the feasibility of applying benefit-cost analysis in evaluation is that one often cannot know the benefits, the costs, or the probabilities of their occurring for a given proposal. Threshold analysis involves handling unknown variables by converting relevant evaluation problems into questions as to whether a given benefit, cost, probability, or combination of these elements is more or less than a threshold. Above the threshold, the proposed program would be relatively profitable; and below the threshold, it would be relatively unprofitable. Applying that analysis may require the drawing of simple graphs to stimulate the minds of knowledgeable persons as to the range of the actual benefits, costs or probabilities. The approach can be applied when the benefits or the costs are either monetary or nonmonetary, and when the problems involve either go/no-go decisions or conflicting-choice decisions.     

Pitfalls in planning: Contrasting perspectives on mental health and corporate planning

Planning is a management activity that has been linked to high quality services, organizational growth, and in austere times, to organizational survival. The present study was undertaken to identify the critical barriers that inhibit or derail effective planning. It was predicted that human service and corporate managers would generally agree about the most and least important planning pitfalls in their systems. Discrepancies would be related to differences in the structural characteristics of organizations in the two sectors. Rankings made by 57 senior mental health administrators of the most and least important pitfalls in planning were contrasted with those of 159 corporate executives. Substantial agreement between the two groups of executives was found.