Artifact in client satisfaction assessment: experience in community mental health settings

Artifact in client satisfaction assessment is discussed and the results of a study of three factors thought to mediate client satisfaction ratings; (a) general life satisfaction, (b) mode of administration, and (c) psychological symptomatology, are reported. A standard client satisfaction questionnaire (CSQ) was modified to yield parallel forms and was administered orally and in writing to 92 clients in two mental health day treatment programs. Satisfaction ratings obtained from these clients were quite similar to out-patient ratings obtained in previous studies conducted in this setting and using the same measures. Oral administration of the CSQ produced 10% higher satisfaction ratings than written administration (p less than .05) and less missing data (p less than .01). Satisfaction ratings were also obtained using a simple graphic instrument. Graphic ratings were comparable to CSQ ratings. Satisfaction with life in general and level of psychiatric symptoms together accounted for 25% of CSQ variance. The implication of these findings for future client satisfaction research is discussed.     

The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome

An 18-item version of the Client Satisfaction Questionnaire (CSQ-18) was included in an experimental study of the effects of pretherapy orientation on psychotherapy outcome. The psychometric properties of the CSQ-18 in this study were compared with earlier findings. In addition, the correlations of the CSQ-18 with service utilization and psychotherapy outcome measures were examined. Results indicated that the CSQ-18 had high internal consistency (coefficient alpha = .91) and was substantially correlated with remainer-terminator status (rs = .61) and with number of therapy sessions attended in one month (r = .54). The CSQ-18 was also correlated with change in client-reported symptoms (r = -.35), indicating that greater satisfaction was associated with greater symptom reduction. Results also demonstrated that a subset of items from the scale (the CSQ-8) performed as well as the CSQ-18 and often better. The excellent performance of the CSQ-8, coupled with its brevity, suggests that it may be especially useful as a brief global measure of client satisfaction.     

Harnessing the reliability of outcome measures

A method is presented for engineering the necessary levels of measurement reliability for evaluating ongoing programs. Two studies of levels of client functioning at a community mental health center (CMHC), in which an outcome measure, the Global Assessment Scale (GAS), did not perform as expected, drew attention to the need for better control of outcome measure reliabilities. Drawing from generalizability theory, a study was conducted of three sources of GAS score variance — clients, raters, and training in the use of the scale. Several estimates of reliability (ERs) were developed, depending on the manner in which the GAS ratings were, or would be, obtained in the CMHC. The differences among these ERs clarified why the GAS had lower reliability when used in our setting. Finally, two hypothetical examples are described to illustrate the utility of applying generalizability theory to achieve higher reliabilities for outcome measures.     

Defining the severely mentally disabled

A definition of the "known target population of severely mentally disabled" in community mental health centers is examined. Criteria include intensive treatment contact and persistent, severe disability, but not diagnosis. Feasible and reliable screening procedures are demonstrated. In the identified population, most subjects at some time had received a diagnosis of schizophrenia and had been on public assistance. These criteria apply to catchmented treatment systems and should be useful in services research.     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

Enhancing program review for evaluation and management

Program review has not received the attention it warrants as a program evaluation tool despite its wide use for evaluation and management purposes. The use of the program review will probably endure on the strength of its face validity and irrespective of other developments in the field of program evaluation. Evaluators should realize this and, accordingly, attempt to improve its effectiveness. This paper presents one organization's approach to achieving this objective through the explication of development principles, implementation guidelines and review items. This paper also discusses benefits that can be expected from a systematic development of this tool and presents various research directions and potentials in this area.     

The evaluator as program consultant

This paper presents a model for the role of the evaluator in which the evaluator acts as a Program Consultant who operates in three domains: Program Development, Planned Change, and Evaluation Technology. This permits the performance of evaluations that have greater validity and utility. The model is presented both from the theoretical perspective of linkage and from the context of the evaluation of a Case Management program operated by a Community Action Agency. Each of the domains is discussed with examples from the Case Management Evaluation. The paper concludes with a discussion of the contexts in which the model is best applied.     

Child global rating scales: Selecting a measure of client functioning in a large mental health system

The Global Assessment Scale for Children (GAS-Children) and the Children's Impairment Scale (CIS) were examined for inter-rater reliability, discrimination of outpatients from inpatients, and clinician acceptance. Forty-four clinicians used the two scales to rate 146 recently admitted children and adolescents in eight mental health programs. An additional study assessed the inter-rater reliability of the two scales compared with the Connors Parent-Teacher Questionnaire and the correlation of the GAS-Children with both the Connors Parents Questionnaire and Achenbach's Child Behavior Checklist. The GAS-Children showed better interrater agreement than any of the four subscales of the CIS, although the sum of the CIS subscales also showed adequate inter-rater reliability. Among adolescents, the GAS-Children correlated highly with the adult GAS, but the scales showed different mean values. Clinicians slightly preferred the GAS-Children over the CIS. To assess comparability of scale usage across sites, clinicians rated six case vignettes. Inpatient clinicians rated the vignettes as more dysfunctional than did their outpatient counterparts. Bias-adjusted scores still discriminated outpatient from inpatient children. Nevertheless, these rater biases should make evaluators cautious about comparing functioning scores across programs, even when the rating scale is ostensibly the same.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

Treatment participation and outcome in a program for problem drinker-drivers

Relationships were examined between 13 predictor variables and two criteria of improvement in a program for problem drinker-drivers. Four measures of social background (income, educations, age, marital status), four measures of psychopathology based on the Minnesota Multiphasic Personality Inventory (MMPI), one measure of general alcoholism, and four indicators of treatment involvement served as predictor variables. Two factorially derived criteria served as outcome indicators, one reflecting general improvement, the other reflecting improved social adjustment, but a failure to reduce drinking. This second factor was interpreted as indicating resistance. Analysis included partial correlation and multiple regression, from which explained variance was completely partitioned into sources unique to each predictor, and sources shared by combinations of predictors. Results suggested that with other factors controlled, treatment involvement was linked both to improvement and to resistance. Similar conclusions were suggested for social background, but slightly different ones for psychopathology and alcoholism. Psychopathology was uniquely related to improvement; alcoholism, to resistance.     

Professionalization of evaluative research: Conflict as a sign of health

Morell and Flaherty (1978) present arguments about the current status of evaluative research as a profession. This paper contains a critique, using a process model of professional development, of the perspectives employed by Morell and Flaherty. Issues discussed are the nature and substance of conflicts in evaluative research, including conflicts over methodological perspectives, the role of the evaluator in program planning, and the appropriate uses and users of evaluative information. The paper concludes with a discussion of the needs within the profession of evaluative research if it is to remain viable and responsive to change.     

Assessment of client/patient satisfaction: development of a general scale

The development and shaping of a general scale to assess client/patient satisfaction is reported. The scale, the CSQ, was constructed empirically by the authors. The CSQ is a response to several problems and issues that currently cloud the measurement of consumer satisfaction in health and human service systems. These problems and issues in assessing satisfaction are described. Finally, we present practical expriences to date in using the CSQ along with general psychometric qualities of the scale and correlations of CSQ results with client characteristics, service utilization, and service outcomes.