Mental health program evaluation: Where you start?

Seduced by the elegance, sophistication and jargon characteristic of new developments in applied research, practicing evaluators are in danger of becoming distracted from achievement of their evaluative goal: that of providing timely, reliable and useful data to program management to facilitate rational data-based decision-making. To avoid such seduction, the author advocates adaptation of simple, inexpensive and efficient evaluative methods tailored to organizational needs and presentation of findings in language readily understood by constituent groups. Likewise, insistance by evaluators upon maintenance of a strict role boundary — that is, limiting role functioning exclusively to those tasks considered evaluation — may be dysfunctional. Rather than detract from accomplishment of his goals, the author maintains that such participation allows development of skills, insights and staff relationships which enhance the evaluator's effective functioning within the organizational context and contribute to organizational acceptance and use of his evaluative results.     

Evaluating a private foundation's health program

Public policy decisions in health are increasingly difficult and expensive. Although there will never be enough information available, private foundations can help to bridge the most important gaps in knowledge. Larger foundations may also wish to respond to those who doubt the value of foundation activities. This article reviews the experiences over the past eight years of The Robert Wood Johnson Foundation in employing evaluation and related social research procedures in the planning and implementation of a major philanthropic effort to improve the health and medical care of Americans. Discussed are the still evolving Foundation evaluation framework, the unanticipated problems in undertaking specific evaluations, and the substantive findings of some of the studies.     

Network analysis in evaluation: some words of caution

Network analysis, a methodology derived from general systems' theory, can be utilized as a community mental health administrative-evaluation procedure. Evaluation parameters derived from analysis of patient data as they “flow” through a network of agency services, provide measures of systemic functioning. These parameters include “longest paths” and various ratio relationships as evaluation measures of service delivery. The limitations of network analysis are examined by means of conceptual analyses, and phenomena that emerged from research experience. The necessity for both quantitative and qualitative data to ensure a meaningful evaluation of mental health services is explained. Conclusions about the value of the network analysis approach are considered.     

Artifact in client satisfaction assessment: experience in community mental health settings

Artifact in client satisfaction assessment is discussed and the results of a study of three factors thought to mediate client satisfaction ratings; (a) general life satisfaction, (b) mode of administration, and (c) psychological symptomatology, are reported. A standard client satisfaction questionnaire (CSQ) was modified to yield parallel forms and was administered orally and in writing to 92 clients in two mental health day treatment programs. Satisfaction ratings obtained from these clients were quite similar to out-patient ratings obtained in previous studies conducted in this setting and using the same measures. Oral administration of the CSQ produced 10% higher satisfaction ratings than written administration (p less than .05) and less missing data (p less than .01). Satisfaction ratings were also obtained using a simple graphic instrument. Graphic ratings were comparable to CSQ ratings. Satisfaction with life in general and level of psychiatric symptoms together accounted for 25% of CSQ variance. The implication of these findings for future client satisfaction research is discussed.     

The effects of psychiatric symptoms on quality of life assessments among the chronic mentally ill

This study of chronically mentally disabled persons in community residences examined the discriminant validity of subjective quality of life indicators and self-report mental health indices to determine the potential confounding effects of psychopathology on the assessment of quality of life (QOL). Factor analyses and difference-score reliabilities identified a general QOL construct and a general mental health construct with 27% common variance. However, psychiatric symptoms did not significantly (p less than .05) alter the bivariate and multivariate relationships among the QOL ratings, except in the health domain in which the correlations of health-related QOL indicators with global QOL were significantly (p less than .05) attenuated after removing the effects of psychopathology. The results suggest that psychopathology does not introduce bias into the overall structure of QOL data, but they also indicate the importance of controlling for mental health effects in the assessment of patients' self-rated health and satisfaction with health care.     

The validity of indices for rural health manpower needs assessment

Population-to-practitioner ratios have long been the primary index in the designation of health manpower shortage areas. This paper documents that application of the widely used population-to-dentist index results in understatement of the need for dental health manpower in rural areas. Through the analysis of utilization data collected from a statewide health screening program in Colorado, the practice of sole reliance on the population-to-dentist indices as an indicator of need was tested. Another measure, the area-(square miles) to-dentist ratio was formulated, examined, and found to be a more useful referent of the need for additional health manpower in rural areas. Utilization of dental services in sparsely settled rural counties of Colorado was unrelated to population-to-dentist ratios. A strong, statistically significant association of utilization with land area-to-dentist ratios was found. The findings of this analysis suggest a need for reevaluation of needs assessment methodologies used in the designation of health manpower shortage areas. Indices more sensitive to consumer circumstance than to the number of health care providers available must be considered.     

Using macro variables in program evaluation

This paper discusses the importance of using macro-level variables for program analysis. It is argued that three types of macro variables--cultural, economic, and demographic--provide a context for evaluation, and offer alternative explanations for the success or failure of a program. The problems of causality and availability of data are discussed. Macro variables are seen to be especially useful in "meta evaluation," and in integrating the results of specific program evaluations into general conclusions.     

The utilization of prevailing evaluation policy in the selection of performance measures

This study considers the problem of inferring a functional evaluation criterion from existing performance measure policies. The objective is to develop a rational basis by which to adaptively evaluate a current best set of performance measures from alternative sets. The analysis is based on a matrix approach for relating system characteristics to performance measures across a set of activities. A procedure for ranking measure combinations which are candidate implementation alternatives is devised using this matrix representation and cost information. An example illustrating the procedure is presented.     

Behavioral evaluation of a state program of deinstitutionalization of the developmentally disabled

The purpose of this study was to evaluate one phase of a statewide program of deinstitutionalizing developmentally disabled individuals. Forty adults who had spent a minimum of six months in community placements were evaluated with respect to changes in behavior associated with movement out of state institutions and into community settings such as group homes and day activity centers. Pre- and post-deinstitutionalization measures were taken with the Behavior Development Survey. Post-deinstitutionalization measures on Cataldo and Risley's Resident Activity Manifest were compared to the same measures obtained on 159 developmentally disabled individuals scheduled for deinstitutionalization. A consistent pattern of positive changes on both instruments favored deinstitutionalization, but changes could not be unequivocally attributed to movement out of the institutions per se.