Mental health program evaluation: Where you start?

Seduced by the elegance, sophistication and jargon characteristic of new developments in applied research, practicing evaluators are in danger of becoming distracted from achievement of their evaluative goal: that of providing timely, reliable and useful data to program management to facilitate rational data-based decision-making. To avoid such seduction, the author advocates adaptation of simple, inexpensive and efficient evaluative methods tailored to organizational needs and presentation of findings in language readily understood by constituent groups. Likewise, insistance by evaluators upon maintenance of a strict role boundary — that is, limiting role functioning exclusively to those tasks considered evaluation — may be dysfunctional. Rather than detract from accomplishment of his goals, the author maintains that such participation allows development of skills, insights and staff relationships which enhance the evaluator's effective functioning within the organizational context and contribute to organizational acceptance and use of his evaluative results.     

Program analysis using the client oriented cost outcome system.

The client oriented cost outcome system has been under development in Pennsylvania community and hospital programs since 1972. The system builds upon the behavioral and decision data generated with or on behalf of consumers. Flexibility for local program system design is permitted if the procedures of consumer intake, review, and termination document each consumer's (a) problems, resources, and goals, (b) overall functioning level in their ordinary community, and (c) services intended and rendered as related to (a), above. While the system's primary application is in providing feedback for local program quality assurance and evaluation procedures, aggregation of data permits program planning and evaluation at county, state, and federal levels in terms of (a) client demographic or diagnostic characteristics, and (b) program service characteristics and objectives.     

Evaluating a private foundation's health program

Public policy decisions in health are increasingly difficult and expensive. Although there will never be enough information available, private foundations can help to bridge the most important gaps in knowledge. Larger foundations may also wish to respond to those who doubt the value of foundation activities. This article reviews the experiences over the past eight years of The Robert Wood Johnson Foundation in employing evaluation and related social research procedures in the planning and implementation of a major philanthropic effort to improve the health and medical care of Americans. Discussed are the still evolving Foundation evaluation framework, the unanticipated problems in undertaking specific evaluations, and the substantive findings of some of the studies.     

Network analysis in evaluation: some words of caution

Network analysis, a methodology derived from general systems' theory, can be utilized as a community mental health administrative-evaluation procedure. Evaluation parameters derived from analysis of patient data as they “flow” through a network of agency services, provide measures of systemic functioning. These parameters include “longest paths” and various ratio relationships as evaluation measures of service delivery. The limitations of network analysis are examined by means of conceptual analyses, and phenomena that emerged from research experience. The necessity for both quantitative and qualitative data to ensure a meaningful evaluation of mental health services is explained. Conclusions about the value of the network analysis approach are considered.     

Artifact in client satisfaction assessment: experience in community mental health settings

Artifact in client satisfaction assessment is discussed and the results of a study of three factors thought to mediate client satisfaction ratings; (a) general life satisfaction, (b) mode of administration, and (c) psychological symptomatology, are reported. A standard client satisfaction questionnaire (CSQ) was modified to yield parallel forms and was administered orally and in writing to 92 clients in two mental health day treatment programs. Satisfaction ratings obtained from these clients were quite similar to out-patient ratings obtained in previous studies conducted in this setting and using the same measures. Oral administration of the CSQ produced 10% higher satisfaction ratings than written administration (p less than .05) and less missing data (p less than .01). Satisfaction ratings were also obtained using a simple graphic instrument. Graphic ratings were comparable to CSQ ratings. Satisfaction with life in general and level of psychiatric symptoms together accounted for 25% of CSQ variance. The implication of these findings for future client satisfaction research is discussed.     

Involving constituents in agency priority setting: a case study

Many government agencies have "constituents" who believe the agencies should hear and consider their needs and desires in establishing program priorities. This article describes how one involves interested constituents (consumers, industry, medical professionals, and state organizations) in the agency's annual priority-setting process. This participation has produced a number of beneficial results-some expected and some unexpected.     

The effects of psychiatric symptoms on quality of life assessments among the chronic mentally ill

This study of chronically mentally disabled persons in community residences examined the discriminant validity of subjective quality of life indicators and self-report mental health indices to determine the potential confounding effects of psychopathology on the assessment of quality of life (QOL). Factor analyses and difference-score reliabilities identified a general QOL construct and a general mental health construct with 27% common variance. However, psychiatric symptoms did not significantly (p less than .05) alter the bivariate and multivariate relationships among the QOL ratings, except in the health domain in which the correlations of health-related QOL indicators with global QOL were significantly (p less than .05) attenuated after removing the effects of psychopathology. The results suggest that psychopathology does not introduce bias into the overall structure of QOL data, but they also indicate the importance of controlling for mental health effects in the assessment of patients' self-rated health and satisfaction with health care.     

The validity of indices for rural health manpower needs assessment

Population-to-practitioner ratios have long been the primary index in the designation of health manpower shortage areas. This paper documents that application of the widely used population-to-dentist index results in understatement of the need for dental health manpower in rural areas. Through the analysis of utilization data collected from a statewide health screening program in Colorado, the practice of sole reliance on the population-to-dentist indices as an indicator of need was tested. Another measure, the area-(square miles) to-dentist ratio was formulated, examined, and found to be a more useful referent of the need for additional health manpower in rural areas. Utilization of dental services in sparsely settled rural counties of Colorado was unrelated to population-to-dentist ratios. A strong, statistically significant association of utilization with land area-to-dentist ratios was found. The findings of this analysis suggest a need for reevaluation of needs assessment methodologies used in the designation of health manpower shortage areas. Indices more sensitive to consumer circumstance than to the number of health care providers available must be considered.     

Using macro variables in program evaluation

This paper discusses the importance of using macro-level variables for program analysis. It is argued that three types of macro variables--cultural, economic, and demographic--provide a context for evaluation, and offer alternative explanations for the success or failure of a program. The problems of causality and availability of data are discussed. Macro variables are seen to be especially useful in "meta evaluation," and in integrating the results of specific program evaluations into general conclusions.