“Paradigms lost”: Some thoughts on choosing methods in evaluation research

Patton agrees with us concerning the need for a catholic approach to method choices in evaluation. However, Patton criticizes us for setting up a straw man when we argued that quantitative and qualitative methods are not inevitably linked to different “paradigms” and that allegiance to a particular paradigm does not force the evaluator into a choice between mutually exclusive qualitative and quantitative methods. The first part of this article shows how some of the past literature creates the impression both of an inevitable method-paradigm linkage and of a forcedchoice between qualitative and quantitative paradigms. It is argued that such impressions could have pernicious consequences for evaluation practice by artificially restricting method choices. In the final section, we consider some of the difficulties of making method choices and suggest that the purposefully developed strengths of different methods should not be completely ignored.     

Behavioral evaluation of a state program of deinstitutionalization of the developmentally disabled

The purpose of this study was to evaluate one phase of a statewide program of deinstitutionalizing developmentally disabled individuals. Forty adults who had spent a minimum of six months in community placements were evaluated with respect to changes in behavior associated with movement out of state institutions and into community settings such as group homes and day activity centers. Pre- and post-deinstitutionalization measures were taken with the Behavior Development Survey. Post-deinstitutionalization measures on Cataldo and Risley's Resident Activity Manifest were compared to the same measures obtained on 159 developmentally disabled individuals scheduled for deinstitutionalization. A consistent pattern of positive changes on both instruments favored deinstitutionalization, but changes could not be unequivocally attributed to movement out of the institutions per se.     

Accountability to whom?

Social scientists are participating evermore frequently in the evaluation of social action programs. Participation, however, does not always imply social action. In keeping with the tradition of social sciences as critical and debunking activities, practitioners might consider the merits of conceiving themselves as agents of social change rather than technicians responding to emissaries of social control. This position also implies a view of applied research as an active enterprise through which reflective practitioners participate in the definition and direction of applied research efforts. After reviewing implications of non-reflective and reflective stances, the conclusion is that applied researchers adopting a social action perspective actually maximize their obligations as scholar-scientists even though a non-neutral posture is assumed.     

Classic 1960s articles in educational evaluation

The primary thrust of the interest in educational evaluation which persists today began with the passage of the Elementary and Secondary Education Act of 1965. A study of journal citations was used to identify which of the early 1960s writings have continued to influence evaluation thought in education throughout the 1970s. Five classic articles were identified and reasons for their continued influence are suggested.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

Enhancing program review for evaluation and management

Program review has not received the attention it warrants as a program evaluation tool despite its wide use for evaluation and management purposes. The use of the program review will probably endure on the strength of its face validity and irrespective of other developments in the field of program evaluation. Evaluators should realize this and, accordingly, attempt to improve its effectiveness. This paper presents one organization's approach to achieving this objective through the explication of development principles, implementation guidelines and review items. This paper also discusses benefits that can be expected from a systematic development of this tool and presents various research directions and potentials in this area.     

The validity of indices for rural health manpower needs assessment

Population-to-practitioner ratios have long been the primary index in the designation of health manpower shortage areas. This paper documents that application of the widely used population-to-dentist index results in understatement of the need for dental health manpower in rural areas. Through the analysis of utilization data collected from a statewide health screening program in Colorado, the practice of sole reliance on the population-to-dentist indices as an indicator of need was tested. Another measure, the area-(square miles) to-dentist ratio was formulated, examined, and found to be a more useful referent of the need for additional health manpower in rural areas. Utilization of dental services in sparsely settled rural counties of Colorado was unrelated to population-to-dentist ratios. A strong, statistically significant association of utilization with land area-to-dentist ratios was found. The findings of this analysis suggest a need for reevaluation of needs assessment methodologies used in the designation of health manpower shortage areas. Indices more sensitive to consumer circumstance than to the number of health care providers available must be considered.     

A multi-dimensional client satisfaction instrument

Development of a client satisfaction measure through successive factor analyses is described. Five factors, each defined by five items and accounting for at least 11% of the total variance, were identified and replicated. A single second order factor also emerged in both studies. Reliability of the second order factor, and of three of the first order factors is sufficient for most purposes. The second order factor, which is a linear combination of the first order factors, was not significantly correlated with sex, age, race, or number of attempts made before receiving a client response. The distribution of scores was highly skewed, with most respondents scoring near the upper (satisfied) end.     

The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome

An 18-item version of the Client Satisfaction Questionnaire (CSQ-18) was included in an experimental study of the effects of pretherapy orientation on psychotherapy outcome. The psychometric properties of the CSQ-18 in this study were compared with earlier findings. In addition, the correlations of the CSQ-18 with service utilization and psychotherapy outcome measures were examined. Results indicated that the CSQ-18 had high internal consistency (coefficient alpha = .91) and was substantially correlated with remainer-terminator status (rs = .61) and with number of therapy sessions attended in one month (r = .54). The CSQ-18 was also correlated with change in client-reported symptoms (r = -.35), indicating that greater satisfaction was associated with greater symptom reduction. Results also demonstrated that a subset of items from the scale (the CSQ-8) performed as well as the CSQ-18 and often better. The excellent performance of the CSQ-8, coupled with its brevity, suggests that it may be especially useful as a brief global measure of client satisfaction.     

Developmental trends in program evaluation

It is suggested that the ontogeny of program evaluators replicates the phytogeny of the profession of program evaluation. A trend across the purposes of program evaluation from program amelioration to program advocacy and ultimately to program accountability is projected.     

Artifact in client satisfaction assessment: experience in community mental health settings

Artifact in client satisfaction assessment is discussed and the results of a study of three factors thought to mediate client satisfaction ratings; (a) general life satisfaction, (b) mode of administration, and (c) psychological symptomatology, are reported. A standard client satisfaction questionnaire (CSQ) was modified to yield parallel forms and was administered orally and in writing to 92 clients in two mental health day treatment programs. Satisfaction ratings obtained from these clients were quite similar to out-patient ratings obtained in previous studies conducted in this setting and using the same measures. Oral administration of the CSQ produced 10% higher satisfaction ratings than written administration (p less than .05) and less missing data (p less than .01). Satisfaction ratings were also obtained using a simple graphic instrument. Graphic ratings were comparable to CSQ ratings. Satisfaction with life in general and level of psychiatric symptoms together accounted for 25% of CSQ variance. The implication of these findings for future client satisfaction research is discussed.     

Using an alternative evaluation measure for assessing juvenile diversion programs

This paper describes the effectiveness of six juvenile diversion programs in the metropolitan area of Denver, Colorado. Program success was measured by recidivism reduction. However, by using a one-for-one statistical client-baseline matching method and a variable definition of recidivism, we managed to overcome drawbacks that often occur when using recidivism as an outcome indicator. Together, program participants showed a 26% reduction in recidivism rate over that of a matched baseline sample. Although four of the six programs studied accounted for the overall success rate, we explain why two of the programs showed no significant reductions in recidivism rates.     

Alternative systems of ethics and their application to education and evaluation

Three schools of moral thought are presented to illustrate differences in moral reasoning and the assignment of value claims. Several educational program alternatives are used as illustrations of moral reasoning within and across the schools of thought. Finally, the schools of moral thought are used in a discussion of evaluation practice. Good practice in the current evaluation literature is shown to be a blend of the three schools of thought.     

Cost/effectiveness analysis of alcoholism and drug abuse treatment programs: the relevance of recidivism and resource absorption.

Cost/effectiveness measures are increasingly being applied to alcoholism and drug abuse treatment programs. Program evaluators usually regard readmissions as inadequate for assessing outcome, and resource absorption is rarely examined; this paper presents the argument that increased attention to recidivism and resource absorption is integral to the program evaluation process. Readmissions formed 53% of admissions to U.S. drug abuse programs, and in alcoholism treatment centers, 4% of the patients accounted for 24% of the detoxications. Problems with assessing cost/effectiveness are associated with duration of treatment, average cost, time span of analyses, quasi-experimental designs, multiple episodes of treatment, "silting up", and extrapolation. Five approaches to dealing with these problems are suggested: analyzing existing program data bases for recidivism and resource absorption, considering recidivism when evaluating programs, tempering clinical enthusiasm with skepticism, matching patients with levels of care appropriate to previous treatment history, and assessing additional benefit derived from increasing amounts of care per patient as part of cost/effectiveness analysis.     

Harnessing the reliability of outcome measures

A method is presented for engineering the necessary levels of measurement reliability for evaluating ongoing programs. Two studies of levels of client functioning at a community mental health center (CMHC), in which an outcome measure, the Global Assessment Scale (GAS), did not perform as expected, drew attention to the need for better control of outcome measure reliabilities. Drawing from generalizability theory, a study was conducted of three sources of GAS score variance — clients, raters, and training in the use of the scale. Several estimates of reliability (ERs) were developed, depending on the manner in which the GAS ratings were, or would be, obtained in the CMHC. The differences among these ERs clarified why the GAS had lower reliability when used in our setting. Finally, two hypothetical examples are described to illustrate the utility of applying generalizability theory to achieve higher reliabilities for outcome measures.