“Paradigms lost”: Some thoughts on choosing methods in evaluation research

Patton agrees with us concerning the need for a catholic approach to method choices in evaluation. However, Patton criticizes us for setting up a straw man when we argued that quantitative and qualitative methods are not inevitably linked to different “paradigms” and that allegiance to a particular paradigm does not force the evaluator into a choice between mutually exclusive qualitative and quantitative methods. The first part of this article shows how some of the past literature creates the impression both of an inevitable method-paradigm linkage and of a forcedchoice between qualitative and quantitative paradigms. It is argued that such impressions could have pernicious consequences for evaluation practice by artificially restricting method choices. In the final section, we consider some of the difficulties of making method choices and suggest that the purposefully developed strengths of different methods should not be completely ignored.     

The evaluation and selection of adequate causal models: a compensatory education example

Procedures for ascertaining relative model adequacy in latent variable structural relations models are discussed. Under diverse methods of estimation, this determination may be assessed using the chi square goodness of fit statistic, incremental fit indices for covariance structure models, and latent variable coefficients of determination. An example from evaluation research is taken (cf. Magidson, 1977; Bentler & Woodward, 1978). Numerical sensitivity of parameter estimates under alternative model specifications is demonstrated. Interpretive implications based on these procedures are discussed in terms of parameter sensitivity to alternative model specifications.     

A protective service system

This article presents the results of a small scale study of the protective service caseload of a mid-western metropolitan area. The basic characteristics of the caseload match the national picture presented by the American Humane Association's National Study on Child Abuse and Neglect. The present study focused on the degree to which “systematic” vs. “random” forces appear to be at work in the progress of cases through the system. The overall picture is of some systematic efforts to conduct the process of referrals to community agencies but of a great deal of “randomness” in the bureaucratic closing of cases. This is a major area of concern.     

Artifact in client satisfaction assessment: experience in community mental health settings

Artifact in client satisfaction assessment is discussed and the results of a study of three factors thought to mediate client satisfaction ratings; (a) general life satisfaction, (b) mode of administration, and (c) psychological symptomatology, are reported. A standard client satisfaction questionnaire (CSQ) was modified to yield parallel forms and was administered orally and in writing to 92 clients in two mental health day treatment programs. Satisfaction ratings obtained from these clients were quite similar to out-patient ratings obtained in previous studies conducted in this setting and using the same measures. Oral administration of the CSQ produced 10% higher satisfaction ratings than written administration (p less than .05) and less missing data (p less than .01). Satisfaction ratings were also obtained using a simple graphic instrument. Graphic ratings were comparable to CSQ ratings. Satisfaction with life in general and level of psychiatric symptoms together accounted for 25% of CSQ variance. The implication of these findings for future client satisfaction research is discussed.     

How experiential education relates to college goals and objectives

Goals and undergraduate education objectives related to experiential learning were identified in college brochures and through interviews with selected college personnel. All college faculty were mailed questionnaires asking the extent of their agreement with the goals and objectives and their opinion as to the learning environment most conducive to achieving these objectives.Lack of faculty agreement with experiential learning goals and faculty unwillingness to endorse off-campus learning experiences conflicted with the published philosophy of the college. A Guttman Scale constructed from faculty responses to goals showed that as the degree of abstraction of each goal increased, so did the degree of faculty agreement.Based on college goals, valid and reliable instruments were developed to determine students' attitudes about field experiences, and to assess performance in field placements. As amounts of fieldwork increased, students' attitudes about self became more positive, performances at field sites showed higher achievement, and understanding of college goal-related concepts increased. Any amount of fieldwork appeared to help focus students' professional goals.     

The efficacy of early intervention programs for children with organic impairment: A quantitative review

The results of studies examining the effectiveness of early intervention for infants and children with organic impairment and developmental delay were reviewed using recently developed quantitative methods that treat the literature review process as a unique type of scientific inquiry. Thirty-eight studies meeting certain predetermined criteria were included in the review. The 38 studies contained a total of 118 statistical hypothesis tests that evaluated the effectiveness of early intervention. An analysis of these tests based on the calculation of effect sizes revealed that subjects receiving early intervention performed better on a wide range of dependent measures than subjects not receiving intervention. The outcomes were found to be related to several design and study characteristics. Larger effect sizes were associated with preexperimental designs, and also with studies in which the internal validity was rated as poor. Several other design variables such as how subjects were assigned to conditions and how the dependent measure was recorded were related to study outcome as measured by effect size. The conclusion was made that an accurate interpretation of the early intervention research literature cannot be made without consideration of specific design variables and study characteristics.     

Alternative systems of ethics and their application to education and evaluation

Three schools of moral thought are presented to illustrate differences in moral reasoning and the assignment of value claims. Several educational program alternatives are used as illustrations of moral reasoning within and across the schools of thought. Finally, the schools of moral thought are used in a discussion of evaluation practice. Good practice in the current evaluation literature is shown to be a blend of the three schools of thought.     

Standard setting: the crucial issues. A case study of accounting & auditing

A study of standard-setting efforts in accounting and auditing is reported. The study reveals four major areas of concern in a professional standard-setting effort: (1) issues related to the rationale for setting standards, (2) issues related to the standard-setting board and its support structure, (3) issues related to the content of standards and rules for generating them, and (4) issues that deal with how standards are put to use. Principles derived from the study of accounting and auditing are provided to illuminate and assess standard-setting efforts in evaluation.     

What we know about child maltreatment

This article presents the results of an informal survey of nationally recognized experts on child maltreatment. The poll asked, “What do we know about child maltreatment?” A process of collating and sequential validation produced an outline of current knowledge that demonstrates some progress in understanding child maltreatment.     

Making a case for an interface between quasi-experimental designs and social impact analyses

This paper describes the community impact assessment conducted after Concorde operations were initiated at Dulles International Airport. Problems in the design and conduct of this study that make causal interpretations implausible are discussed. The present authors report an improved design to eliminate these problems as well as increase the external validity of the results. A discussion of how the careful implementation of quasiexperimental designs allows for treatment-related causal interpretations to be made is also included. Finally, it is emphasized that practical application of these data make it possible to improve the implementation of the treatment in communities receiving it at a later date.     

Child deaths attributed to abuse and neglect: An empirical study

This study of 267 child deaths associated with abuse or neglect in Texas during 1975 through 1977 suggests a number of indicators for identifying potential child fatalities. Families where abuse or neglect is implicated in a child fatality are characterized by small family size, young parents, and under-utilization of community support services. Over three-fourths of the families in the study had never come to the attention of the state's child protective services agency. In addition, when fathers were present in the home, they were as likely to be involved in the abuse or neglect as mothers. Neglect was implicated in the death of a child as often as abuse. The study suggests the need for further examination of child fatality profiles associated with abuse and neglect and increased community outreach efforts to provide support to high-risk families, The study also has implications for social policy decisions, particularly in relation to efforts to resist attempts to exclude neglect from child protective service responsibility.     

Dealing with unknown variables in policy/program evaluation

A leading objection to the feasibility of applying benefit-cost analysis in evaluation is that one often cannot know the benefits, the costs, or the probabilities of their occurring for a given proposal. Threshold analysis involves handling unknown variables by converting relevant evaluation problems into questions as to whether a given benefit, cost, probability, or combination of these elements is more or less than a threshold. Above the threshold, the proposed program would be relatively profitable; and below the threshold, it would be relatively unprofitable. Applying that analysis may require the drawing of simple graphs to stimulate the minds of knowledgeable persons as to the range of the actual benefits, costs or probabilities. The approach can be applied when the benefits or the costs are either monetary or nonmonetary, and when the problems involve either go/no-go decisions or conflicting-choice decisions.