The role of financial incentives in utility-sponsored residential conservation programs: A review of customer surveys

A number of utility programs are encouraging residential customers to invest in energy-efficient equipment by providing financial incentives for these actions. Subsidized loans are the most common type of incentive offered by utility programs, although discounts, rebates, lower rates, and free materials or labor also have been provided.Given the large sums involved in utility loan programs, an understanding of the impact of these financial incentives on retrofit investment decisions has a high potential policy value. The purpose of this paper is to review the available evidence on this issue and to suggest ways that additional evidence can be obtained.     

The use of metaevaluation to plan evaluations of conservation programs

The U.S. Department of Energy (DOE) is sponsoring and will be required to evaluate a wide range of conservation programs. The development of evaluation plans for such programs can often be improved by reviewing evaluations that have been done on similar programs. The Residential Conservation Service (RCS), sponsored by DOE, is discussed as an example of how conducting metaevaluations can improve evaluation planning. Many utilities have sponsored programs that are similar to the RCS. A review of the utility program evaluations suggested key methodological issues and data problems that should be considered in the development of an RCS evaluation plan. The implications of these metaevaluation findings for an RCS evaluation plan are discussed.     

Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Pacific Northwest utilities have sponsored the nation's earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility-sponsored program evaluations. This article critically reviews the methods and findings of four major Northwest utility program evaluations. Recommendations for future evaluation and program management efforts also are discussed.     

A response to the BPA evaluation plan: Some modest but utilitarian considerations

In response to the complex and expensive Bonneville Power Administration (BPA) evaluation plan, this article attempts to identify some pragmatic issues and alternative approaches to conservation program evaluation. In particular, the need for simpler, lower-cost evaluation options and the importance of process evaluation to program management are emphasised. This is followed by the delineation of several key principles to be observed in conducting a more modest, but still methodologically sound and managerially useful, evaluation of energy conservation programs.     

The use of participant observation in evaluation research

In the last several years considerable interest has developed in the use of qualitative methods in evaluation research. Recent works indicate that the tradition of qualitative research has much that will be useful to evaluation researchers. This paper does not seek to cover the range of qualitative methods available to the evaluation researcher but rather is concerned with one methodology in that tradition, participant observation. The paper describes the methodology of participant observation for those who are interested in its characteristics and for those who would like to consider it for evaluation or other types of research. Examples are given of how it has been used and references to additional sources of information are included as well. The paper is organized into three sections: characteristics of participant observation, when it is appropriate to use it, and specific techniques of the method.     

The effects of psychiatric symptoms on quality of life assessments among the chronic mentally ill

This study of chronically mentally disabled persons in community residences examined the discriminant validity of subjective quality of life indicators and self-report mental health indices to determine the potential confounding effects of psychopathology on the assessment of quality of life (QOL). Factor analyses and difference-score reliabilities identified a general QOL construct and a general mental health construct with 27% common variance. However, psychiatric symptoms did not significantly (p less than .05) alter the bivariate and multivariate relationships among the QOL ratings, except in the health domain in which the correlations of health-related QOL indicators with global QOL were significantly (p less than .05) attenuated after removing the effects of psychopathology. The results suggest that psychopathology does not introduce bias into the overall structure of QOL data, but they also indicate the importance of controlling for mental health effects in the assessment of patients' self-rated health and satisfaction with health care.     

The use of the psychosocial maturity inventory in evaluations of youth conservation corps programs

Although work experience has been widely recommended as beneficial to adolescent development, there is little empirical support for this recommendation. The Youth Conservation Corps (YCC), combining as it does some of the best features of both paid work and unpaid community service, offers an opportunity to explore the developmental impact of work experience. However, the need to pay as much attention to the problems of measurement as to the programs themselves dictates simultaneous evaluation of both programs and measures.     

Evaluating and improving prevention programs: a strategy from drug abuse.

This paper describes a methodology used to overcome the difficulties in evaluating and improving prevention programs. The methodology was applied in this case to a drug abuse prevention program. The methodology includes using health-oriented rather than pathology-oriented outcome measures, random assignment to groups, multiple settings, multiple outcome measures, goal-free evaluation, and the recording of process variables. The first year results showed program effectiveness in one setting and not in the other. Process analyses pointed toward improvements that could be made. Those improvements were implemented and led to greater program effectiveness in the second year.     

Ethnic and class factors affecting mental health clinic service

Service delivery by a mental health clinic to clients from an older suburban catchment area containing both lower and middle class blacks and whites was studied. The epidemiologic method was employed, relating service delivery to the characteristics of the various population groups within the catchment. The findings indicate that low service rates for blacks were not related to ethnicity alone, but to the combination of black ethnicity and low social status; however, middle class blacks and middle class whites who entered treatment received similar service. Poor blacks had higher admission rates than poor whites with similar need indicators. In continued treatment, poor blacks received lowest service rates while minority status poor whites living in a predominantly black area received highest service rates. Social class mediated black service utilization. Severity of illness, interacting with social class and minority status mediated white service utilization. Outreach is suggested not only for poor blacks but also for poor whites, especially those living in areas in which they represent an ethnic minority.     

The subjective well-being of mental patients in nursing homes.

This article reports a survey of self-reports of well-being by chronic mental patients in nursing homes. Patients reported lower levels of well-being than the general population, but not lower than other socially disadvantaged groups including urban renewal blacks. Nursing home mental patients reported levels of well-being that were generally similar to mental patients in other settings, including day hospital patients, participants in an innovative community care program, and patients receiving traditional hospital and follow-up care. These reports were mostly unrelated to levels of symptomatology or social integration, but were strongly related to patient perceptions of the quality of the environment. Results have important implications concerning the justifications for community care goals like noninstitutional care, reduction of symptomatology, and the fostering of social integration.     

Evaluating multicultural curricula in the community college: A sociocultural framework

This paper addresses the need for models for assessing multicultural programs in the community college, the most culturally and socioeconomically diverse educational institutions in the country. A three-dimensional framework presents faculty, student, and curriculum variables critical to the implementation and outcomes of multicultural programs. The framework emerged from the formative evaluation of a new interdisciplinary social science curriculum and guided the design of the national field test of that curriculum in 30 community college classrooms. Three kinds of results are reported: implementation patterns; appropriateness to faculty members' teaching goals; and impact on reading behavior, interest, overall learning, and political efficacy of students with diverse ages, ethnic and sociocultural backgrounds, and political positions. Political efficacy gains of older students and students with lower socioeconomic backgrounds are discussed. The importance of such a framework in documenting the interaction between a curriculum and its sociocultural context is stressed.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

The efficacy of early intervention programs for children with organic impairment: A quantitative review

The results of studies examining the effectiveness of early intervention for infants and children with organic impairment and developmental delay were reviewed using recently developed quantitative methods that treat the literature review process as a unique type of scientific inquiry. Thirty-eight studies meeting certain predetermined criteria were included in the review. The 38 studies contained a total of 118 statistical hypothesis tests that evaluated the effectiveness of early intervention. An analysis of these tests based on the calculation of effect sizes revealed that subjects receiving early intervention performed better on a wide range of dependent measures than subjects not receiving intervention. The outcomes were found to be related to several design and study characteristics. Larger effect sizes were associated with preexperimental designs, and also with studies in which the internal validity was rated as poor. Several other design variables such as how subjects were assigned to conditions and how the dependent measure was recorded were related to study outcome as measured by effect size. The conclusion was made that an accurate interpretation of the early intervention research literature cannot be made without consideration of specific design variables and study characteristics.     

Youth service caseworkers: Social and legal factors affecting their recommendation to place youths in secure care facilities

This study examines the effects of social and legal variables on youth service caseworkers' recommendation to place youths in a secure care setting. While the findings of this study indicate that caseworkers' recommendation to place youths in such a facility is primarily influenced by their juvenile delinquency or legal background characteristics, the social variable “family composition” has a significant impact on the recommendation. This suggests that caseworkers view single-parent families as less capable than two-parent families of managing youths' serious delinquency problems and, therefore, are more likely to recommend youths from single-parent families for secure care placement consideration. The fact that legal variables are the strongest predictors of secure care decision outcomes indicates that objective criteria are being used to determine which youths to consider for secure care placement. Also, it is found that the more offenses that are committed the more likely that one will be a violent offense. This finding implies that the issue of serious juvenile delinquency should be examined and dealt with from a broad juvenile justice perspective directed at prevention as opposed to a limited focus on only the secure care process.     

The Tellado case study exercise: A metacritique

This paper provides an overview and critique of the papers in the Tellado Case Study Exercise. Two main themes are found in the critiques: the belief that Tellado lacked a clear audience for his evaluation and that he lacked an appropriate change orientation that would insure utilization. The sources of these criticisms are discussed from the perspective of evaluation as a linkage activity. The use of this perspective leads to the recognition that all parties to the case study may have focused on the wrong stakeholders in the utilization process. Thus, a major problem with utilization-focused and stakeholder based approaches to evaluation is revealed. Possible ways to avoid this problem are discussed. The paper concludes with a request for additional case studies to be contributed to the journal.     

The ers standards for program evaluation: Guidance for a fledging profession

THE ERS Standards, together with those of the Joint Committee, symbolize a commitment to the professionalization of evaluation. The two sets of standards, however, differ on at least five important dimensions: the perceived linearity of evaluation activities; preferred and permitted approaches (with one set heavily favoring quantitative/experimental approaches over any others); the purposes of evaluation; obligations to clients and stakeholders; and whether or not social science can be value-free (objective). Because these fundamental assumptions are so divergent, the possibility of combining or effecting a compromise between the two sets appears unlikely at this time.