Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Pacific Northwest utilities have sponsored the nation's earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility-sponsored program evaluations. This article critically reviews the methods and findings of four major Northwest utility program evaluations. Recommendations for future evaluation and program management efforts also are discussed.     

Evaluating utility conservation programs: The BPA plan

This paper summarizes a detailed plan for evaluation of the residential energy conservation programs offered by the Bonneville Power Administration (BPA). The major purpose of the evaluation is to determine the energy savings attributable to the programs and to identify differences between program participants and nonparticipants. The focus of this paper is on the sampling issues associated with reliable estimates of energy savings. The methodology attempts to correct for the self-selection bias likely to occur in conservation program evaluation by using intentional nonrandom sampling. In addition, the paper suggests a variety of analytical approaches to analyzing the data with respect to energy savings.The proposed design, using four different groups to estimate program effects, is a comprehensive endeavor. This is appropriate for BPA because of the substantial investment it is making in conservation. However, the design is flexible and allows for use offewer groups to reduce evaluation costs.     

The role of financial incentives in utility-sponsored residential conservation programs: A review of customer surveys

A number of utility programs are encouraging residential customers to invest in energy-efficient equipment by providing financial incentives for these actions. Subsidized loans are the most common type of incentive offered by utility programs, although discounts, rebates, lower rates, and free materials or labor also have been provided.Given the large sums involved in utility loan programs, an understanding of the impact of these financial incentives on retrofit investment decisions has a high potential policy value. The purpose of this paper is to review the available evidence on this issue and to suggest ways that additional evidence can be obtained.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

A response to the BPA evaluation plan: Some modest but utilitarian considerations

In response to the complex and expensive Bonneville Power Administration (BPA) evaluation plan, this article attempts to identify some pragmatic issues and alternative approaches to conservation program evaluation. In particular, the need for simpler, lower-cost evaluation options and the importance of process evaluation to program management are emphasised. This is followed by the delineation of several key principles to be observed in conducting a more modest, but still methodologically sound and managerially useful, evaluation of energy conservation programs.     

The use of the psychosocial maturity inventory in evaluations of youth conservation corps programs

Although work experience has been widely recommended as beneficial to adolescent development, there is little empirical support for this recommendation. The Youth Conservation Corps (YCC), combining as it does some of the best features of both paid work and unpaid community service, offers an opportunity to explore the developmental impact of work experience. However, the need to pay as much attention to the problems of measurement as to the programs themselves dictates simultaneous evaluation of both programs and measures.     

Whose rights are they anyway?: The supreme court and the rights of Juveniles

This article examines selected United States Supreme Court decisions from 1923 through the 1984 term of the Court. The selection, which includes Wisconsin v. Yoder, In re Gault, In re Winship, and Schall v. Martin, traces and illustrates the triangle of interest among parent, child, and state as parens patriae, when the issue raised is the interest of children. The focal question is: Have children been recognized as having their own individual Constitutional rights, or have their rights been merged with those of their parents and/or the state standing in the place of the parents?     

An industrial application of meta-evaluation

This article describes the efforts of an industrial training organization to increase the number and to improve the quality of “follow-up” evaluations (i.e., evaluations in terms of post-training job performance). Efforts included an analysis of the problems encountered in evaluating training, development and implementation of corrective actions, and an evaluation of the corrective actions. The project illustrates the process of “meta-evaluations.”     

Applicability of standards for evaluations of educational programs, projects and materials in an international setting

This paper starts with a short discussion of the book Standards for Evaluations of Educational Programs, Projects and Materials by the Joint Committee on Standards for Educational Evaluation (McGraw-Hill, 1981). It then goes on to examine the usefulness in countries other than the United States of these standards and argues that evaluation of “programs” is more complicated and more politically biased than evaluation of “projects” and “materials.” In Europe, evaluation of educational programs (school systems) has proved to be highly correlated with the overriding objectives of different political parties, e.g., conservative, liberal, socialist, whereas educational policy in the United States is less bound to political parties, varying rather from case to case. In order to show how generally agreed upon educational goals can be interpreted according to different values, the author has reanalysed some studies of mean achievement in reading comprehension and science for 10- and 14-year-old students in 15 countries (IEA Project, 1973). The rank order of these countries is changed if, instead of national mean achievement, one takes proportion of students under a minimum level or above a prescribed top level as the criterion. Also the national (program) standard is evaluated in terms of between-school variance in results and in proportion of single-school units with exceptionally low or exceptionally high mean results. The conclusion is that the book on standards for evaluations of educational programs provides a good checklist of prerequisites for a reliable and valid evaluation. However, it is also concluded that—due to differences in values of program outcomes—such standards do not guarantee that the result of the evaluation will be indisputable.     

Ecologically defining the mistreatment of adolescents

Ecologically defined, the problem of adolescent/family conflict for otherwise “normal” families is a reaction to crisis in the life cycle of the family. A family's success in weathering the crisis is indicated by their successful maturation to the next stage of the cycle; we can assume that in such cases the ratio of stress to coping ability was such that the family could integrate that stress. When conflict escalates to abuse and neglect, however, the indication is that stress has outstripped coping ability, and the implication is for stress-reducing intervention quite unlike the standard existing interventions in either the juvenile justice or child welfare systems, which negatively label family members, make them adversaries, fragment their coping capabilities, and even subject them at times to situations worse than those for which they needed help originally. The interventions called for are those that support beleaguered families; that are based on conceptualizations of people as adaptable and capable of solving their own problems if they are given adequate supports.     

Applicability of standards for evaluations of educational programs, projects and materials in an international setting: Qualitative research

Qualitative studies are much more the practice in Europe and increasingly elsewhere than in the United States. For an international audience, therefore, there needs to be much greater emphasis on qualitative techniques than there is in the standards report. The most difficult ethical problem arises from the conflict between the right to privacy of research subjects and the right to know of audiences. By its nature, qualitative research focuses on the particular, the detailed analysis of the specific incident. It is therefore often difficult to conceal the identity of the individual organisations and people concerned. Even a confidential report might work to the detriment of the subjects studied. Published reports pose the same problem. Have subjects the right to have material they believe to their disadvantage excised from a report even if that renders the report valueless? There are two technical points in the standards document that should be amplified. The first concerns the verification of data. We need to specify procedures to ensure credibility for qualitative studies. The second concern is the specification of quantitative techniques for the analysis of qualitative data. The statement in the standards document is too weak. We need to define much more clearly the steps that should be taken to ensure that qualitative data can be tested rigorously. Finally, there is the problem of collusion, the danger that the qualitative evaluators will select from their material the data which are in conformity with their own predilictions or presuppositions, the expectations of their audience, or the requirements of their sponsors. This problem may be acute in some parts of the world. To be of maximum international value, the standards report needs to be extended and to pay much greater attention to problems arising with qualitative evaluation.     

A systematic approach to increasing use of management-oriented program evaluation data

With the implementation of Management Information Systems (MISs) in human service delivery settings, came the expectation that managers would make data-based decisions. There is substantial evidence, however, that this has not occurred. The present authors describe a systematic approach to correcting this deficiency. The approach involves a behavioral definition of management information use, the identification of disruptions in the behaviors comprising information use, and an intervention designed to reduce disruptions and promote use. Conclusions about the effectiveness of this approach gained from its implementation in a community mental health center are described.     

Pre- and post-planned evaluations are complements not alternatives: A reply to Strasser and Deniston

Strasser and Deniston's own analysis shows that post-planned evaluations are unsuitable substitutes for pre-planned evaluations. When viewed as post-experimental interviews, however, post-planned evaluations can produce valuable information which complements traditional experimental and quasi-experimental evaluations.     

Differential validity: Another threat to compensatory education evaluations

The effects of the changing nature of intelligence test items during the follow-up period for compensatory education program evaluations on the judgements about the efficacy of such programs are examined. Data generated from Stanford-Binet test items, assuming known “true” underlying motor and verbal scores, as well as data from three compensatory education evaluations, are examined. The changing nature of the IQ test items, i.e., the differential validity of the tests, is put forth as an explanation for the often disheartening results of such program evaluations. Methods for dealing with the problem of differential validity are suggested.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

Evaluating a private foundation's health program

Public policy decisions in health are increasingly difficult and expensive. Although there will never be enough information available, private foundations can help to bridge the most important gaps in knowledge. Larger foundations may also wish to respond to those who doubt the value of foundation activities. This article reviews the experiences over the past eight years of The Robert Wood Johnson Foundation in employing evaluation and related social research procedures in the planning and implementation of a major philanthropic effort to improve the health and medical care of Americans. Discussed are the still evolving Foundation evaluation framework, the unanticipated problems in undertaking specific evaluations, and the substantive findings of some of the studies.     

The impact of general accounting office program evaluations on government

General Accounting Office (GAO) audit reports have an impact on government operations by encouraging growth in the size and scope of the state and by encouraging the Congress and executive agencies to attend to the auditor's dominant values of economy, efficiency, and effectiveness in government. Through its financial, management, and program audits, the GAO program evaluators seek to rationalize agency management and programs so that more resources are required. The GAO's program evaluation efforts also serve Congress by providing for oversight of public policy, a bludgeon to coax agencies into pursuing activities preferred by congressmen, and a tool for electoral activities. In the end, the evaluation auditors are doomed to frustration in achieving their ideals for effective democratic government. This is so because “practical business policy” is irrelevant to government operations and inconsistent with the core values of a democratic state.     

Ethical dilemmas in evaluating programs with family court related clients

Two ethical dilemmas are discussed; breaching client confidentiality in order to intervene in the client's illegal behavior, and defining the role of evaluator as expert witness. The evaluation context concerns a combined program development and evaluation project with clients involved in family court proceedings for child abuse or neglect. Different perspectives, approaches, and experiences in resolving each dilemma are reviewed. The conclusion argues that approaches to ethical dilemmas in evaluation projects must go beyond techniques to disguise data, to include an assessment of the values and value commitments connected to the different roles an evaluator must perform.