Evaluations of utility residential conservation programs in the Pacific Northwest: A critical review

Pacific Northwest utilities have sponsored the nation's earliest and most thorough residential conservation programs. The Northwest region also leads in the quality, quantity, and usefulness of utility-sponsored program evaluations. This article critically reviews the methods and findings of four major Northwest utility program evaluations. Recommendations for future evaluation and program management efforts also are discussed.     

Evaluating utility conservation programs: The BPA plan

This paper summarizes a detailed plan for evaluation of the residential energy conservation programs offered by the Bonneville Power Administration (BPA). The major purpose of the evaluation is to determine the energy savings attributable to the programs and to identify differences between program participants and nonparticipants. The focus of this paper is on the sampling issues associated with reliable estimates of energy savings. The methodology attempts to correct for the self-selection bias likely to occur in conservation program evaluation by using intentional nonrandom sampling. In addition, the paper suggests a variety of analytical approaches to analyzing the data with respect to energy savings.The proposed design, using four different groups to estimate program effects, is a comprehensive endeavor. This is appropriate for BPA because of the substantial investment it is making in conservation. However, the design is flexible and allows for use offewer groups to reduce evaluation costs.     

The role of financial incentives in utility-sponsored residential conservation programs: A review of customer surveys

A number of utility programs are encouraging residential customers to invest in energy-efficient equipment by providing financial incentives for these actions. Subsidized loans are the most common type of incentive offered by utility programs, although discounts, rebates, lower rates, and free materials or labor also have been provided.Given the large sums involved in utility loan programs, an understanding of the impact of these financial incentives on retrofit investment decisions has a high potential policy value. The purpose of this paper is to review the available evidence on this issue and to suggest ways that additional evidence can be obtained.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

Data and the measurement of R&D program impacts

The purpose of this paper is to propose a research agenda for the measurement of economic impacts of Canadian government R&D support programs. Different methodologies and indicators used to assess benefits from government support programs/agencies for R&D are discussed first. Using available information on major business-related R&D federal programs, the paper will assess which indicators and methodologies can be implemented. The specific programs/agencies under investigation include: Technology Partnerships Canada sponsored by Industry Canada, Industrial Research Assistance Program sponsored by National Research Council, Atlantic Innovation Fund sponsored by Atlantic Canada Opportunities Agency, Canadian Space Agency and National Defence.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Program evaluation of services for the homeless : Challenges and strategies

The rapid growth of the homeless population in North America has given rise to a sharp increase in the number of task forces and studies aimed at setting up either programs for the homeless, or policies for controlling the phenomenon. Whenever there is a need for program evaluations in this process, numerous methodological and strategic challenges have to be faced. Due to the characteristics of the clientele, evaluation studies of services for the homeless call for innovative approaches. This paper reviews the research strategies (types of evaluations, designs, and indicators) that have been developed to assess programs for chronic inebriates and mentally ill homeless. The findings from previous evaluations are then summarized and their consequences for evaluation practice are considered.     

Program decisions : Evaluating a bird in the hand versus two in the bush

In new programs that have not reached full effectiveness, or in programs that are designed to produce future benefits (i.e., become “institutionalized”), evaluations that fail properly to assess future performance may bias decisions against the programs. For such programs, consideration of future benefits may significantly enhance the assessment of their cost-effectiveness. This paper describes an evaluation methodology based on decision theory that can take future benefits into account in evaluating a program. The methodology was applied to the Metropolitan Atlanta Crime Commission (MACC) Community Anti-Crime Project and indicated the potential for substantial future benefits subsequent to cessation of federal funding. The implications of use of this methodology and the MACC results for the overall Community Anti-Crime Program are also discussed.     

Paving the Way for EAP Evaluations:

The theme of this chapter centers on the development of key process questions that precede and direct the development of EAP evaluations. The main assumptions of this chapter are: (1) that nonthreatening process-type questions may be used in helping EAP evaluators plan their evaluations; and (2) that the profession of social work can meaningfully assist in developing and answering such questions. These process questions emerge from considcring issues related to: (1) policies and procedures; (2) services; And (3) the administration of EAP programs. The role of the social worker in this process is described, and the unique knowledge and skill of the social work profession in this particular role is explored and identified.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Resolving conflicting demands on evaluation: A test demonstration of videotex for farmers

Evaluating an innovation for federal, state, and local policymakers and program managers alike entails conflicting demands on the evaluation study. Policymakers at federal, state, and local levels are best assisted by impact evaluations, whereas state and local program managers are best assisted by process evaluations. In-house evaluators often have an advantage in conducting process evaluations; external evaluators generally have an advantage in conducting impact evaluations. A cost-effective approach may be to combine in-house process evaluation and external impact evaluation. This dual approach was found to reduce conflicting demands on the evaluation of an experimental videotex system for agricultural producers.     

State-level evaluations of school programs funded under the Drug-Free Schools and Communities Act.

Although the Drug-Free Schools and Communities Act of 1986 and the 1989 Amendments to the Act require states to evaluate their drug-education programs, no guidelines for conducting these evaluations have been produced, and little has been reported on how the states are conducting such evaluations. In this article, the results of a telephone survey on current state-level efforts to evaluate school programs funded under the Act are reported. Some states report studies of the implementation of the program and some report drug- and alcohol-use surveys. Together, these two types of evaluation efforts form the foundation of an approach for conducting evaluations under the Act. Reasons are presented why experimental and quasi-experimental designs might be inappropriate and impractical for the evaluations, and an evaluation approach linking program implementation findings and drug- and alcohol-use survey results is suggested.     

Building evaluation capacity: definitional and practical implications from an Australian case study

Internationally, evaluation capacity-building activities have mushroomed as demands have increased for government-funded programs to demonstrate that they are effective and efficient. Despite this, there is a lack of clarity in the way in which evaluation capacity-building is defined and conceptualized. The current paper presents a case study of a national evaluation capacity building exercise that we are in the midst of conducting in Australia, and discusses the findings in relation to definitional, conceptual and practical issues. Specifically, we describe an evaluation capacity building exercise involving over 100 mental health projects, detailing the methods that we employ, some of the challenges that we have faced, and the benefits we feel we are achieving. Our key message is that definitions of evaluation capacity-building should not only make reference to equipping organizations to routinely conduct evaluations, but should also stress the varied uses to which evaluation findings can be put. In addition, such definitions should acknowledge some of the valuable by-products of evaluation capacity building activities, such as the development of shared understandings of the program or project being evaluated.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

The power of the context map: Designing realistic outcome evaluation strategies and other unanticipated benefits

Developing a feasible evaluation plan is challenging when multiple activities, often sponsored by multiple agencies, work together toward a common goal. Often, resources are limited and not every agency's interest can be represented in the final evaluation plan. The article illustrates how the Antecedent Target Measurement (ATM) approach to logic modeling was adapted to meet this challenge. The key adaptation is the context map generated in the first step of the ATM approach. The context map makes visually explicit many of the underlying conditions contributing to a problem as possible. The article also shares how a prioritization matrix can assist the evaluator in filtering through the context map to prioritize the outcomes to be included in the final evaluation plan as well as creating realistic outcomes. This transparent prioritization process can be especially helpful in managing evaluation expectations of multiple agencies with competing interests. Additional strategic planning benefits of the context map include pinpointing redundancies caused by overlapping collaborative efforts, identifying gaps in coverage, and assisting the coordination of multiple stakeholders.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

Methodological strategies for evaluating youth gang prevention programs

This article reflects on common challenges and lessons learned during the evaluation of gang prevention programs based on case studies of three federally funded Canadian programs. Elements of evaluation design, implementation, data analysis and reporting of results are discussed. More specifically, the article highlights issues that occur when evaluating community projects focused on preventing extreme risks for violence and the complexity of working in potentially dangerous and/or unstable work environments. Topics covered include the problem with quasi-experimental designs, model fidelity adherence, program documentation, client recruitment and retention, and data collection. Recommendations are provided to improve evaluations of youth gang prevention programs and similar community-based interventions that focus on the prevention of youth violence.