The use of metaevaluation to plan evaluations of conservation programs

The U.S. Department of Energy (DOE) is sponsoring and will be required to evaluate a wide range of conservation programs. The development of evaluation plans for such programs can often be improved by reviewing evaluations that have been done on similar programs. The Residential Conservation Service (RCS), sponsored by DOE, is discussed as an example of how conducting metaevaluations can improve evaluation planning. Many utilities have sponsored programs that are similar to the RCS. A review of the utility program evaluations suggested key methodological issues and data problems that should be considered in the development of an RCS evaluation plan. The implications of these metaevaluation findings for an RCS evaluation plan are discussed.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Evaluability assessment of a community development leisure program in Spain

This paper presents the results of an evaluability assessment (EA) conducted in Spain. EAs are assessments that minimize the uncertainty and risk inherent in a subsequent evaluation (e.g. an impact or results assessment) that may require excessive resources. We designed our own EA model by combining three theoretical approaches, with a holistic and pedagogic vision of the process and a flexible methodology to enable adaptation to different programs and contexts, thus fostering transferability. The EA was carried using a qualitative methodology: documentary analysis, seven semi-structured interviews with stakeholders, eight direct observation sessions and a check-list for politicians and experts. The outcome report helped promote organizational learning, provided the program with a logical and assessable theoretical model, detected its strengths and helped overcome the barriers to efficient assessment. Based on these achievements, an evaluation of the results was recommended, with repeated pretest-posttest measurements and an equivalent control group.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

Monitoring and evaluating transition and sustainability of donor-funded programs: Reflections on the Avahan experience

PurposeIn low and middle-income countries, programs funded and implemented by international donors frequently transition to local funding and management, yet such processes are rarely evaluated. We reflect upon experience evaluating the transition of a large scale HIV/AIDS prevention program in India, known as Avahan, in order to draw lessons about transition evaluation approaches and implementation challenges.ResultsIn terms of conceptualizing the transition theory, the evaluation team identified tensions between the idea of institutionalizing key features of the Avahan program, and ensuring program flexibility to promote sustainability. The transition was planned in three rounds allowing for adaptations to transition intervention and program design during the transition period. The assessment team found it important to track these changes in order to understand which strategies and contextual features supported transition. A mixed methods evaluation was employed, combining semi-structured surveys of transitioning entities (conducted pre and post transition), with longitudinal case studies. Qualitative data helped explain quantitative findings. Measures of transition readiness appeared robust, but we were uncertain of the robustness of institutionalization measures. Finally, challenges to the implementation of such an evaluation are discussed.ConclusionsGiven the scarceness of transition evaluations, the lessons from this evaluation may have widespread relevance.     

A response to the BPA evaluation plan: Some modest but utilitarian considerations

In response to the complex and expensive Bonneville Power Administration (BPA) evaluation plan, this article attempts to identify some pragmatic issues and alternative approaches to conservation program evaluation. In particular, the need for simpler, lower-cost evaluation options and the importance of process evaluation to program management are emphasised. This is followed by the delineation of several key principles to be observed in conducting a more modest, but still methodologically sound and managerially useful, evaluation of energy conservation programs.     

Evaluating utility conservation programs: The BPA plan

This paper summarizes a detailed plan for evaluation of the residential energy conservation programs offered by the Bonneville Power Administration (BPA). The major purpose of the evaluation is to determine the energy savings attributable to the programs and to identify differences between program participants and nonparticipants. The focus of this paper is on the sampling issues associated with reliable estimates of energy savings. The methodology attempts to correct for the self-selection bias likely to occur in conservation program evaluation by using intentional nonrandom sampling. In addition, the paper suggests a variety of analytical approaches to analyzing the data with respect to energy savings.The proposed design, using four different groups to estimate program effects, is a comprehensive endeavor. This is appropriate for BPA because of the substantial investment it is making in conservation. However, the design is flexible and allows for use offewer groups to reduce evaluation costs.     

A qualitative exploratory study of substance abuse prevention outcomes in a heterogeneous prevention system

The prevention of abuse of alcohol and other drugs is a concern for parents, policy-makers, educators, and social service professionals. Prevention programs are sponsored by many different types of social and educational agencies using a variety of intervention strategies. This article reports a study of a sample of such programs in the state of Nevada. The overall prevention system in the state espouses a "risk and resiliency" approach to prevention. Focus group methodology was used to study perception of outcomes of these programs from the viewpoints of various program stakeholders (youth participating in the programs, parents of participants and program staff). Analysis of the qualitative data yielded findings about potential outcomes as well as implicit program theories. Implications for future planning efforts as well as further evaluation efforts are discussed.     

Community-Based Program Research

ABSTRACT

At first glance, the worlds of program professionals and evaluators may seem quite separate. However, there are common issues whose resolution will enhance both program development and research on programs. Program professionals and evaluators have a great deal to learn from each other. As that learning occurs, both program delivery and evaluation research will benefit. Both are concerned about matters of validity, whether it pertains to the nature of the program intervention itself or to the nature of the assessment of the program intervention. To reinforce the development of program evaluation in partnership, this paper discusses key points about evaluation research relevant to both program professionals and researchers. These include the contextual influences on a program, the “readiness” of a program for evaluation, and whether the evaluation research “works” for the program. Underlying our discussion is a call for the development of partnerships around research on programs. In that light, this article elaborates the process steps that should be taken to build program evaluation partnerships, including a discussion of what evaluators and program professionals need to know about each other, and their respective values, interests, and professional perspectives.     

An evaluability assessment of a West Africa based Non-Governmental Organization's (NGO) progressive evaluation strategy

While program evaluations are increasingly valued by international organizations to inform practices and public policies, actual evaluation use (EU) in such contexts is inconsistent. Moreover, empirical literature on EU in the context of humanitarian Non-Governmental Organizations (NGOs) is very limited. The current article focuses on the evaluability assessment (EA) of a West-Africa based humanitarian NGO's progressive evaluation strategy. Since 2007, the NGO has established an evaluation strategy to inform its maternal and child health care user-fee exemption intervention. Using Wholey's (2004) framework, the current EA enabled us to clarify with the NGO's evaluation partners the intent of their evaluation strategy and to design its program logic model. The EA ascertained the plausibility of the evaluation strategy's objectives, the accessibility of relevant data, and the utility for intended users of evaluating both the evaluation strategy and the conditions that foster EU. Hence, key evaluability conditions for an EU study were assured. This article provides an example of EA procedures when such guidance is scant in the literature. It also offers an opportunity to analyze critically the use of EAs in the context of a humanitarian NGO's collaboration with evaluators and political actors.     

Organizational structure,leadership and readiness for change and the implementation of organizational cultural competence in addiction health services

Increasing representation of racial and ethnic minorities in the health care system and on-going concerns about existing health disparities have pressured addiction health services programs to enhance their cultural competence. This study examines the extent to which organizational factors, such as structure, leadership and readiness for change contribute to the implementation of community, policy and staffing domains representing organizational cultural competence. Analysis of a randomly selected sample of 122 organizations located in primarily Latino and African American communities showed that programs with public funding and Medicaid reimbursement were positively associated with implementing policies and procedures, while leadership was associated with staff having greater knowledge of minority communities and developing a diverse workforce. Moreover, program climate was positively associated with staff knowledge of communities and having supportive policies and procedures, while programs with graduate staff and parent organizations were negatively associated with knowledge of and involvement in these communities. By investing in funding, leadership skills and a strategic climate, addiction health services programs may develop greater understanding and responsiveness of the service needs of minority communities. Implications for future research and program planning in an era of health care reform in the United States are discussed.     

The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

Evaluability assessment: Reflections on the process

As a formal process, evaluability assessment is 10 years old. In the Cooperative Extension Service (CES), the process is 5 years old. Since 1984, evaluability assessments have been implemented in five states with six different major programs in an effort to define the process in a practical methodological sense and to encourage its adoption in CES. Implementation proceeded iteratively and after each study, procedures were analyzed, clarified, and, wherever possible, simplified to make the process more usable and more “operator robust.” The purpose of this article is to reflect on the process as it evolved in applications in CES. Underlying philosophy and methodological considerations are discussed. To put these in perspective, background information is provided on the evaluability assessment process and on the studies conducted in CES.     

Scaling up a community-led health promotion initiative: Lessons learned and promising practices from the Healthy Weights for Children Project

The increase in overweight and obesity among children has emerged as an important public health issue. This trend has highlighted the need for accessible and novel approaches to support healthy weights for children and their families to prevent childhood obesity. The purpose of this article is to describe the iterative development and scale-up of a community-led, national-level project to promote healthy weights among Canadian children and families who may be experiencing vulnerabilities. In this project, the Healthy Together program was designed to engage families in an interactive program to support healthy lifestyles. The program also provides a platform for creating supportive environments for healthful lifestyles through practice and policy change. Based on a process evaluation, we describe the iterative development of Healthy Together from Phase 1 through 3 to shed light on processes shaping implementation and scale-up of the program. Lessons learned during each phase were used to refine the program and further expansion. Indicators of successful scale-up include the Healthy Together program’s cross-jurisdictional reach and promising evaluation results in real-world conditions. The practice-based program scaling approach provides practical guidance for planning and implementing similar health promotion programs in diverse communities.     

Acceptability of a stage-matched expert system intervention for domestic violence offenders

Most interventions for men who batter are standardized and "one-size-fits-all," neglecting individual differences in readiness to change. A multimedia expert system intervention based on the transtheoretical model (the "stage model") was developed as an adjunct to traditional court-mandated programs. The expert system assesses stage of change, decisional balance, self-efficacy, and processes of change and provides immediate individualized stage-matched feedback designed to increase readiness to end the violence. Fifty-eight male batterer intervention program clients were invited by agency staff to complete an expert system session and an evaluation of the program; 33 men were recruited at program intake and the remainder from ongoing groups. Responses to the intervention were very positive. For example, 87% of participants reported that they found the program to be easy to use, and 98% said it could probably or definitely help them change their attitudes or behaviors. Findings provide encouraging evidence of the acceptability of this stage-matched approach to intervention for domestic violence offenders.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.     

Evaluability assessment of the adult education program (AEP): The results and their use

This paper describes the conduct and the results of an evaluability assessment (EA) of the Adult Education Program (AEP), PL 91-230, as amended (20 U.S.C. 1201–1211c). It presents the findings on the intended program, the actual program, the plausibility of the objectives, the measurement of the objectives, and the program evaluability and options for improvement. The paper concludes with a discussion of the management use of the results following the study and within the current context for the program.