Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.     

"Flying by the seat of our pants": current processes to share best practices to deal with elder abuse

We examined current processes used to identify, communicate, and adopt "best practices" in the field of elder abuse using an iterative process involving literature and Internet reviews, surveys completed by stakeholders, and key informant interviews. Results indicate minimal research evidence exists to support current assessment and intervention strategies; there is an immediate need for more evaluation and research in this area and for improved knowledge exchange and translation. Two strategic directions are recommended: to (a) build capacity for research and program evaluation to advance knowledge of effective practices, and (b) build capacity for knowledge exchange to enhance professionals' efforts.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

Dissemination and implementation science in program evaluation: A telemental health clinical consultation case example

Increased attention has been placed on evaluating the extent to which clinical programs that support the behavioral health needs of youth have effective processes and result in improved patient outcomes. Several theoretical frameworks from dissemination and implementation (D&I) science have been put forth to guide the evaluation of behavioral health program implemented in the context of real-world settings. Although a strong rationale for the integration of D&I science in program evaluation exists, few examples exist available to guide the evaluator in integrating D&I science in the planning and execution of evaluation activities.This paper seeks to inform program evaluation efforts by outlining two D&I frameworks and describing their integration in program evaluation design. Specifically, this paper seeks to support evaluation efforts by illustrating the use of these frameworks via a case example of a telemental health consultation program in pediatric primary care designed to improve access to behavioral health care for children and adolescents in rural settings. Lessons learned from this effort, as well as recommendations regarding the future evaluation of programs using D&I science to support behavioral health care in community-based settings are discussed.     

ADOPTING AND TEACHING EVIDENCE-BASED PRACTICE IN MASTER'S-LEVEL SOCIAL WORK PROGRAMS

This article makes specific suggestions for teaching evidence-based practice (EBP) in the master's-in-social-work (MSW) curriculum. The authors use the model of EBP as it was originally conceived: a process for posing empirically answerable questions, finding and evaluating the best available evidence, and applying that evidence in conjunction with client characteristics and practitioner judgment. The authors suggest that EBP, in its original form, is both sufficiently well operationalized and flexible to serve as a primary paradigmatic component of social work education. Furthermore, EBP carries a series of distinct advantages for MSW education that have not been widely recognized to date. These include the provision of a structure for more explicitly recognizing client factors, bridging the micro–macro and researcher–clinician divides, and emphasizing the professionalism of MSW-level practice. Specific curricular components are proposed and discussed.     

Using evaluation to improve program quality based on the BELL model

Building Educated Leaders for Life (BELL) is a national not-for-profit organization whose mission is to increase the educational achievements, self-esteem, and life opportunities of elementary school children living in low-income urban communities. BELL has been engaged in formal evaluation, internally and externally, for more than five years and has built internal evaluation capacity by investing in a specialized full-time evaluation team. As part of a continuous program improvement model of evaluation, BELL uses the data to refine program implementation and replicate successful elements of the services and operations. In this chapter, the authors highlight best practices from the field by outlining BELL's approach to using evaluation data for continuous program improvement. Key strategies include (1) carefully identifying intended users of the evaluation throughout the organization and among its external stakeholders, then working closely with intended users throughout the evaluation process, ensuring full engagement at every step of the process; (2) reporting findings in a readable, user-friendly format and timing the reporting so that it is aligned with programmatic decision making and planning cycles; and (3) making and supporting explicit recommendations for the next program cycle, where intended users have agreed to recommendations and ownership is assigned. BELL's successful use of data for improvement is evidenced by the consistently strong outcomes for the students it serves as well as increased efficiency and satisfaction related to service delivery that has supported the replication of BELL's programs nationally.     

A toolkit for MSDs prevention--WHO and IEA context

Many simple MSD risk management tools have been developed by ergonomists for use by workers and employers with little or no training to undertake injury prevention programs in their workplace. However, currently there is no "toolkit" which places such tools within an holistic, participative ergonomics framework and provides guidance on how best to use individual tools. It is proposed that such an holistic approach should entail initial analysis and evaluation of underlying systems of work and related health and performance indicators, prior to focusing in assessment of MSD risks stemming from particular hazards. Depending on the context, more narrowly focused tools might then be selected to assess risk associated with jobs or tasks identified as problematic. This approach ensures that biomechanical risk factors are considered within a broad context of organizational and psychosocial risk factors. This is consistent with current research evidence on work- related causes of MSDs.     

Planning for permanency planning in foster care

The perceived severity of the problem of “drift” in the foster care system has encouraged agencies nationwide to accept permanency planning, but perhaps with limited attention to the planning process. This article examines the phenomenon of permanency planning from a planning perspective by focusing on: (a) definition of the permanency planning problem, (b) understanding of issues relevant to permanency planning (caseworker, program and systemic), (c) appreciation of the operational context (political, economic and social factors), and (d) evaluation of available options and possible consequences in developing or improving a permanency planning program. Permanency planning is not a simple, ready-made program applicable to any situation; it involves coordinated efforts by workers, service systems, and society in general, and requires time, commitment and resources. Successful planning for permanency planning can permit more children to grow up in homes providing continuity and stability.     

Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Using video vignettes in research and program evaluation for people with intellectual and developmental disabilities: A case study of the Leadership for Empowerment and Abuse Prevention (LEAP) project

People with intellectual and developmental disabilities (IDD) are often excluded from meaningful participation in research and program evaluation for various reasons, including protocols and measures that are inaccessible for people with varying cognitive and communication abilities. Emancipatory research models emphasize the importance of inclusive research practices. Video vignettes are a promising tool for research and program evaluation with people with IDD because they are standardized, they use visual imagery rather than relying on written or verbal communication, and they allow for distance from sensitive topics. The Leadership for Empowerment and Abuse Prevention (LEAP) project used video vignettes to evaluate a healthy relationship program for people with IDD. The authors discuss the process of piloting various protocols and measures, which then ultimately led to the use of video vignettes in the evaluation.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

Mediators and moderators in the evaluation of programs for children. Current practice and agenda for improvement

The author examines the role of mediators and moderators in the evaluation of programs for children. The terms are defined and examples of each are presented. Using bibliometric analysis, the author examines how evaluators use mediators and moderators in treatment studies in education, juvenile justice, health care, child protection, and mental health. The use of mediators and moderators is sporadic and vague at best. An agenda for improvement is outlined that includes greater use of program theory and intensive case studies to find out why researchers in prevention and health promotion incorporate mediators and moderators more effectively in their evaluations.     

Using a community-created multisite evaluation to promote evaluation use across a sector

Multisite evaluation has been cited as a strategy for building knowledge about a program's success and enhancing evaluation capacity and evaluation use. Even so, the extent to which data from these efforts are made available to individual project participants varies. This article describes a community-created multisite evaluation that was designed to overcome this limitation. Based on the negotiated centralized evaluation model and implemented within the context of a community of practice, the EvalFest community-created multisite evaluation has used shared measures to collect data from 30,000 participants from partner sites across the community over three years. This article describes the design characteristics of the community-created multisite approach, and presents results related to partners’ evaluation use that demonstrate the early success of this method. We conclude by considering how this strategy might be applied to other sectors.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Evaluating a management and organization development effort in mental health agencies.

Although the use of management and organization development techniques in health and human service settings is increasing, there are few systematic studies of their effects. Those which do exist are problematic in many respects. An 18-month program incorporating both management and organization development is described along with the results of its evaluation, which were positive and well-utilized. A discussion of the methods employed, the problems encountered, and the outcomes obtained illustrates that these evaluation efforts are needed and can be useful to program evaluators involved in planned-change efforts.     

Prevention efforts underlying decreases in binge drinking at institutions of higher education

Analyses of 94 Fund for the Improvement of Post-Secondary Education (FIPSE)-sponsored drug-prevention programs and their outcomes used the Core Survey to identify 34 institutions where college students' binge drinking increased (M = 5.44%) and 60 institutions where it decreased (M = -4.59%) during 2 years of program operation. The authors used an inductively derived taxonomy of prevention program elements, student variables, student substance use, use-related variables, and institutional variables to compare the 2 groups of institutions. Only prevention program elements discriminated between groups. Factor analysis of discriminating elements identified 8 prevention factors that improved base-rate prediction of institutional decrease in binge drinking by 28.1%. Factor synthesis yielded a 3-construct binge-drinking prevention model based on student participation and involvement strategies, educational and informational processes, and campus regulatory and physical change efforts. This model improved base-rate prediction of decreased binge drinking by 33.2%.     

Striking a Balance between Program-Specific and Portfolio-Level Evaluation: Lessons Learned from a Multi-Site Evaluation on the Texas-Mexico Border

Funders are increasingly making strategic investments across multiple grantees, aiming for their portfolio to improve targeted outcomes in a specific issue area. To this end, funders might use multi-site evaluation (MSE) approaches to examine the impact of their collective investments. However, it is important to recognize that each program—and its own program evaluation—must be tailored to its setting, population, and local context to best meet the needs of its target population. Therefore, multi-site evaluations need to account for this complexity. This paper describes the Sí Texas project, a large initiative of eight grantees implementing different integrated behavioral health models to improve physical and mental health outcomes along the Texas-Mexico border. With over 4,200 MSE study participants, the evaluation for Sí Texas used a partnership-centered approach to both enhance the evidence base and build local organizational capacity. This paper describes this approach, the process of tailoring evaluation practices to the grantees’ context, and the challenge of balancing consistency at the grantee-level for the portfolio multi-site evaluation. Successes, challenges, and lessons learned related to study design, data collection, grantee partnership, and capacity building are discussed.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems—A case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors.This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring.An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult.From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems--a case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors. This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring. An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult. From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.