Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

An evaluation capacity building toolkit for principal investigators of undergraduate research experiences: A demonstration of transforming theory into practice

This paper describes the approach and process undertaken to develop evaluation capacity among the leaders of a federally funded undergraduate research program. An evaluation toolkit was developed for Computer and Information Sciences and Engineering¹ Research Experiences for Undergraduates² (CISE REU) programs to address the ongoing need for evaluation capacity among principal investigators who manage program evaluation. The toolkit was the result of collaboration within the CISE REU community with the purpose being to provide targeted instructional resources and tools for quality program evaluation. Challenges were to balance the desire for standardized assessment with the responsibility to account for individual program contexts. Toolkit contents included instructional materials about evaluation practice, a standardized applicant management tool, and a modulated outcomes measure. Resulting benefits from toolkit deployment were having cost effective, sustainable evaluation tools, a community evaluation forum, and aggregate measurement of key program outcomes for the national program. Lessons learned included the imperative of understanding the evaluation context, engaging stakeholders, and building stakeholder trust. Results from project measures are presented along with a discussion of guidelines for facilitating evaluation capacity building that will serve a variety of contexts.     

Puberty Education in a Global Context: Knowledge Gaps,Opportunities, and Implications for Policy

Youth programs and policies provide opportunities for institutions and societies to support healthy adolescent development. Puberty education programs are universally important, as they provide crucial knowledge and skills to help youth and their caregivers navigate the physical, emotional, and interpersonal changes of puberty with positive outcomes. However, few puberty programs have been rigorously evaluated, resulting in a lack of evidence‐based knowledge and practice in this area. This review examines the status of research on puberty education and related programs and draws on the broader intervention literature and recent research findings on adolescence to identify program features that might improve program effectiveness. Implications for policy are also discussed. The need for rigorous program evaluation is emphasized throughout.     

Empowerment evaluation: An approach that has literally altered the landscape of evaluation

The quest for credible and actionable evidence to improve decision making, foster improvement, enhance self-determination, and promote social betterment is now a global phenomenon. Evaluation theorists and practitioners alike have responded to and overcome the challenges that limited the effectiveness and usefulness of traditional evaluation approaches primarily focused on seeking rigorous scientific knowledge about social programs and policies. No modern evaluation approach has received a more robust welcome from stakeholders across the globe than empowerment evaluation.Empowerment evaluation has been a leader in the development of stakeholder involvement approaches to evaluation, setting a high bar. In addition, empowerment evaluation’s respect for community knowledge and commitment to the people’s right to build their own evaluation capacity has influenced the evaluation mainstream, particularly concerning evaluation capacity building. Empowerment evaluation’s most significant contributions to the field have been to improving evaluation use and knowledge utilization.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

The We-Lead Model for Bridging the Low-Income Community Leadership Skills-Practice Gap

There are several community leadership development programs that devote tremendous resources to help improve the vitality and wealth of individuals and communities through the development of local leadership capacities. One great challenge is that these programs fail to coordinate existing community resources to create opportunities to engage program graduates. This article interrogates the effectiveness of leadership development programs in creating opportunities for engagement. Expanding on U-Lead model, the article proposes a WE-Lead model that can bridge the skills-practice gap by emphasizing the importance of aligning community leadership development programs with community needs through building community coordinated coalitions.     

Pre-, post-, and longitudinal evaluation of juvenile justice education

This article describes two stages of the Juvenile Justice Educational Enhancement Program's pre-, post-, and longitudinal evaluation research. Pilot studies were used to explore how to design statewide research of pre- and postassessment scores and community reintegration outcomes. Preliminary findings suggest that higher performing educational programs produce greater educational gains as measured by academic achievement tests, credits earned, and pupil progression rates. The findings also indicate that these programs have more students returning to school and lower recidivism rates. Building on the pilot studies, refinements were made to the research designs to enable more comprehensive statewide evaluation. Current research includes collection of pre- and postassessment scores from official sources on approximately 16,000 juvenile justice youths. In addition, a research design has been developed to examine program effectiveness by measuring community reintegration variables. Multiple data sources, including official and self-reported data on family, school, employment, and subsequent crime involvement, will be used in the longitudinal study.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).     

Evaluation design for a community prevention trial. An environmental approach to reduce alcohol-involved trauma

The Community Prevention Trial was 5-year effort to reduce alcohol-involved injuries and death through a comprehensive program of community awareness and policy activities. The three experimental communities were of approximately 100,000 population each (one in Northern California, one in Southern California, and one in South Carolina). Matched comparison communities were used for each experimental community. This article describes the evaluation approach used in a program that sought to change environmental factors not a specific population or target group. This approach demanded unique evaluation approaches for determining overall community aggregate effects, that is, distal outcomes, as well as changes in key mediating variables, that is, process effects. The problem of trending and lagged effects of community prevention programs are discussed.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.     

Growing with youth: A lifewide and lifelong perspective on youth-adult partnership in youth programs

Youth-adult partnership (Y-AP) shows promise for positive youth development, yet research seldom examines how youth programs employ Y-AP as a developmental practice. This paper provides a developmental ecological perspective on Y-AP in programs and communities with data collected across a mid-sized city. In Study 1, interview data suggest adult practitioners hold three distinct goals for Y-AP: voice, decisionmaking, and leadership. We identify program practices for carrying out YAP including building positive adult-youth relationships, engaging youth in first-hand learning, and addressing developmental progression; i.e., gradually increasing opportunities and responsibilities as youth age. In Study 2 we investigate this practice of supporting developmental progression with a case study focused on adolescent opportunities and supports in a multi-age program. Study 2 findings present a picture of YAP opportunities highly integrated into program operations; in particular, integrated into the multiage context. These studies offer insight into strengthening program and community capacity for Y-AP.     

Faith-based evaluation: accountable to whom, for what?

Findings, issues, and lessons learned about program evaluation are examined from a national qualitative study of 15 faith-based human service programs targeting those in need in urban areas. Using a grounded theory design, five properties emerge as part of the evaluation network: (1) philosophy of accountability, (2) legitimacy, (3) evaluation design, (4) feedback loop, and (5) barriers to evaluation. While funders expect measurable outcomes to evaluate service effectiveness, respondents acknowledge other competing expectations of multiple constituents in religious and secular communities. What emerges is an excellent example of managing multiple program evaluation demands in programs that are particularly facile at process evaluation in the interest of quality service and relationship building. The article concludes with important lessons learned about the process of program evaluation.     

Constructing a consensus-based prevention outcome measurement instrument

This article describes Ohio's Prevention Evaluation Project (PEP). The purpose of the project was to develop a process and instrument to assess behavioral and attitudinal outcomes in primary drug abuse prevention programs targeting young people aged twelve to seventeen. One of PEP's principal achievements was the inclusion of community prevention program providers from throughout the state in the evaluation instrument development. The effort produced a self-administered questionnaire to capture data on young peoples' drug use practices, attitudes toward drugs, and problematic behaviors. Most significantly, the product produced was the acceptance and endorsement of community-based programs. The forty-one-item questionnaire has good to excellent reliability for virtually all measures, as assessed by the test-retest method. The instrument is a product of a process that brought together a university-based research team, state alcohol and drug abuse prevention administrators, and representatives of public programs. PEP demonstrates how to productively link service providers, administrators, and evaluators to develop a process for assessing the effectiveness of prevention programs. The process and tools described serve as a model for other communities wanting to evaluate their prevention programming.     

Building evaluation capacity: definitional and practical implications from an Australian case study

Internationally, evaluation capacity-building activities have mushroomed as demands have increased for government-funded programs to demonstrate that they are effective and efficient. Despite this, there is a lack of clarity in the way in which evaluation capacity-building is defined and conceptualized. The current paper presents a case study of a national evaluation capacity building exercise that we are in the midst of conducting in Australia, and discusses the findings in relation to definitional, conceptual and practical issues. Specifically, we describe an evaluation capacity building exercise involving over 100 mental health projects, detailing the methods that we employ, some of the challenges that we have faced, and the benefits we feel we are achieving. Our key message is that definitions of evaluation capacity-building should not only make reference to equipping organizations to routinely conduct evaluations, but should also stress the varied uses to which evaluation findings can be put. In addition, such definitions should acknowledge some of the valuable by-products of evaluation capacity building activities, such as the development of shared understandings of the program or project being evaluated.     

The theory of change of the evaluation support program: Enhancing the role of community organizations in providing an ecology of care for neurological disorders

This paper discusses the Ontario Brain Institute’s theory of change for the Evaluation Support Program, a program designed to enhance the role of community organizations in providing care and services for people living with a brain disorder. This is done by helping community organizations build evaluation capacity and foster the use of evidence to inform their activities and services. Helping organizations to build capacities to track the ‘key ingredients’ of their successes will help ensure that successes are replicated and services can be improved to maximize the benefit that people receive from them. This paper describes the hypothesized outcomes and early impacts of the Evaluation Support Program, as well as how the program will contribute to the field of evaluation capacity building.     

A Trauma-Informed Model for Empowerment Programs Targeting Vulnerable Youth

Youth-led program development, organizing, research, and health promotion have been identified as an important practices for community practitioners. Since such practices target vulnerable youth, it is critical that such empowerment programs are trauma-informed. This paper addresses trauma, its potential impacts, and cultural differences in understanding trauma as well as the assumptions of empowerment and youth-led programming. The relationship between health and empowerment is described as well as how youth empowerment programs can specifically target symptoms of powerlessness, low self-esteem and interpersonal difficulties, commonly experienced by youth living in socially toxic environments. Implications for program development, research, and policy are considered.     

Can a workbook work? Examining whether a practitioner evaluation toolkit can promote instrumental use

In large-scale, multi-site contexts, developing and disseminating practitioner-oriented evaluation toolkits are an increasingly common strategy for building evaluation capacity. Toolkits explain the evaluation process, present evaluation design choices, and offer step-by-step guidance to practitioners. To date, there has been limited research on whether such resources truly foster the successful design, implementation, and use of evaluation findings. In this paper, we describe a multi-site project in which we developed a practitioner evaluation toolkit and then studied the extent to which the toolkit and accompanying technical assistance was effective in promoting successful completion of local-level evaluations and fostering instrumental use of the findings (i.e., whether programs directly used their findings to improve practice, see Patton, 2008). Forensic nurse practitioners from six geographically dispersed service programs completed methodologically rigorous evaluations; furthermore, all six programs used the findings to create programmatic and community-level changes to improve local practice. Implications for evaluation capacity building are discussed.     

Evaluating health services research capacity building programs: Implications for health services and human resource development

Evaluation of research capacity building programs presents challenges, including defining “research capacity building” and measuring it. The present study evaluates infrastructure capacity-building strategies and barriers and facilitators of two federally funded health-related research capacity building programs within the United States. We developed a conceptual framework and abstracted 223 data elements from existing project files. We administered questionnaires to 17 principal investigators and conducted 2-h interviews of 15 principal investigators. Finally, we conducted one-and-a-half-day site visits with five projects. We aggregated data and analysis results across the two programs. Impacts and strategies were also analyzed as to whether host institutions had little or no existing research infrastructure (“seed projects”) or had well-developed research infrastructure (“fertilizer projects”). Projects were associated with increases in publication and presentation rates by Principal Investigators and staff, increases in research funding, changes in organizational research culture, and positive impacts on the community. Project characteristics and activities associated with each of these outcomes were identified. It was also noted that the correlations between these outcomes were small and non-significant.