Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Mentoring Policy Research Organizations: Project Evaluation Results

The international community has identified a positive link between good governance and economic development. There is an increasing appreciation of the effective role that local policy research organizations (PROs) in transition and developing countries can play in providing evidence-based policy recommendations as the basis for sound legislation and in assessing the efficacy of existing programs. Capacity building programs for PROs are seen as a key component of growth strategies. The donor community has responded with support for such efforts, which take many forms. On the other hand, the evaluation of the effectiveness of these capacity building and mentoring programs has been inadequate. This article presents the evaluation of one program, focusing on two important capacity building program outcomes: improved research capacity and greater effectiveness in the policy arena. A reflexive design is used. The results suggest the program??s approach is promising.     

Environmental education: A time of change,a time for change

We join the authors in this special issue in their call to embrace a culture of evaluation. Obstacles to change are formidable. Educators debate their purpose – provide knowledge or achieve environmental goals – and we have limited evidence of the effectiveness of environmental programs and policies. Change requires collaboration across organizations and disciplines, targeted capacity building, and building systems of assessment into programs that enable more sophisticated evaluations. As in other fields, an evidence-based movement will increase the credibility and effectiveness of environmental education. A culture of evaluation offers educators a solid platform to collaboratively and efficiently achieve society's environmental goals.     

Organizational capacity building: addressing a research and practice gap

The purpose of this article is to address the gap between evaluation research, and the practice of capacity building with nonprofits. This study describes a 5-year capacity building initiative with grassroots organizations including a longitudinal evaluation of the implementation and outcomes achieved. Formative processes yielded many lessons that were used to improve the capacity building model of services. The results show that the majority of groups met a priori expectations for participation success. Organizational staff valued technology, consultants, and program funding the most. Increases were found in board membership and perceptions of visibility of the organization were enhanced. Executive directors reported greater awareness of needs and improved management knowledge. These small organizations fill many unmet needs and more capacity building evaluation studies are needed to understand the mechanisms that support their efforts and the impact on their sustainability.     

Strengthening child and youth programs: A look at inter-organizational mentoring strategies

Community-based non-profit organizations rarely have access to research or evaluation evidence to inform their programs and often lack the capacity to gather or use this information independently. In 2016, Wisdom2Action—a network of knowledge mobilization (KMb) experts, policy makers and service providers across Canada—launched an inter-organizational mentorship program to facilitate the implementation and sharing of best and promising practices within community-based programs for young people. This article outlines the findings from a developmental evaluation of eight mentoring relationships. Drawing on the Promoting Action on Research in Health Sciences (PARiHS) model of KMb, we look at mentoring as a type of facilitation that supports the increased use of evidence and evaluation information by non-profit organizations and identify key themes that support effective organizational mentorship in this sector. Findings reinforce the importance of establishing connected relationships and understanding context in mentoring relationships, creating adaptive and responsive work plans, ensuring consistent communication, and maintaining a focus on capacity-building if knowledge mobilization is to occur.     

The Interrelationship Between Program Development Capacity and Readiness for Change Among Small to Mid-Sized Nonprofits

ABSTRACT

Based on survey data from 396 small to mid-sized nonprofit organizations operating within Southeast lower Michigan, this article examines the relationship between program development capacity and readiness for change and the predictors of these two components of capacity building among small to mid-sized nonprofits. It uses a simultaneous equation approach as the basis for examining the relationship between program development capacity and readiness for change. The findings indicate that program development capacity and readiness for change are mutually enforcing processes and that the levels of program development capacity and readiness for change are predicted by two distinct sets of predictors.     

Accommodating diverse clients' needs in evaluation capacity building: A case study of the Tobacco Control Evaluation Center

In this case study, we detail and analyze how the Tobacco Control Evaluation Center (TCEC), an evaluation technical assistance center that serves approximately 100 local tobacco control organizations in California, endeavors to build capacity among the state-funded local providers it serves by using evaluation capacity building activities with an utilization-focused evaluation framework. We call this a "blended approach" and describe these methods. Satisfaction and demand for TCEC services are documented to provide measurements for evaluation capacity building. Final evaluation report scores from two intervention cycles (2004-2007 and 2007-2010) submitted to the California Health Department, Tobacco Control Division are also assessed and compared. These measures demonstrate an increase in evaluation capacity by local projects under TCEC's purview.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

Survival of the fittest: capacity building for small nonprofit organizations

This paper discusses capacity building activities designed for small nonprofits who are members of the Second Harvest Food Bank of Central Florida's ADEPT program. The Second Harvest Food Bank of Central Florida (SHFBCF) is a nonprofit organization that collects, stores and distributes donated food to more than 450 nonprofit partners in Brevard, Lake, Orange, Osceola, Seminole and Volusia counties. This project sought to delineate, design, and implement the capacity building trainings desired by ADEPT member agencies. It also analyzed the relationship between the number of clients served, number of staff, number of volunteers, and the training needs. At the conclusion of the capacity building trainings, data was collected to gauge participants' perceptions of the capacity building trainings and their perceived impact on the effectiveness of the ADEPT Program and its member agencies. The generalizability and applicability of the research results to other small community-based organizations providing social and human services is also discussed.     

Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Challenges for environmental education evaluation

The articles in this special issue cover a range of practices in environmental education evaluation, from program evaluations to training and organizational impact. This article reflects on this collection and offers six recommendations on three broad themes that the issue's authors identify as shortcomings or opportunities for change: capacity building, program theory, and learning organizations.     

University Civic Engagement with Community-Based Organizations

Abstract

This article compares and contrasts two case studies of large research universities involved in civic engagement projects with urban nonprofit community-based organizations and neighborhood associations. The article uses a community building framework in which organizational, interorganizational, and community-level features are examined. The study found that each university used a different approach through which to achieve a university-community partnership. A dispersed model favored an entrepreneurial approach for individual faculty and student involvement, while the coordinated model requested faculty and students from different departments to work together toward a community-driven goal. The extent to which these different models of civic engagement delivered what community organizations wanted was based on five factors: (1) the university's geographic proximity to a tar get low-income neighborhood, (2) leadership for institutional social commitment, (3) use of community-based research, (4) funding as a social strategy, and (5) a flexible curriculum. Challenges faced by faculty, students, and practitioners are addressed, and directions for future research are suggested.     

Enhancing community-based organizations’ capacity for HIV/AIDS education and prevention

The catalytic potential of community-based organizations to promote health, prevent disease, and address racial, ethnic, and socio-economic disparities in local communities is well recognized. However, many CBOs, particularly, small- to medium-size organizations, lack the capacity to plan, implement, and evaluate their successes. Moreover, little assistance has been provided to enhance their capacity and the effectiveness of technical assistance to enhance capacity is likewise limited. A unique private–academic partnership is described that simultaneously conducted program evaluation and addressed the capacity needs of 24 CBOs funded by the Pfizer Foundation Southern HIV/AIDS Prevention Initiative. Assessments of key program staff members at 12 and 18 months after the initial cross-site program assessment survey indicated a significant improvement in the CBOs’ knowledge, skills, and abilities and a substantial reduction in their technical assistance needs for HIV/AIDS prevention. Full participation of CBOs in technical assistance and a concurrent empowerment evaluation framework were necessary to enhance prevention capacity.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Community-based behavioral health interventions: Developing strong community partnerships

The popularity of community-based interventions has experienced a revival over the last two decades. The general theme behind this trend is that greater community involvement provides researchers and practitioners with culturally relevant information to make interventions successful. This involvement is usually solicited through the formation of community coalitions and advisory boards. Although the positive influence of community partnerships is well established, strategies for successfully developing and sustaining these partnerships are less clear. To address this gap, this paper presents evaluation findings from The Making Connections Initiative, a national effort emphasizing community-level prevention strategies for improving the mental health of men/boys. Sixteen grantees were tasked with developing a coalition of community organizations to provide input into the development and implementation of a local prevention initiative. An evaluation of the 16 grantees’ planning activities identified four themes related to developing and sustaining successful community partnerships. This article reports on the strategies that were found most successful across grantees. With the increasing popularity of community-based programs, strategies for successfully partnering with local organizations are particularly important. This study adds to the limited research on best practices for forming community partnerships and should serve as a guide for community-based evaluators and program planners.     

The theory of change of the evaluation support program: Enhancing the role of community organizations in providing an ecology of care for neurological disorders

This paper discusses the Ontario Brain Institute’s theory of change for the Evaluation Support Program, a program designed to enhance the role of community organizations in providing care and services for people living with a brain disorder. This is done by helping community organizations build evaluation capacity and foster the use of evidence to inform their activities and services. Helping organizations to build capacities to track the ‘key ingredients’ of their successes will help ensure that successes are replicated and services can be improved to maximize the benefit that people receive from them. This paper describes the hypothesized outcomes and early impacts of the Evaluation Support Program, as well as how the program will contribute to the field of evaluation capacity building.     

A framework of operating models for interdisciplinary research programs in clinical service organizations

A framework of operating models for interdisciplinary research programs in clinical service organizations is presented, consisting of a "clinician-researcher" skill development model, a program evaluation model, a researcher-led knowledge generation model, and a knowledge conduit model. Together, these models comprise a tailored, collaborative approach to enhancing research-informed practice in community-based clinical service organizations. The models place different degrees of emphasis on the development of research-related skills in practitioners, the generation of knowledge tailored to clinical practice, and knowledge sharing. The nature, philosophical basis, roles of research staff members, outputs and impacts, and strengths and limitations of each model are described, in the context of a long-standing, interdisciplinary research program in a children's rehabilitation service organization. The use of the model framework as a tool for the design of interdisciplinary, community-based research programs is discussed.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).