Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).     

Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Enhancing community-based organizations’ capacity for HIV/AIDS education and prevention

The catalytic potential of community-based organizations to promote health, prevent disease, and address racial, ethnic, and socio-economic disparities in local communities is well recognized. However, many CBOs, particularly, small- to medium-size organizations, lack the capacity to plan, implement, and evaluate their successes. Moreover, little assistance has been provided to enhance their capacity and the effectiveness of technical assistance to enhance capacity is likewise limited. A unique private–academic partnership is described that simultaneously conducted program evaluation and addressed the capacity needs of 24 CBOs funded by the Pfizer Foundation Southern HIV/AIDS Prevention Initiative. Assessments of key program staff members at 12 and 18 months after the initial cross-site program assessment survey indicated a significant improvement in the CBOs’ knowledge, skills, and abilities and a substantial reduction in their technical assistance needs for HIV/AIDS prevention. Full participation of CBOs in technical assistance and a concurrent empowerment evaluation framework were necessary to enhance prevention capacity.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

The theory of change of the evaluation support program: Enhancing the role of community organizations in providing an ecology of care for neurological disorders

This paper discusses the Ontario Brain Institute’s theory of change for the Evaluation Support Program, a program designed to enhance the role of community organizations in providing care and services for people living with a brain disorder. This is done by helping community organizations build evaluation capacity and foster the use of evidence to inform their activities and services. Helping organizations to build capacities to track the ‘key ingredients’ of their successes will help ensure that successes are replicated and services can be improved to maximize the benefit that people receive from them. This paper describes the hypothesized outcomes and early impacts of the Evaluation Support Program, as well as how the program will contribute to the field of evaluation capacity building.     

Mentoring Policy Research Organizations: Project Evaluation Results

The international community has identified a positive link between good governance and economic development. There is an increasing appreciation of the effective role that local policy research organizations (PROs) in transition and developing countries can play in providing evidence-based policy recommendations as the basis for sound legislation and in assessing the efficacy of existing programs. Capacity building programs for PROs are seen as a key component of growth strategies. The donor community has responded with support for such efforts, which take many forms. On the other hand, the evaluation of the effectiveness of these capacity building and mentoring programs has been inadequate. This article presents the evaluation of one program, focusing on two important capacity building program outcomes: improved research capacity and greater effectiveness in the policy arena. A reflexive design is used. The results suggest the program??s approach is promising.     

Supporting and Sustaining Grassroots Youth Organizations: The Case of New Detroit

Grassroots organizations have a central place in service provision within many urban communities, particularly as they work with young people. Enhancing the potential of youth serving organizations is vital as many of these small groups lack infrastructure and resources. One organization has gone beyond grant making to organizations to include providing technical assistance, then moving to a model of capacity building to improve their ability to fulfill their mission. This paper describes the efforts of New Detroit to improve the infrastructure of youth serving grassroots organizations through a multi-faceted model and the outcomes evidenced from the first cohort. In addition we explore the contradictions inherent in working with small groups. How do you improve effectiveness without destroying the “homegrown” character these groups possess? The model presented here shows the value of empowering small grassroots groups to build capacity.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Management‐capacity building in the Pittsburgh region

This article describes the strategy for management‐capacity building employed by The Forbes Funds in Pittsburgh, Pennsylvania. Like many other cities, Pittsburgh's capacity‐building industry has grown dramatically in recent years and is somewhat fragmented. But The Forbes Funds, along with other philanthropic organizations, have played an important role in developing a coherent system of management support services for nonprofit organizations in the Pittsburgh area. The article concludes with lessons that can be applied in other regions of the country.     

Survival of the fittest: capacity building for small nonprofit organizations

This paper discusses capacity building activities designed for small nonprofits who are members of the Second Harvest Food Bank of Central Florida's ADEPT program. The Second Harvest Food Bank of Central Florida (SHFBCF) is a nonprofit organization that collects, stores and distributes donated food to more than 450 nonprofit partners in Brevard, Lake, Orange, Osceola, Seminole and Volusia counties. This project sought to delineate, design, and implement the capacity building trainings desired by ADEPT member agencies. It also analyzed the relationship between the number of clients served, number of staff, number of volunteers, and the training needs. At the conclusion of the capacity building trainings, data was collected to gauge participants' perceptions of the capacity building trainings and their perceived impact on the effectiveness of the ADEPT Program and its member agencies. The generalizability and applicability of the research results to other small community-based organizations providing social and human services is also discussed.     

Accommodating diverse clients' needs in evaluation capacity building: A case study of the Tobacco Control Evaluation Center

In this case study, we detail and analyze how the Tobacco Control Evaluation Center (TCEC), an evaluation technical assistance center that serves approximately 100 local tobacco control organizations in California, endeavors to build capacity among the state-funded local providers it serves by using evaluation capacity building activities with an utilization-focused evaluation framework. We call this a "blended approach" and describe these methods. Satisfaction and demand for TCEC services are documented to provide measurements for evaluation capacity building. Final evaluation report scores from two intervention cycles (2004-2007 and 2007-2010) submitted to the California Health Department, Tobacco Control Division are also assessed and compared. These measures demonstrate an increase in evaluation capacity by local projects under TCEC's purview.     

Environmental education: A time of change,a time for change

We join the authors in this special issue in their call to embrace a culture of evaluation. Obstacles to change are formidable. Educators debate their purpose – provide knowledge or achieve environmental goals – and we have limited evidence of the effectiveness of environmental programs and policies. Change requires collaboration across organizations and disciplines, targeted capacity building, and building systems of assessment into programs that enable more sophisticated evaluations. As in other fields, an evidence-based movement will increase the credibility and effectiveness of environmental education. A culture of evaluation offers educators a solid platform to collaboratively and efficiently achieve society's environmental goals.     

Toward More Targeted Capacity Building: Diagnosing Capacity Needs Across Organizational Life Stages

Investments to build nonprofit organizational capacity have increased rapidly in recent years as both funders and nonprofits seek ways to improve performance. Yet, while research has elucidated the variety of capacity areas organizations generally depend on, we do not yet have a clear empirical understanding of which capacity areas are important to focus on developing or how these capacity needs vary across organizations. Using a life-cycle perspective and extensive data on 71 human service organizations, we examine the relative strength of ten capacity areas at different stages of the nonprofit life-cycle. Findings contribute greater empirical grounding for research on organizational capacity development and have implications for researchers, managers, and funders interested in nonprofit capacity building.     

Building evaluation capacity: definitional and practical implications from an Australian case study

Internationally, evaluation capacity-building activities have mushroomed as demands have increased for government-funded programs to demonstrate that they are effective and efficient. Despite this, there is a lack of clarity in the way in which evaluation capacity-building is defined and conceptualized. The current paper presents a case study of a national evaluation capacity building exercise that we are in the midst of conducting in Australia, and discusses the findings in relation to definitional, conceptual and practical issues. Specifically, we describe an evaluation capacity building exercise involving over 100 mental health projects, detailing the methods that we employ, some of the challenges that we have faced, and the benefits we feel we are achieving. Our key message is that definitions of evaluation capacity-building should not only make reference to equipping organizations to routinely conduct evaluations, but should also stress the varied uses to which evaluation findings can be put. In addition, such definitions should acknowledge some of the valuable by-products of evaluation capacity building activities, such as the development of shared understandings of the program or project being evaluated.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

Smartphones and young publics: A new challenge for public relations practice and relationship building

This study focuses on smartphones and their potential contribution to relationship building, between organizations and young publics. A survey among a representative sample of 550 young, Israelis, aged 21–31, and sixty personal in-depth interviews with undergraduate students were, conducted. The findings reveal that the smartphone satisfies both interaction-related and cognitive-related, needs, but organizations do not utilize the dialogic potential of the smartphone to engage and, to build relationships with young publics.     

How activist organizations are using the Internet to build relationships

This study examines the mediated communication of activist organizations to understand how these groups use their Web sites to build relationships with publics. A study of one hundred environmental organization Web sites identified common features and examined the incorporation of dialogic communication into this new medium. The data suggest that while most activist organizations meet the technical and design aspects required for dialogic relationship building on the Web, they are not yet fully engaging their publics in two-way communication. Moreover, it appears that the activist organizations are better prepared to address the needs of member publics rather than media needs.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.