Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).     

Evaluating Dancing With Parkinson's: Reflections from the perspective of a community organization

In 2015, Dancing With Parkinson's (DWP), a Toronto-based community organization, participated in the Ontario Brain Institute's (OBI) newly launched Evaluation Support Program. This paper reflects on that experience. In particular, we identify the key lessons derived from the OBI initiative, discuss how these lessons have informed DWP practice going forward, and highlight what we consider to be the most valuable aspects of the Evaluation Support Program. While we now recognize the need to establish an evaluation culture within DWP, we find that there are significant challenges associated with both building and sustaining evaluation capacity in the context of a small community-based organization. Whereas DWP has built considerable strengths in terms of informal evaluation capacity, on its own, such capacity is insufficient to, for example, demonstrate DWP's impact to outside audiences or successfully scale up the program.     

Empowerment evaluation: An approach that has literally altered the landscape of evaluation

The quest for credible and actionable evidence to improve decision making, foster improvement, enhance self-determination, and promote social betterment is now a global phenomenon. Evaluation theorists and practitioners alike have responded to and overcome the challenges that limited the effectiveness and usefulness of traditional evaluation approaches primarily focused on seeking rigorous scientific knowledge about social programs and policies. No modern evaluation approach has received a more robust welcome from stakeholders across the globe than empowerment evaluation.Empowerment evaluation has been a leader in the development of stakeholder involvement approaches to evaluation, setting a high bar. In addition, empowerment evaluation’s respect for community knowledge and commitment to the people’s right to build their own evaluation capacity has influenced the evaluation mainstream, particularly concerning evaluation capacity building. Empowerment evaluation’s most significant contributions to the field have been to improving evaluation use and knowledge utilization.     

Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Through community eyes: the transition of international organizations from community aid to development in postconflict Sierra Leone

Sierra Leone experienced an 11-year civil war, brutalizing its people and destroying its communities. With the cessation of violence, international organizations helped to secure peace, deliver aid and supplies, and, after, assist with development projects. This grounded theory study, which aims to understand the role these organizations played from the viewpoint of community members in 2 communities, posits that community members’ regard of international organizations lessened as their efforts transitioned from securing the peace and relief efforts to aid for development. Highlighted are the successes and challenges of this work and a broad discussion of implications and recommendations.     

Evaluation of the Neurodegenerative Conditions Coordinated Care Program (NCCCP) in Western Australia: barriers to better service provision

This paper details a 2009-2010 evaluation of the Neurodegenerative Conditions Coordinated Care Program (NCCCP) commissioned by the Disability Services Commission (DSC) in Western Australia (WA). The program, run by the Multiple Sclerosis Society of WA (Inc.) provides in-home care and supports and respite for people who are under the age of sixty-five and diagnosed with a rapidly degenerative neurological condition. In 2009, the identified barriers to the better provision of services included: shortage of quality out-of-home respite providers, and convoluted paperwork requirements to qualify for the program. Some service providers were unaware of the program and so were not referring them into the NCCCP, a number of service providers were unwilling to refer their clients into the program, neurologists were unwilling to give a final diagnosis for a client, that would enable them to qualify for the service, there was a general lack of knowledge about what services were available in the general community, and existing equipment pools were under-resourced. In 2010, the study found that most of the issues had been addressed however, out-of-home respite services were still very limited especially in rural areas, and the eligibility criteria for entry to the program is confined to those 65 years and under. This paper discusses the issues that still remain with the program as examples of barriers to better provision of services.     

Accommodating diverse clients' needs in evaluation capacity building: A case study of the Tobacco Control Evaluation Center

In this case study, we detail and analyze how the Tobacco Control Evaluation Center (TCEC), an evaluation technical assistance center that serves approximately 100 local tobacco control organizations in California, endeavors to build capacity among the state-funded local providers it serves by using evaluation capacity building activities with an utilization-focused evaluation framework. We call this a "blended approach" and describe these methods. Satisfaction and demand for TCEC services are documented to provide measurements for evaluation capacity building. Final evaluation report scores from two intervention cycles (2004-2007 and 2007-2010) submitted to the California Health Department, Tobacco Control Division are also assessed and compared. These measures demonstrate an increase in evaluation capacity by local projects under TCEC's purview.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

Volunteering in a School Kitchen Garden Program: Cooking Up Confidence,Capabilities, and Connections!

This paper reports on the evaluation of a kitchen garden program in primary schools in Victoria, Australia. It focuses on the motivations, impacts, and issues associated with volunteering in the program. The study revealed that volunteers are drawn from a range of sources, including: families of current and former students, former teachers, local residents, clients of aged care and/or disability services, other schools and communities, local universities, community organizations, the community services sector, and the corporate sector. Benefits to volunteers included: opportunities to use time productively, an increased sense of belonging, learning opportunities, and an increased sense of self-worth and enjoyment. For schools, volunteers enhanced engagement between the school and the local community, enabled them to engage more effectively with hard to reach groups, and increased student engagement. In addition, the involvement of volunteers improved the sustainability of the program, improved communication between teachers and families of students from minority ethnic groups, and gave students the chance to relate to new people, to learn from their experience and to have fun in working with the volunteers. Perhaps the most telling benefits to flow both to students and to volunteers were not the “three Rs—reading, w’riting and a’rithmetic” but the three Cs—confidence, capabilities, and connections. However, a clearly identified issue was the importance of matching volunteers’ motivations and needs with the roles they play to sustain current levels of volunteering and, therefore, the program itself.     

Using an action research approach to involving service users in the assessment of professional competence

This paper reports on a piece of action research that has involved people who use mental health services in systematically providing feedback from a user perspective on participants’ assessed work completed for one module of a masters’ training programme in mental health. In an attempt to improve professional practice and include people who were accessing mental health services in so doing, it outlines how users were trained to provide feedback and the training methods employed. The findings summarise the kind of issues users raised in their feedback to participants about the evidence professionals provided to demonstrate their learning from the training programme. A focus group interview with professionals provides a contrasting insight into the participants’ experience of having their work commented upon from a user perspective. The paper draws on the experience of a five-year external evaluation of an interdisciplinary programme in community mental health at Birmingham University in the UK which has highlighted the involvement of people who use mental health services as a particular innovation in the design, delivery and evaluation of the curriculum.     

Participatory evaluation of a community–academic partnership to inform capacity-building and sustainability

The Tampa Bay Community Cancer Network (TBCCN) was formed as a partnership comprised of committed community based organizations (grassroots, service, health care organizations) and a National Cancer Institute designated cancer center working together to reduce cancer health disparities. Adhering to principles of community-based participatory research, TBCCN's primary aims are to develop and sustain outreach, training, and research programs that aim to reach medically underserved, multicultural and multilingual populations within the Tampa Bay tri-county area.Using a participatory evaluation approach, we recently evaluated the partnerships’ priorities for cancer education and outreach; perspectives on the partnerships’ adherence to CBPR principles; and suggestions for sustaining TBCCN and its efforts. The purpose of this paper is to describe implementation and outcomes of this participatory evaluation of a community/academic partnership, and to illustrate the application of evaluation findings for partnership capacity-building and sustainability. Our evaluation provides evidence for partners’ perceived benefits and realized expectations of the partnership and illustrates the value of ongoing and continued partnership assessment to directly inform program activities and build community capacity and sustainability.     

Evaluation of a Housing Support Program for People Who Are at Risk of Homelessness: Perspective of People Who Refer Clients

Abstract

Strategies assisting people to maintain their tenancies are at the forefront of current government policy to prevent homelessness. Based on an intensive case management model, the Housing Support Program (HSP) aimed to prevent homelessness by providing ongoing assessment, advocacy, assistance and coordination of services for clients who are deemed at risk of losing their secure accommodation.

The aim of this study was to evaluate the HSP from the perspective of the people who referred clients to the program. Twenty nine referral agents were surveyed. A focus group was also conducted. A large majority of participants (n=25, 86%) ‘strongly agreed’ or ‘agreed’ that the housing outcomes improved for referred clients after HSP intervention. Over three quarters of participants (n=23, 80%) believed that there are too few agencies actively working to support those at risk of loss of tenancy. The focus group discussion reaffirmed the role HSP plays providing continuity of care and ‘plugging the gap’ in services for disadvantaged clients.     

The Value of the Relationship in Volunteer Home Support Programs Working with Vulnerable Families

This paper reports on a program evaluation project that included interviews with 35 parents from high-risk backgrounds who received home volunteer support as part of the Dalwood Volunteer Home Support Program. It provides insight into the meaning and value parents attach to the volunteer relationship. The evaluation findings suggest that home volunteer support was associated with parent perception of increased wellbeing, decreased social isolation, and an improved relationship with their child. The results also suggest that parents perceived their relationship with their volunteer as significant in providing a secure base relationship.     

Enhancing community-based organizations’ capacity for HIV/AIDS education and prevention

The catalytic potential of community-based organizations to promote health, prevent disease, and address racial, ethnic, and socio-economic disparities in local communities is well recognized. However, many CBOs, particularly, small- to medium-size organizations, lack the capacity to plan, implement, and evaluate their successes. Moreover, little assistance has been provided to enhance their capacity and the effectiveness of technical assistance to enhance capacity is likewise limited. A unique private–academic partnership is described that simultaneously conducted program evaluation and addressed the capacity needs of 24 CBOs funded by the Pfizer Foundation Southern HIV/AIDS Prevention Initiative. Assessments of key program staff members at 12 and 18 months after the initial cross-site program assessment survey indicated a significant improvement in the CBOs’ knowledge, skills, and abilities and a substantial reduction in their technical assistance needs for HIV/AIDS prevention. Full participation of CBOs in technical assistance and a concurrent empowerment evaluation framework were necessary to enhance prevention capacity.