Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method

Most Transdisciplinary Research (TdR) projects combine scientific research with the building of decision making capacity for the involved stakeholders. These projects usually deal with complex, societally relevant, real-world problems. This paper focuses on TdR projects, which integrate the knowledge of researchers and stakeholders in a collaborative transdisciplinary process through structured methods of mutual learning. Previous research on the evaluation of TdR has insufficiently explored the intended effects of transdisciplinary processes on the real world (societal effects). We developed an evaluation framework for assessing the societal effects of transdisciplinary processes. Outputs (measured as procedural and product-related involvement of the stakeholders), impacts (intermediate effects connecting outputs and outcomes) and outcomes (enhanced decision making capacity) are distinguished as three types of societal effects. Our model links outputs and outcomes of transdisciplinary processes via the impacts using a mediating variables approach. We applied this model in an ex post evaluation of a transdisciplinary process. 84 out of 188 agents participated in a survey. The results show significant mediation effects of the two impacts "network building" and "transformation knowledge". These results indicate an influence of a transdisciplinary process on the decision making capacity of stakeholders, especially through social network building and the generation of knowledge relevant for action.     

Accommodating diverse clients' needs in evaluation capacity building: A case study of the Tobacco Control Evaluation Center

In this case study, we detail and analyze how the Tobacco Control Evaluation Center (TCEC), an evaluation technical assistance center that serves approximately 100 local tobacco control organizations in California, endeavors to build capacity among the state-funded local providers it serves by using evaluation capacity building activities with an utilization-focused evaluation framework. We call this a "blended approach" and describe these methods. Satisfaction and demand for TCEC services are documented to provide measurements for evaluation capacity building. Final evaluation report scores from two intervention cycles (2004-2007 and 2007-2010) submitted to the California Health Department, Tobacco Control Division are also assessed and compared. These measures demonstrate an increase in evaluation capacity by local projects under TCEC's purview.     

Knowledge Transfer and Capacity Building Through the Temporary Return of Qualified Nationals to Afghanistan

Increasing research has illustrated the positive effects of the diaspora on the country of origin. The majority of this research, however, has focused on the effects of diaspora remittances and investments. This article examines the effects of the diaspora on knowledge transfer and capacity building in a post‐conflict environment. The article is based on a case study of the International Organization for Migration the Netherlands, Temporary Return of Qualified Nationals to Afghanistan project, in which 59 highly skilled Dutch‐Afghans returned to work with a host institution (public or private) in Afghanistan for three months to transfer knowledge and build capacity. In‐depth interviews were conducted with project participants, host institutions, participants' colleagues, and key informants to gather an understanding of how participants contributed. The article finds that diaspora temporary returnees were most effective in forms of tacit knowledge transfer and that the knowledge transfer process effectively led to capacity building in Afghanistan.     

Strengthening child and youth programs: A look at inter-organizational mentoring strategies

Community-based non-profit organizations rarely have access to research or evaluation evidence to inform their programs and often lack the capacity to gather or use this information independently. In 2016, Wisdom2Action—a network of knowledge mobilization (KMb) experts, policy makers and service providers across Canada—launched an inter-organizational mentorship program to facilitate the implementation and sharing of best and promising practices within community-based programs for young people. This article outlines the findings from a developmental evaluation of eight mentoring relationships. Drawing on the Promoting Action on Research in Health Sciences (PARiHS) model of KMb, we look at mentoring as a type of facilitation that supports the increased use of evidence and evaluation information by non-profit organizations and identify key themes that support effective organizational mentorship in this sector. Findings reinforce the importance of establishing connected relationships and understanding context in mentoring relationships, creating adaptive and responsive work plans, ensuring consistent communication, and maintaining a focus on capacity-building if knowledge mobilization is to occur.     

Significant Conversations or Reduced Relational Capacity? Exploring Couple and Family Therapists' Expectations for Including a Client Feedback Procedure

Little information exists on the specific reflective processes that precede therapists' choice of including or not including a structured feedback procedure in their established practice. In the present study, we aim to describe and explore experienced couple and family therapists' expectations for how the comprehensive therapy feedback procedure Systemic Therapy Inventory of Change might influence their practices. Through a thematic analysis of six focus group interviews, two main themes were elicited. The first main theme involved values embedded in current practice, with the following subthemes: (1) balance between the therapist's expert knowledge and the client's experience‐based knowledge; and (2) making time to build alliances. The second main theme concerned the presumed influence on current practice, with the following subthemes: (1) ‘Systemic Therapy Inventory of Change will disturb the therapist's capacity to stay present’; (2) ‘Systemic Therapy Inventory of Change will shape conversations’; and (3) ‘Systemic Therapy Inventory of Change will contribute to increased qualitative depth’. We discuss these results in relation to the notion of Systemic Therapy Invention of Change as a ‘third party in the room’. We also discuss the results in relation to implications for collaborative practices, and for client and therapist autonomy.     

Evaluation of a youth agency's supervision practices: A mixed-method approach

This research presents the findings from an evaluation and organizational development initiative that was requested by a Canadian youth agency working in a large urban setting. A team of four researchers affiliated with the Center for Research on Educational and Community Services (CRECS) at the University of Ottawa conducted the evaluation. The purpose of the evaluation was to identify the supervision needs and challenges of coordinators and front line staff, assess the efficiency of the current supervision practices, and evaluate the supervisors’ and supervisees’ satisfaction with these current practices. A literature review was performed to help provide a clear definition of ‘supervision’ and the different professional roles it encompasses. Additionally, research evidence pertaining both to what contributes to supervision efficacy and supervisor competency was reviewed to distill the most robust findings in the existing literature. The lines of evidence consisted of a document and file review, an online employee survey, group discussions (i.e. focus groups), and interviews with key informants. The results of the evaluation helped the research team formulate recommendations to the agency for the development of enhanced supervision practices across its various service areas.     

Forging and sustaining research-minded professionals: teaching undergraduate research through student-community partnerships

The teaching of research is a well-established core component of social work curricula around the world. While educators in social work recognize the value, and need for research they have encountered persistent discomfort and reluctance from students to recognize this need. Practitioners in the field of social work often share this reserve either due to limited self-efficacy with regard to research knowledge and skills or practical constraints. With the growing need for practice-based knowledge, accountability and evidence-based practices to drive outcomes and service effectiveness, it is becoming imperative to not only prepare future social work professionals with the required skill sets but also support current professionals to meet these expectations. This paper showcases a pedagogical endeavor with an undergraduate research course in an American rural university, designed and implemented to create and support research-minded professionals through student–agency research partnerships. Through a collaborative, service-oriented learning, students have gained a new appreciation for research and agency connections while agencies have welcomed the path created to pursue their research agenda. This experience has valuable implications for research capacity building, and for community engaged research empowering future and current professionals to suitably use research toward effective and meaningful service provision.     

Back to Basics

Nonprofits are under increased accountability pressures to demonstrate their effectiveness. Output measurement (how much is produced) is disregarded as simplistic. Emphasis is made instead on measuring outcomes (changes in the lives of beneficiaries) or impacts (effects developed relative to the mission of the nonprofit, or the overall public good), and a growing portion of organizations state that they measure these effects. However, we question the assumption that outputs such as the number of beneficiaries served are being adequately measured. We first review existing research gaps on results measurement practices and discuss the main types of obstacles to the quality and utility of evaluation data. In this context, we argue for the need to reground nonprofit evaluation in the profound knowledge available about beneficiary populations. We discuss the potential and limitations of reach, a basic output indicator that is defined as the number of individuals directly affected by a nonprofit, and explore the organizational drivers of reach measurement. Evidence from 2,229 nonprofits shows they still lack adequate data on the beneficiaries they serve, face relevant conceptual and practical hurdles when trying to identify them, and are significantly influenced by organizational factors in their capacity to track them. Our research not only shows that nonprofits fail to adequately measure outputs, but also that measuring the number of beneficiaries served and how they are served is not as straightforward as outcome and impact advocates suggest. Practitioners and funders are reminded of the need to place beneficiaries at the core of their evaluation efforts.     

Testing Faith

Abstract

The U.S. federal government, through Charitable Choice, has opened public funding for the delivery of social services to faith-based organizations (FBOs) more than ever before. This increased access to governmental funding at all levels has led to a closer examination of the evidence base on the effectiveness of the services provided by FBOs, and the capacity of FBOs to respond to data needs for accountability and program improvement efforts. This paper discusses the current status of evaluation research on FBO services and the emerging data needs among faith-based providers. Promising avenues for enhancing the current understanding of outcomes of FBO services are explored such as (1) adopting outcome measurement practices in use within the current nonprofit sector, and (2) developing more rigorous research designs tailored to the special contexts of faith-based services. The paper suggests an agenda in which FBO operators, funders, and evaluators work together to move forward in improving the evidence base on faith-based services.     

Empowerment evaluation: An approach that has literally altered the landscape of evaluation

The quest for credible and actionable evidence to improve decision making, foster improvement, enhance self-determination, and promote social betterment is now a global phenomenon. Evaluation theorists and practitioners alike have responded to and overcome the challenges that limited the effectiveness and usefulness of traditional evaluation approaches primarily focused on seeking rigorous scientific knowledge about social programs and policies. No modern evaluation approach has received a more robust welcome from stakeholders across the globe than empowerment evaluation.Empowerment evaluation has been a leader in the development of stakeholder involvement approaches to evaluation, setting a high bar. In addition, empowerment evaluation’s respect for community knowledge and commitment to the people’s right to build their own evaluation capacity has influenced the evaluation mainstream, particularly concerning evaluation capacity building. Empowerment evaluation’s most significant contributions to the field have been to improving evaluation use and knowledge utilization.     

Application of an organizational evaluation capacity self-assessment instrument to different organizations: Similarities and lessons learned

Organizational evaluation capacity (EC) has received significant attention in the evaluation research literature in the past decade. Much of the focus has been on defining organizational evaluation capacity, which can be thought of as the competencies and structures required to conduct high-quality evaluation studies (capacity to do), as well as the organization's ability to integrate evaluation findings into its decision-making processes (capacity to use). This paper seeks to contribute to this growing body of knowledge through a multiple case study of EC across three different organizations (e.g., non-profit, provincial government and federal government, herein named sectors); the novelty of this particular study is that each case study is based on the use of a common measurement tool developed by Bourgeois, Toews, Whynot and Lamarche (2013). The cross-case analysis presented in the paper reveals that evaluation capacity tends to be higher, both in terms of capacity to do and capacity to use, in organizations that have developed systematic mechanisms to institute an evaluation culture within their walls. Interestingly, however, we also found that capacity to use does not first require capacity to do, as evidenced in the non-profit organization under study.     

From "Us" to "Them": Diasporic Linkages and Identity Politics

Global diasporas-a type of social formation mediating economic, political, and cultural affairs across borders-have been a focus of globalization researchers for some time. However, up to now, little knowledge exists on how social identification affects business participation in diaspora communities and how such participation modifies social identification. This article, based on empirical research on diasporic linkages between Taiwanese transnationals and ethnic Chinese overseas, serves to illustrate a) how globalization has enhanced the practical and economic roles of diasporas, and b) how economic practices and ethnic identification interact within diasporic communities. The author argues that ethnic membership still remains contested, despite diasporas serving as flexible forms of social organization in the mediation of capital flow.     

Autism self-advocacy in the Netherlands: past,present and future

This article presents the results of participatory research into the roles and practices of autistic self-advocates in the Netherlands, and the outcomes of their activities. The article discusses the history of Dutch autism self-advocacy, situating it within the history and practices of self-advocacy internationally and the socio-cultural context of the Netherlands. Particular reference is made to Judi Chamberlin’s model for building effective self-advocacy organisations. Key findings include the scope of significant achievements, and the identification of barriers to efficacy in the areas of governance, personal and organisational capacity, relationships with other organisations, and coalition-building. The research concludes by considering what practices could serve to build increased capacity and efficacy, based on the experiences of these and other self-advocates.     

BEST PRACTICES INQUIRY: A MULTIDIMENSIONAL,VALUE-CRITICAL FRAMEWORK

This article offers a multidimensional framework that broadens current approaches to "best practices" inquiry to include (1) the perspectives of both the consumers of services and professional practitioners and (2) a value-based critique. The predominant empirical approach to best practices inquiry is a necessary, but not sufficient, component of seeking state-of-the-art knowledge. A best practices inquiry is improved when it is broadened to include experiences and preferences of consumers, the wisdom of professionals in a given field, and a critical appraisal of the values inherent in best practices interventions themselves. Implications for social work education are discussed.     

Reconfiguring Social Work Education to Fight HIV–AIDS: A Model for Developmental Contexts

This article charts the response of social work educators in addressing HIV–AIDS. Based at the University of the West Indies in the Republic of Trinidad and Tobago, the faculty within the Social Work Unit developed a model of teaching, research and practice that was innovative and transcended cultural, geographical and professional boundaries. The model has contributed a lasting legacy of knowledge and culturally relevant practices to enhance the capacity and effectiveness of social workers working with people living with HIV–AIDS (PLWHIV). Inherent in the propositions that underpinned the model was a belief in the role of education in alleviating psychosocial and other impacts of social problems. This article describes this unique initiative and discusses its relevance to current social work practice and the implications for contemporary social work education. The authors argue that while social work draws on a common body of knowledge, an established set of professional skills and a universal code of values, these central tenets of the profession should not be regarded as concrete and fixed but must be figured and reconfigured as needed to address emerging contemporary problems and their specific sociocultural manifestations.     

100 years of "just say no" versus "just say know". Reevaluating drug education goals for the coming century

Through comparative socio-historical analysis of American school-based drug education, this review critically examines past perspectives and practices and how they shaped current programs. Among the key findings emerging from this analysis: Contrary to the popular belief that drug education began in the 1960s, its roots actually go back at least 115 years to the advent of compulsory temperance instruction. Although the particular substances targeted by such approaches have changed, the underlying approaches and dominant "no-substance-use" injunction has not. Despite the existence of "informed choice" approaches, throughout much of this period, evaluation efforts continue to be constrained by the limited dictates of "no-use" perspectives. A pragmatic alternative to contemporary "Just Say No" education is offered that strives to minimize potential harm resulting from the uninformed misuse of licit and illicit substances. A unique evaluative strategy designed to assess the effectiveness of this form of "informed choice" or "harm reduction" drug education is discussed.     

Mobilities and Knowledge Transfer: Understanding the Contribution of Volunteer Stays to North–South Healthcare Partnerships

In this paper, I review recent literature on the relationship between human mobility and knowledge transfer processes, and apply some of the more theoretical ideas to an evaluation of the kinds of mobility in which UK participants engage in the context of North–South healthcare partnerships (HCPs). Much of the contemporary research on highly skilled mobility highlights the role that short stays and circular mobility play in knowledge transfer processes. I consider the value of short stays in the travel–mobility–migration continuum and the contribution they make to the transfer, translation and effectiveness of knowledge. Healthcare partnerships are best conceptualized as knowledge networks, with various forms of mobility (both physical and virtual) playing a critical role in network‐generation, evolution and impact. I outline the findings of two studies. The first is an evaluation of HCPs (Ackers and Porter, 2011) and the second an in‐depth ongoing evaluation of one HCP known as the Liverpool–Mulago Partnership or LMP¹ (Porter et al., 2011). The results raise questions about the relationship between the different types of mobility that participants in HCPs (predominantly professional volunteers) engage in, particularly the duration and repeated quality of stays, and knowledge transfer processes. Williams argues that “there are still major gaps in our understanding of the specific contribution of international migration to knowledge transfer, of the processes involved, and of the conditions that facilitate this” (2006 588). In this paper, I begin to close some of these gaps and advance our understanding of the relationship between different forms of mobility and knowledge transfer processes, in the hopes that this will build the evidence base supporting the contribution of North–South HCPs to the achievement of the Millennium Development Goals ² .