Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Evaluation capacity building (ECB) interventions and the development of sustainable evaluation practice: An exploratory study

Evaluation capacity building (ECB) is a practice that can help organizations conduct and use evaluations; however, there is little research on the sustainable impact of ECB interventions. This study provides an empirical inquiry into how ECB develops sustained evaluation practice. Interviews were conducted with 15 organizational leaders from non-profits, higher education institutions, and foundations that “bought in” to ECB and were at least six months removed from an ECB contract. The result of this work highlights how sustained evaluation practice developed over time and what these practices looked like in real-world settings. A developmental, iterative cycle for how ECB led organizations to sustain evaluation practice emerged around key components to sustainability. First, leadership supported ECB work and resources were dedicated to evaluation. Staff began to conduct and use evaluation, which led to understanding the benefits of evaluation, and promoted value and buy-in to evaluation among staff. Common barriers and emerging sustainability supports not previously identified by ECB literature—the “personal” factor and ongoing ECB practitioner contact—are described. Practical tips for ECB practitioners to promote sustainability are also detailed.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

Civil Society Capacity Building for Democratic Reform: Experience and Lessons from Indonesia

This article examines civil society strengthening experience in Indonesia to illuminate issues, challenges, and lessons for non-governmental organization (NGO) capacity building and international donor-supported democratic reform. The authors conceive of capacity as a function of contextual factors, and internal factors associated with an individual NGO or a network of NGOs. Contextual factors that need to be taken into account in Indonesia include weak reform implementation, state distrust of NGOs, and backsliding on some basic freedoms. Among the important internal features of NGOs in democracy promotion are overreliance on confrontational advocacy strategies, shallow organizational capacity, inability to cooperate to leverage impact, limited outreach to indigenous constituencies and sustainability problems. Indonesia’s democracy-promotion NGO coalitions have largely operated as instruments of donor-supported reforms. As they seek to become socially embedded actors pursuing indigenous agendas, they face the need to confront the various expectations of their stakeholders regarding their roles and legitimacy, develop flexibility to respond to new engagements with government and with citizens, and address their internal capacity gaps. Three cases are presented that illustrate both the problems and the encouraging progress with government–NGO collaborations in democratic governance.     

Who's on board? Child welfare worker reports of buy-in and readiness for organizational change

This study views the extent to which staff buy-in for an organizational innovation in child welfare (CW) relates to implementation progress. The study occurs during implementation of a statewide practice model that was supported with technical assistance from the Mountains and Plains Child Welfare Implementation Center (MPCWIC) and framed around the National Implementation Research Network model. Mixed methods were used to address three study questions: (1) what is the level and nature of buy-in related to the innovation? (2) does buy-in vary according to staff characteristics, and (3) what is the relationship between buy-in, local level agency readiness, and implementation status one year after project start? Survey data were collected from 568 CW staff in 13 local county agencies and 12 implementation specialists assigned as coaches. Focus groups and interviews were conducted with 52 staff in four agencies. Bivariate chi-square analyses and multivariate regression using a cumulative logit model showed that buy-in was related to gender and agency tenure. Implementation progress was higher among smaller agencies, and agencies with lower levels of job stress. Qualitative themes centered on staff inclusivity in project design, communication, and supervisor support. Findings highlight the need to adapt implementation strategies in urban and rural locales, and to attend strongly to staff selection, supervision, and inclusion during implementation. Addressing job stress may help bolster implementation.     

Research evidence utilization in policy development by child welfare administrators

An exploratory qualitative study was conducted to explore how child welfare administrators use research evidence in decision-making. Content analysis revealed that a cultural shift toward evidence-based practice (EBP) is occurring in Canadian child welfare organizations and multiple types of evidence inform policy decisions. Barriers to using evidence include individual, organizational, and environmental factors. Facilitating factors include the development of internal champions and organizational cultures that value EBP. Integrating research into practice and policy decisions requires a multifaceted approach of creating organizational cultures that support research utilization and supporting senior bureaucrats to use research evidence in policy development.     

Contextual influence on evaluation capacity building in a rapidly changing environment under new governmental policies

Evaluation capacity building (ECB) is a context-dependent process. Contextual factors affecting ECB implementation have been explored theoretically and practically, but their influence within a changing environment has seldom been discussed. This study examined essential context-sensitive parameters, particularly those involved in implementing new governmental policies regarding higher education. Taiwan was used as a case study for exploring the effect of contextual change on ECB attributes from the perspectives of training receivers and providers. Surveys and interviews were used for data collection and importance–performance analysis was applied for data analysis. Four prominent features were identified. First, the ECB attributes perceived as important by receivers were performed adequately, whereas those perceived as less important were performed less well. Second, under new policies, training provider designed training covering a wide range of ECB, whereas receivers focused on those can be directly applied in evaluation process. Third, in a small education system such as Taiwan’s, the complexity of peer review is high and ethical issues become important. Fourth, because the evaluation structure has been changed from single- to dual-track, receivers expect more training for institution staff, whereas providers insist on hierarchical training. Aligning ECB supply and needs is paramount for adaptation to new policies.     

Valuing and embracing complexity: How an understanding of complex interventions needs to shape our evaluation capacities building initiatives

This paper describes some of the main challenges of evaluating complex interventions, as well as the implications of such challenges for evaluation capacity building. It discusses lessons learned from a case study of an evaluation of Dancing with Parkinson’s, an organization that provides dance classes to people with Parkinson’s disease in Toronto, Canada. These implications are developed from a realist evaluation lens. Key lessons include the need to develop skills to understand program mechanisms and contexts, recognize multiple models of causality, apply mixed method designs, and ensure the successful scaling up and spread of an intervention.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Case study of an evaluation coaching model: Exploring the role of the evaluator

This study examined the role of the external evaluator as a coach. More specifically, using an evaluative inquiry framework (Preskill and Torres, 1999a, Preskill and Torres, 1999b), it explored the types of coaching that an evaluator employed to promote individual, team and organizational learning. The study demonstrated that evaluation coaching provided a viable means for an organization with a limited budget to conduct evaluations through support of a coach. It also demonstrated how the coaching processes supported the development of evaluation capacity within the organization. By examining coaching models outside of the field of evaluation, this study identified two forms of coaching — results coaching and developmental coaching — that promoted evaluation capacity building and have not been previously discussed in the evaluation literature.     

Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey

The demand for improved quality of health promotion evaluation and greater capacity to undertake evaluation is growing, yet evidence of the challenges and facilitators to evaluation practice within the health promotion field is lacking. A limited number of evaluation capacity measurement instruments have been validated in government or non-government organisations (NGO), however there is no instrument designed for health promotion organisations. This study aimed to develop and validate an Evaluation Practice Analysis Survey (EPAS) to examine evaluation practices in health promotion organisations. Qualitative interviews, existing frameworks and instruments informed the survey development. Health promotion practitioners from government agencies and NGOs completed the survey (n = 169). Principal components analysis was used to determine scale structure and Cronbach’s α used to estimate internal reliability. Logistic regression was conducted to assess predictive validity of selected EPAS scale. The final survey instrument included 25 scales (125 items). The EPAS demonstrated good internal reliability (α > 0.7) for 23 scales. Dedicated resources and time for evaluation, leadership, organisational culture and internal support for evaluation showed promising predictive validity. The EPAS can be used to describe elements of evaluation capacity at the individual, organisational and system levels and to guide initiatives to improve evaluation practice in health promotion organisations.     

Empowerment evaluation: An approach that has literally altered the landscape of evaluation

The quest for credible and actionable evidence to improve decision making, foster improvement, enhance self-determination, and promote social betterment is now a global phenomenon. Evaluation theorists and practitioners alike have responded to and overcome the challenges that limited the effectiveness and usefulness of traditional evaluation approaches primarily focused on seeking rigorous scientific knowledge about social programs and policies. No modern evaluation approach has received a more robust welcome from stakeholders across the globe than empowerment evaluation.Empowerment evaluation has been a leader in the development of stakeholder involvement approaches to evaluation, setting a high bar. In addition, empowerment evaluation’s respect for community knowledge and commitment to the people’s right to build their own evaluation capacity has influenced the evaluation mainstream, particularly concerning evaluation capacity building. Empowerment evaluation’s most significant contributions to the field have been to improving evaluation use and knowledge utilization.     

Assessing organizational capacity for diffusion: A school-based social network analysis case study

The purposeful design of social networks is increasingly recognized as a fundamental organizational improvement strategy. In the PK-12 education sector, school-based teacher collaboration is the primary vehicle through which educators are able to gain access to essential social capital, and through which leaders promulgate diffusion of innovation and continuous organizational learning. In partnership with school administrators, the authors undertook an evaluation to examine the size, structure, and composition of school-based networks. Social network analysis (SNA) was used to measure and visualize connections (or lack thereof) of ties between teams and between educators. Isolate and disconnected network actors were revealed through visual inspection of the sociograms. Administrators used findings to reconfigure team membership to enhance teacher ability to give and receive support and collaboratively problem-solve, and to ensure greater capacity for diffusion of instructional innovation and organizational learning. This paper contributes to the field’s understanding of how evaluators and organizational leaders can use SNA to measure, visualize, and more purposefully design effective patterns of connection between people through which professional knowledge, support, and innovation will travel.     

Enhancing Employee Engagement via Leaders’ Motivational Language in times of crisis: Perspectives from the COVID-19 outbreak

By bridging theoretical perspectives from diverse disciplines including public relations, organizational communication, psychology, and management, this study advances a sequential mediation process model that links leaders’ motivational communication—specifically, direction-giving, empathetic, and meaning-making language—to employees’ organizational engagement during times of crisis. The model incorporates employees’ psychological needs satisfaction and their subsequent crisis coping strategies so as to explain the process that underlies the effects of leader communication on employee engagement. We tested the model in a unique yet underexplored crisis context: organizational crises triggered by the global pandemic of COVID-19. The results of an online survey of 490 full-time U.S. employees provide strong support to the model’s predictions. Our research extends internal crisis communication scholarship in public relations by addressing what types of leader communication strategies as well as how these strategies contribute to employee engagement in a holistic fashion. It also advances theoretical development of the motivational language theory, self-determination theory, transactional model of stress and coping, and organizational engagement—the four theoretical bases of our study—in the context of organizational crises. Lastly, the study results provide timely practical insights on effective internal crisis communication.     

Reflections on the application of the Delphi method: lessons from a case in public transport research

The paper outlines the author’s experience employing the Delphi method, using as an example a particular application of the Delphi in the field of public transport research. Attention is given to aspects such as the choice of method, selection of experts, design of questionnaires, interaction between survey coordinator and participants, and also the analysis of experts’ responses. Some of the challenges encountered during the survey, the way they were dealt with, and risk mitigation strategies used by the Delphi coordinator are highlighted too. The primary objective of this paper is to offer insights that can support other researchers or practitioners preparing to apply the Delphi methodology. Furthermore, the article contributes to the methodological debate by reflecting on the introduction of novel practices that can help overcome some typical pitfalls of the Delphi: a dedicated blog supporting the survey, safety-net questions, and a constant-sum type question.     

Evaluating health services research capacity building programs: Implications for health services and human resource development

Evaluation of research capacity building programs presents challenges, including defining “research capacity building” and measuring it. The present study evaluates infrastructure capacity-building strategies and barriers and facilitators of two federally funded health-related research capacity building programs within the United States. We developed a conceptual framework and abstracted 223 data elements from existing project files. We administered questionnaires to 17 principal investigators and conducted 2-h interviews of 15 principal investigators. Finally, we conducted one-and-a-half-day site visits with five projects. We aggregated data and analysis results across the two programs. Impacts and strategies were also analyzed as to whether host institutions had little or no existing research infrastructure (“seed projects”) or had well-developed research infrastructure (“fertilizer projects”). Projects were associated with increases in publication and presentation rates by Principal Investigators and staff, increases in research funding, changes in organizational research culture, and positive impacts on the community. Project characteristics and activities associated with each of these outcomes were identified. It was also noted that the correlations between these outcomes were small and non-significant.     

Striking a Balance between Program-Specific and Portfolio-Level Evaluation: Lessons Learned from a Multi-Site Evaluation on the Texas-Mexico Border

Funders are increasingly making strategic investments across multiple grantees, aiming for their portfolio to improve targeted outcomes in a specific issue area. To this end, funders might use multi-site evaluation (MSE) approaches to examine the impact of their collective investments. However, it is important to recognize that each program—and its own program evaluation—must be tailored to its setting, population, and local context to best meet the needs of its target population. Therefore, multi-site evaluations need to account for this complexity. This paper describes the Sí Texas project, a large initiative of eight grantees implementing different integrated behavioral health models to improve physical and mental health outcomes along the Texas-Mexico border. With over 4,200 MSE study participants, the evaluation for Sí Texas used a partnership-centered approach to both enhance the evidence base and build local organizational capacity. This paper describes this approach, the process of tailoring evaluation practices to the grantees’ context, and the challenge of balancing consistency at the grantee-level for the portfolio multi-site evaluation. Successes, challenges, and lessons learned related to study design, data collection, grantee partnership, and capacity building are discussed.     

Collaborative research in child welfare: a rationale for rigorous participatory evaluation designs to promote sustained systems change

Expansion of the child welfare evidence base is a major challenge. The field must establish how organizational systems and practice techniques yield outcomes for children and families. Needed research must be grounded in practice and must engage practitioners and administrators via participatory evaluation. The extent to which successful practices are transferable is also challenged by the diversity of child welfare systems. The U.S. Department of Health and Human Services' Children's Bureau funded Quality Improvement Centers (QICs) that were designed to promote collaborative, multisite research that would address these evaluation needs. This article, based on the findings of a regional and a national QIC, describes the challenges facing research collaboration and the strategies for achieving success.