Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

Evaluation capacity building (ECB) interventions and the development of sustainable evaluation practice: An exploratory study

Evaluation capacity building (ECB) is a practice that can help organizations conduct and use evaluations; however, there is little research on the sustainable impact of ECB interventions. This study provides an empirical inquiry into how ECB develops sustained evaluation practice. Interviews were conducted with 15 organizational leaders from non-profits, higher education institutions, and foundations that “bought in” to ECB and were at least six months removed from an ECB contract. The result of this work highlights how sustained evaluation practice developed over time and what these practices looked like in real-world settings. A developmental, iterative cycle for how ECB led organizations to sustain evaluation practice emerged around key components to sustainability. First, leadership supported ECB work and resources were dedicated to evaluation. Staff began to conduct and use evaluation, which led to understanding the benefits of evaluation, and promoted value and buy-in to evaluation among staff. Common barriers and emerging sustainability supports not previously identified by ECB literature—the “personal” factor and ongoing ECB practitioner contact—are described. Practical tips for ECB practitioners to promote sustainability are also detailed.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

An assessment scale of ECB: perspectives from the faculty and staff of technical colleges in Taiwan

Faculty and staff in Taiwanese technical colleges are required to perform school self-evaluations. Most members of technical colleges come from postsecondary schools and are thus unfamiliar with school self-evaluation procedures. An effective school self-evaluation involves a complete ECB,(1) making it necessary to build faculty and staff evaluation capacity. In this research, we analyzed previous studies, interviewed experts, conducted expert reviews, constructed an ECB assessment scale, and collected the opinions of 1054 faculty and staff, to confirm the validity of the proposed assessment scale. This scale included 21 items summarized in four dimensions: understanding the evaluation guidelines, familiarizing with evaluation requirements, participating in the evaluation activity, and implementing the evaluation connotation. We explain the four dimensions behind the ECB model, and propose three lessons learned from it.     

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.     

Contextual influence on evaluation capacity building in a rapidly changing environment under new governmental policies

Evaluation capacity building (ECB) is a context-dependent process. Contextual factors affecting ECB implementation have been explored theoretically and practically, but their influence within a changing environment has seldom been discussed. This study examined essential context-sensitive parameters, particularly those involved in implementing new governmental policies regarding higher education. Taiwan was used as a case study for exploring the effect of contextual change on ECB attributes from the perspectives of training receivers and providers. Surveys and interviews were used for data collection and importance–performance analysis was applied for data analysis. Four prominent features were identified. First, the ECB attributes perceived as important by receivers were performed adequately, whereas those perceived as less important were performed less well. Second, under new policies, training provider designed training covering a wide range of ECB, whereas receivers focused on those can be directly applied in evaluation process. Third, in a small education system such as Taiwan’s, the complexity of peer review is high and ethical issues become important. Fourth, because the evaluation structure has been changed from single- to dual-track, receivers expect more training for institution staff, whereas providers insist on hierarchical training. Aligning ECB supply and needs is paramount for adaptation to new policies.     

Multidisciplinary eHealth survey evaluation methods

This paper describes the development process of an evaluation framework for describing and comparing web survey tools. We believe that this approach will help shape the design, development, deployment, and evaluation of population-based health interventions. A conceptual framework for describing and evaluating web survey systems will enable the developers of health interventions to identify, select, and use the most appropriate and relevant survey tools. The primary contributors included survey methodologists, computer scientists, informaticians, technical communications specialists, educators, evaluators, clinical practitioners, and health services researchers, who together developed a comprehensive ontology to describe the electronic survey domain. This paper will describe the survey ontology development process and detail lessons learned in the creation of the web survey system evaluation framework.     

Participatory shaping of community futures in e‐waste processing hubs: Complexity,conflict and stewarded convergence in a Palestinian context

This article describes an unconventional participatory development approach carried out in an informal e‐waste hub in South‐West Hebron, an area that has collected and processed the bulk of Israeli e‐waste for over a decade. Our approach contributes to the critique and recovery of community representation in participatory development and the search for ways to facilitate representative community engagement. Specifically, we describe our use of a novel Delphi‐like method that allowed us to facilitate a broadly endorsed development trajectory within a heterogeneous and conflicted community. We show how the results yielded by this approach diverged from those that were likely to emerge from more facile forms of participation in ways that are important for other similar e‐waste hubs internationally, which face a destructive status quo on the one hand, or the economically ruinous international policies that ban e‐waste trade from “developed” to “developing” countries on the other. Despite real tensions and cleavages within the affected communities, the process described facilitated a shift from deadlocked environmental versus livelihood positions towards building capacity and regulating existing informal e‐waste trades to preserve livelihoods dependent on these.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems--a case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors. This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring. An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult. From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems—A case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors.This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring.An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult.From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

Strategic Thinking about Employee Communication Behavior (ECB) in Public Relations: Testing the Models of Megaphoning and Scouting Effects in Korea

In public relations, employees are regarded as one of the most important strategic constituencies because they interact with external publics on a daily basis. However, employees' communication behavior (ECB) has not been extensively researched in public relations. The purpose of this research was to conceptualize and develop concrete measurements of ECB that can help theoretical development and strategic planning in public relations. In particular, the study identified two unique aspects of ECB—megaphoning and scouting—that refer to employees' voluntary efforts to collect and circulate strategic information externally and internally. In addition, a new concept, microboundary spanning, based on the two aspects of ECB is conceptualized. In this study, symmetrical internal communication and employee-organization relationships were posited as antecedent variables to the proposed ECB variables. A survey research was conducted with 300 employees in different types of organizations in Korea.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Workforce training in family therapy and systemic practices: An evaluation framework and case study

Internationally, while the interdisciplinary field of family therapy and systemic practices is well established and evidence for therapeutic impact is advanced, evaluation of training and measurement of its impact lags behind. This paper addresses this gap, utilising a single case study at the Bouverie Centre, the largest family therapy and workforce development service in Australia. We describe (a) the scope of workforce training and implementation at the Centre, (b) the development of our training research and evaluation framework, and (c) standardised data collection methodology, including a pre-post training reflective survey. Collectively, this work comprises our current evaluation framework through which we will subsequently examine training impact. Future research and practice recommendations are offered to address challenges to workforce training evaluation and progress efficacy of family therapy training.     

Using developmental evaluation and community-based participatory research to develop a model of supportive housing

Community-based participatory research (CBPR) and developmental evaluation (DE) have emerged over recent decades as separate approaches for addressing complex social issues. Current literature offers little with respect to the use of CBPR and DE in combination, although the two approaches are complementary. Through the current paper, we outline how CBPR and DE were used to develop a model of supportive housing for teen families. More specifically, we describe the structures and processes that contributed to this development, including (1) our partnership approach, (2) pooled resources, (3) regular opportunities for collaboration and reflection, (4) integration of multiple data sources, (5) ongoing feedback and knowledge dissemination, and (6) adjustments to program practices. We end by providing insights into the lessons that we learned through this project. Through this paper, we describe how researchers and community partners can collaboratively use CBPR and DE to develop a program model in complex community settings. Insights are offered that will be important for researchers, evaluators, and practitioners seeking to develop programming in response to complex community issues.     

Ownership claims,valuation practices,and the unpacking of energy-landscape conflicts

There is now a substantive body of academic literature which focuses on protests against local infrastructural developments. This literature is often characterised by the key words ‘NIMBY’ or (facility-) ‘siting controversies’. The rapid development of renewable energy technologies – which are largely sited in rural areas – has created a new version of this controversy; energy-landscape conflicts. In many countries, large infrastructural developments are regulated through spatial planning legislation, often causing various tensions between new technologies, an evolving policy agenda, and a legislative framework which was largely conceived in a different era and which is slow to adapt. Alternatively, and in line with neo-liberal thinking, the logic of development can be subjected to cost-benefit analysis, whereby the value of the wind farm can be compared with the value of the ‘unspoiled’ landscape. This paper takes a more holistic approach to energy-landscape conflicts, by examining claims of ownership and notions and measures of value inherent in different claims and value systems which (seek to) influence decision-making. We examine both the logic of monetary valuation and the implicit value statements in various policy intervention options to point at the need for a more heterogeneous and multidisciplinary approach to policy evaluation. We then look at notions of ownership, rights and duties in relation to landscape and to our energy future, and we highlight the potential for using an analytical property rights framework which cuts across various levels of claims and value statements, from the national and ideological to the personal and practice-based.     

Theory of change models deepen online learning evaluation

Massive Open Online Courses (MOOCs) have gained traction as resources for professional development. This article presents the method that we used to evaluate a professional development MOOC for postdoctoral trainees that was created by a university consortium in the US. Most approaches to evaluating MOOCs focus on analysis of participation, outcomes from course assignments, self-reported learning outcomes, course completion and user pathways through the online content or clickstream data. Few published evaluations describe in detail how learning happens within online courses and the anticipated medium and longer term cognitive and behavioral outcomes on participants. This work aims to guide those who are designing, implementing and evaluating MOOCs through applying theories of change to focus evaluation on the process of learning. This approach can be used as a complement to traditional approaches for evaluating MOOCs. We described how we worked with the MOOC team building the content to develop a theory of change for each module (or lesson) within the MOOC and used the theory of change to guide evaluation of short and medium term participant cognitive, affective and attitudinal, and behavioral outcomes. Finally, we share lessons learned and suggestions for implementing theories of change in both the design and evaluation phases of MOOC development.