How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.     

Where do we stand? Recent AEA member views on professionalization

The professionalization of evaluation continues to be debated at numerous conferences in the U.S. and abroad. At this time, AEA member views on the potential benefits and negative side effects of professionalization are essential as the discussion evolves. This study provides recent views on major topics in professionalization, including potential benefits, negative side effects, processes, competencies, and procedures. Results from in-depth interviews and an online survey demonstrate that AEA members view potential benefits of professionalization to be stakeholder trust, evaluator reputation and identity, while concerns about a potential negative side effect known as the “narrowing effect” (i.e., some evaluators will be alienated based on their background, competencies, etc.) were expressed by participants. These recent findings can inform the ongoing discussion of professionalization, and suggest new directions for future research on evaluation.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

基于神经网络的测井评价油层污染研究

利用测井资料,在剖析传统的测井评价油(气)层污染方法的基础上,介绍了利用BP神经网络这一现代数理统计新技术来实现测井多参数评定油气层污染程度的方法和技巧;以华北二连油田实际资料为例,建立了测井判释油气层污染的数学物理模型,并进行了模拟预测。结果表明:该法简便可行,对污染评价参数预测精度较高,对污染程度评价合理准确,可作为油(气)层污     

Multiple imputation for missing edge data: A predictive evaluation method with application to Add Health

Recent developments have made model-based imputation of network data feasible in principle, but the extant literature provides few practical examples of its use. In this paper, we consider 14 schools from the widely used In-School Survey of Add Health (Harris et al., 2009), applying an ERGM-based estimation and simulation approach to impute the network missing data for each school. Add Health's complex study design leads to multiple types of missingness, and we introduce practical techniques for handing each. We also develop a cross-validation based method – Held-Out Predictive Evaluation (HOPE) – for assessing this approach. Our results suggest that ERGM-based imputation of edge variables is a viable approach to the analysis of complex studies such as Add Health, provided that care is used in understanding and accounting for the study design.     

Experiences with systematic triangulation at the Global Environment Facility

Systematic triangulation may address common challenges in evaluation, such as the scarcity or unreliability of data, or the complexities of comparing and cross-checking evidence from diverse disciplines. Used to identify key evaluation findings, its application has proven to be effective in addressing the limitations encountered in country-level evaluation analysis conducted by the Independent Evaluation Office of the Global Environment Facility (GEF). These include the scarcity or unreliability of national statistics on environmental indicators and data series, especially in Least Developed Countries; challenges in evaluating the impacts of GEF projects; and inherent difficulties in defining the GEF portfolio of projects prior to the undertaking of the evaluation. In addition to responding to the need for further developing triangulation protocols, procedures and/or methodologies advocated by some authors, the approach offers a contribution to evaluation practice. This applies particularly to those evaluation units tasked with country-level evaluations in international organizations, facing similar constraints.     

An evaluation capacity building toolkit for principal investigators of undergraduate research experiences: A demonstration of transforming theory into practice

This paper describes the approach and process undertaken to develop evaluation capacity among the leaders of a federally funded undergraduate research program. An evaluation toolkit was developed for Computer and Information Sciences and Engineering¹ Research Experiences for Undergraduates² (CISE REU) programs to address the ongoing need for evaluation capacity among principal investigators who manage program evaluation. The toolkit was the result of collaboration within the CISE REU community with the purpose being to provide targeted instructional resources and tools for quality program evaluation. Challenges were to balance the desire for standardized assessment with the responsibility to account for individual program contexts. Toolkit contents included instructional materials about evaluation practice, a standardized applicant management tool, and a modulated outcomes measure. Resulting benefits from toolkit deployment were having cost effective, sustainable evaluation tools, a community evaluation forum, and aggregate measurement of key program outcomes for the national program. Lessons learned included the imperative of understanding the evaluation context, engaging stakeholders, and building stakeholder trust. Results from project measures are presented along with a discussion of guidelines for facilitating evaluation capacity building that will serve a variety of contexts.     

Goal attainability and performance: Evidence from Boston marathon qualifying standards

In this paper we test if performance improves once goals become more attainable. Goal-setting literature suggests that workers respond to challenging but achievable goals with increased performance. Empirical evidence supports the notion of goals increasing performance; however the evidence on how attainability of goals affects performance is mixed. This paper tests whether efforts increase, improving performance as the goals become more attainable. We are employing a unique set of publicly available marathon data from 1970 to 2015 to directly analyze the effect of goal attainability on performance. With the probable goal of qualifying for the Boston Marathon, we test if runners increase their effort, and consequently improve their performance if they enter a new age group and as a result have a more attainable goal. We find that runners who enter a new age group perform better than the runners whose qualifying time did not change. This effect is seen with runners in younger age groups, but not found in the results of runners in more advanced years.