How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.     

Development of a fostering purpose intervention

Evolutionary Evaluation is a framework for understanding the evolution of programs. Just as programs grow and develop, so can standalone interventions. This paper focused on building and evaluating a purpose intervention to scaffold adolescents and young adults’ search for purpose. Four studies were conducted at each phase of the intervention’s lifespan to assess if modifications were needed and to test the viable validity of using online activities to increase both the search for and the discovery of a purpose in life. These studies can be viewed through the lens of the Evolutionary Evaluation framework. Study 1 evaluated the intervention at its initiation phase, which consisted of developing and testing the intervention activities. Study 2 tested the intervention during the development phase to determine which activities most effectively increased rates of purpose. Study 3 evaluated the intervention in its stability phase by experimentally testing all activities together in a full curriculum via Amazon’s Mechanical Turk. Finally, study 4 assessed the dissemination of the intervention by testing the full curriculum with over 500 high school and college students. These studies suggest that the fostering purpose intervention cultivated purpose among adolescents and young adults. Additionally, the success of the intervention demonstrates that the Evolutionary Evaluation framework may be an effective model with which to develop an intervention, from initiation through dissemination phases.     

Will evolutionary psychology become extinct? Evolutionary psychology as the Leaning Tower of Pisa

ABSTRACT

Evolutionary psychology has as its foundation the classical Darwinian-Wallace theory of evolution. Using this theory as a guideline, evolutionary psychologists have interpreted human behaviors from an adaptationist outlook using a circular logic wherein no matter what the behavior is being looked at it is always given an evolutionary rationale. Additionally, there is much speculation as to evolutionary effects in the past and these speculations are seen as factual. Furthermore, there is evidence that the classical theory is flawed in that the emphasis on adaptation as the basis for evolution is incorrect. In that there is no actual evidence that Natural Selection (i.e., adaptation) has led to speciation. Previous scientists who have criticized the classical theory are cited, who made the case for speciation having occurred through a sudden “saltationist” process rather than the classical gradual process. As such, the author predicts that evolutionary psychology may ultimately become extinct as a subdiscipline.     

The theory of change of the evaluation support program: Enhancing the role of community organizations in providing an ecology of care for neurological disorders

This paper discusses the Ontario Brain Institute’s theory of change for the Evaluation Support Program, a program designed to enhance the role of community organizations in providing care and services for people living with a brain disorder. This is done by helping community organizations build evaluation capacity and foster the use of evidence to inform their activities and services. Helping organizations to build capacities to track the ‘key ingredients’ of their successes will help ensure that successes are replicated and services can be improved to maximize the benefit that people receive from them. This paper describes the hypothesized outcomes and early impacts of the Evaluation Support Program, as well as how the program will contribute to the field of evaluation capacity building.     

Mentoring Policy Research Organizations: Project Evaluation Results

The international community has identified a positive link between good governance and economic development. There is an increasing appreciation of the effective role that local policy research organizations (PROs) in transition and developing countries can play in providing evidence-based policy recommendations as the basis for sound legislation and in assessing the efficacy of existing programs. Capacity building programs for PROs are seen as a key component of growth strategies. The donor community has responded with support for such efforts, which take many forms. On the other hand, the evaluation of the effectiveness of these capacity building and mentoring programs has been inadequate. This article presents the evaluation of one program, focusing on two important capacity building program outcomes: improved research capacity and greater effectiveness in the policy arena. A reflexive design is used. The results suggest the program??s approach is promising.     

Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.     

UPSTREAM! Together evaluation results from community efforts to prevent mental,emotional, and behavioral health problems

Mental, emotional, and behavioral (MEB) health problems are prevalent globally. Despite effective programs that can prevent MEB problems and promote mental health, there has not been widespread adoption. UPSTREAM! Together was a planning project in three Colorado communities. Communities partnered with academic and policy entities to 1) translate evidence about MEB problem prevention into locally-relevant messages and materials and 2) develop long-term plans for broad implementation of interventions to prevent high-priority MEB problems. Community members recognized the need to talk about MEB problems to prevent them. The UPSTREAM! communities localized messages designed to start conversations and sustain attention on preventing MEB problems. The communities understood that prevention takes sustained community attention and advocacy, knowing that important outcomes may be years away. Long-term implementation plans aimed to strengthen families and enhance social connections among youth. Despite community readiness and capacity to implement evidence-based programs, there were few funding opportunities, delaying program implementation and revealing gaps between funding policies and community readiness. This community-engaged experience suggests an achievable approach, acceptable to communities, and worthy of further development and testing. Policies that cultivate and support local expertise may help to increase wider community adoption of evidence-based programs that promote mental health among youth.     

Value for money: A utilization-focused approach to extending the foundation and contribution of economic evaluation

Value for Money (VfM) is an evaluative question about the merit, worth, and significance of resource use in social programs. Although VfM is a critical component of evidence-based programming, it is often overlooked or avoided by evaluators and decision-makers. A framework for evaluating VfM across the dimensions of economy, effectiveness, efficiency, and equity has emerged in response to limitations of traditional economic evaluation. This framework for assessing VfM integrates methods for engaging stakeholders in evaluative thinking to increase acceptance and utilization of evaluations that address questions of resource use. In this review, we synthesize literature on the VfM framework and position it within a broader theory of Utilization-Focused Evaluation (UFE). We then examine mechanisms through which the VfM framework may contribute to increased evaluation use. Finally, we outline avenues for future research on VfM evaluation.     

Evolutionary Theory and Neuroscience: An Explanatory Theory for Social Work

Evolutionary theory and neuroscience are recommended as a foundational theory for social work. Advantages of evolutionary theory include explanatory power, suggestion of interventions to beginning social work practitioners, and accommodation of more specific theories of human behavior. The explanatory power of neuroscience is also discussed, along with its evidence base and compatibility with the strengths perspective and destigmatization work of the profession. Connections between evolutionary theory, neuroscience, and social work values are also discussed.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).     

Tailored program evaluation: Past, present, future

This paper discusses measurement issues related to the evaluation of computer-tailored health behavior change programs. As the first generation of commercially available tailored products is utilized in health promotion programming, programmers and researchers are becoming aware of the unique challenges that the evaluation of these programs presents. A project is presented that used an online tailored health behavior assessment (HBA) in a worksite setting. Process and outcome evaluation methods are described and include the challenges faced, and strategies proposed and implemented, for meeting them. Implications for future research in tailored program development, implementation, and evaluation are also discussed.     

Comparing rating paradigms for evidence-based program registers in behavioral health: Evidentiary criteria and implications for assessing programs

Decision makers need timely and credible information about the effectiveness of behavioral health interventions. Online evidence-based program registers (EBPRs) have been developed to address this need. However, the methods by which these registers determine programs and practices as being “evidence-based” has not been investigated in detail. This paper examines the evidentiary criteria EBPRs use to rate programs and the implications for how different registers rate the same programs. Although the registers tend to employ a standard Campbellian hierarchy of evidence to assess evaluation results, there is also considerable disagreement among the registers about what constitutes an adequate research design and sufficient data for designating a program as evidence-based. Additionally, differences exist in how registers report findings of “no effect,” which may deprive users of important information. Of all programs on the 15 registers that rate individual programs, 79% appear on only one register. Among a random sample of 100 programs rated by more than one register, 42% were inconsistently rated by the multiple registers to some degree.     

Evaluating development programs: building joint activity

There has been an increasing plea for a participatory approach to development programs. It is argued that for programs to be sustainable, they need to be conceptualized, negotiated and run by all stakeholders jointly. Concurrent with this trend, there has been an increasing demand for program evaluation strategies, which encompass an evaluation of the nature and level of participation of the different stakeholders in the program. This paper proposes a theoretical framework for such evaluation and subsequently provides a practical illustration. Engeström's notions of ‘activity system’ and ‘learning by expanding’, are proposed as a framework for the evaluation of participation and an action evaluation methodology is suggested as the practical tool within such framework. The paper presents a case study of a conservation education program in Africa, which operationalizes the proposed theoretical framework.     

“They give you back that dignity”: understanding the intangible resources that make a transitional house a home for homeless families

Approximately 33% of the homeless population in 2017 were families with children. Researchers have argued for interventions and services to employ an ecological perspective to mitigate some of the unintentional harm inherent in existing programs designed to serve homeless families. This paper utilized a case study of a transitional housing program to (1) determine how well the program met recommendations informed by the ecological perspective and (2) explore underlying themes across recommendations. Past and current residents of a transitional housing program participated in semi-structured interviews as part of a program evaluation. Interviews were deductively and inductively coded to meet aims 1 and 2, respectively. Residents identified several aspects of the program that met the ecological perspective recommendations including culturally-responsive services, trauma-based services, and social and community connections. Major underlying themes spanning across recommendations included interpersonal resources (caring, sense of belonging), empowerment resources (advocacy, family preservation), impact on family (family life, positive youth development), and communication. This case study, through the analysis of residents’ perspectives, highlights the important role of transitional housing programs in providing a supportive, empathetic, empowering, and home-like environment that gives residents “back that dignity” so they are emotionally and physically prepared to successfully transition to permanent housing.     

A Brief History of the Theory of Evolution

Abstract

A brief history of evolutionary theory illustrates its long development and sociopolitical context. Major theoretical concepts in evolutionary theory are introduced that lay the groundwork for more detailed discussion in later texts. Two incarnations of evolutionary theory—sociobiology and evolutionary psychology—are described in some detail. Criticisms of these ideas are reviewed and evaluated. Evolutionary psychology, with its emphasis on sexual selection theory, has proven to be a rich approach, generating many new insights into human sexuality.     

The whole elephant: Defining evaluation

Definitions help us understand the characteristics of an object or phenomenon and are a necessary precursor to understanding what a good version of it looks like. Evaluation as a field has resisted a common definition (Crane, 1988; Morell & Flaherty, 1978; M. F. Smith, 1999), which has implications for marketing, training, practice, and quality assurance. In this position paper, I describe the benefits and challenges of not having a clear, agreed-upon definition, then propose and explore the implications of two definitions for the evaluation profession based on values and valuation as the core of evaluation practice. The purpose is to describe a possible way forward through definition that would increase our professional profile, power, and contribution to social justice. The paper concludes with implications for evaluator competencies and evaluation education and questions for further research.     

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.