How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.     

Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.     

Using a logic model to evaluate the Kids Together early education inclusion program for children with disabilities and additional needs

Despite clear evidence that learning and social opportunities for children with disabilities and special needs are more effective in inclusive not segregated settings, there are few known effective inclusion programs available to children with disabilities, their families or teachers in the early years within Australia. The Kids Together program was developed to support children with disabilities/additional needs aged 0–8 years attending mainstream early learning environments. Using a key worker transdisciplinary team model, the program aligns with the individualised package approach of the National Disability Insurance Scheme (NDIS).AimThis paper reports on the use of a logic model to underpin the process, outcomes and impact evaluation of the Kids Together program.MethodsThe research team worked across 15 Early Childhood Education and Care (ECEC) centres and in home and community settings. A realist evaluation using mixed methods was undertaken to understand what works, for whom and in what contexts. The development of a logic model provided a structured way to explore how the program was implemented and achieved short, medium and long term outcomes within a complex community setting.Discussion and conclusionKids Together was shown to be a highly effective and innovative model for supporting the inclusion of children with disabilities/additional needs in a range of environments central for early childhood learning and development. The use of a logic model provided a visual representation of the Kids Together model and its component parts and enabled a theory of change to be inferred, showing how a coordinated and collaborative approached can work across multiple environments.     

Explaining how unexploded ordnance clearance enhances livelihoods in the Lao PDR

BackgroundFollowing violent conflict, the continued presence of landmines and unexploded ordnance pose a barrier to rebuilding livelihoods. Mine action removes these explosive remnants of conflict to enable communities to safely return contaminated land to productive use. There is limited understanding, however, of how, why, in what context and in what respects mine action contributes to livelihoods. Yet, such information is required for effective resource allocation, checking underlying program assumptions, understanding benefits and potential harms.MethodsThe evaluation was undertaken in the Lao People's Democratic Republic. It used an interpretive case study design and applied the principles of realist evaluation. Program staff and local government authorities were interviewed (N = 37) and program beneficiaries. In total, 38 individual interviews with program beneficiaries were conducted and eighteen focus group interviews (9 with males, 9 with females), each with 6–9 participants.ResultsThe evaluation identified two main mechanisms through which the program ‘worked’: (1) communication pre- and post-clearance and (2) the delivery of the product (cleared land).ConclusionThe realist approach helped to refine the program theory, highlighted the role of self- and task-efficacy and community communication, assisted in identifying contextual factors that influence outcomes and suggested a revision of expected outcomes.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Using network analysis to conduct a system-wide program evaluation within a university

ObjectiveTo conduct a system-wide assessment using social network analysis (SNA) to examine how 14 important issues (e.g., consent; racism) are addressed through education, training, and programming at a university.MethodsEvaluators conducted interviews with campus departments responsible for educating/training on the 14 issues. Interviews revealed which programs (n = 52) were offered that addressed the 14 issues, and data on audience characteristics, date of delivery, and which issues were covered in each program were collected. SNA was used to calculate degree and create visualization graphs illustrating patterns of content-coverage across all 52 programs.ResultsThe average degree was 19.38 (SD = 9.70), meaning programs overlapped in topic area with nearly 20 other programs, on average. Most programs (n = 36; 69.2 %) were attended by audiences of 500 people or less. “Diversity and inclusion” represented the topic area with the most programs (n = 23), whereas “suicide” and “bullying/hazing” had the least number of programs (n = 3). Degree was negatively correlated with attendance numbers (r=-.310, p < .001), indicating the more a program overlapped in content with other programs, the smaller the audience.ConclusionsThis study supports the use of network analysis in conducting systemic evaluations of programs offered at a university, complementing the work of ongoing, local-level program evaluations.     

A walk on the wild side: Urban ethnography meets the Flâneur

Abstract

This paper focuses on the concept of the flâneur, deriving largely from the works of Baudelaire and Walter Benjamin, and attempts to reveal its contemporary relevance for sociological practice. The flâneur is treated as an instructive metaphor for the sociologist's relationship with modernity and urban life, and therefore as providing insight into the social, historical and theoretical contexts for the analysis of the world today. More than this, the idea of the flâneur is treated as highly instructive of research strategies confronting urban life, namely ethnography. The paper seeks to expose a critical space between ‘realist’ and ‘textual’ ethnographies and provide a locus for the discussion of associated methodological problems. A burgeoning literature and tradition of work in this area is examined which will also be of interest to the sociological theorist.     

Quasi-experimental evaluation using confirmatory procedures: A case study of an economic and social empowerment intervention to reduce girls’ vulnerability to HIV in rural Mozambique

PurposeOur goal was to determine whether an economic and social empowerment intervention implemented in Zambézia Province, Mozambique reduced girls’ vulnerability to HIV. We use this experience to discuss challenges of evaluating real-world interventions.MethodsTwo rounds of data were collected from 885 girls, 13–19 years, for this clustered, non-equivalent (two-stage) cohort trial. We used multi-level exact matching and difference-in-differences estimation to estimate intervention effects on two outcomes: girls’ knowledge of gender-based violence and school attendance.ResultsEstimates of two outcomes analysed indicated no statistically significant intervention effects. Preliminary analysis of data from the intervention group revealed this study was unable to obtain accurate measures for five outcomes related to HIV vulnerability.ConclusionsAlthough our study did not find evidence of impact on the a priori selected outcomes, we report on our experience implementing this robust methodologic design and describe how the challenges encountered in this program setting affected our ability to attain results. We recommend prospective evaluation designs with random allocation be accommodated early during planning. When not possible, quasi-experimental studies should collect data from large samples. To reduce measurement bias, biological endpoints such sexually transmitted infections should serve as primary outcomes for programs intending to reduce sexual behaviors.     

Understanding innovation in health program planning and development

IntroductionA need for innovative public health programs is evident as the field adapts to address changes in health priorities and target populations. The Innovative Teen Pregnancy Prevention Program (iTP₃) was created to support and enable innovation in teenage pregnancy prevention, developing programs to reach the most at risk youth.MethodsA formative evaluation was conducted to understand what innovation means in the context of program development, and examine the process of innovation. Qualitative data was collected through baseline interviews with program development teams, referred to as Innovators, prior to the start of the project period and follow-up interviews conducted at the end of a 12-month funding period. Additional open-ended written responses were collected in the middle of the funding cycle. A thematic analysis with an open-coding scheme was used to identify emergent themes.ResultsInnovators considered programs innovative because of the target population of focus, program delivery mechanism, and/or program development approach. They specifically identified that a “culture” of innovation must be present if new programs are to be developed. Over time, Innovators began to shift their definition of innovation toward unique design processes and ecological approaches.DiscussionThrough creating a culture of innovation and utilizing systems thinking, this project provides important insights in how to develop innovations in public health.     

Applying RE-AIM to evaluate two community-based programs designed to improve access to eye care for those at high-risk for glaucoma

IntroductionGlaucoma is a leading cause of vision loss and blindness in the U.S. Risk factors include African American race, older age, family history of glaucoma, and diabetes. This paper describes the evaluation of a mobile eye health and a telemedicine program designed to improve access to eye care among people at high-risk for glaucoma.MethodsThe RE-AIM (reach, efficacy, adoption, implementation, and maintenance) evaluation framework was used to harmonize indicators. Both programs provided community-based eye health education and eye services related to glaucoma detection and care. Each program reported data on participants and community partners. An external evaluator conducted site visit interviews with program staff and community partners. Quantitative and qualitative data were integrated and analyzed using the RE-AIM dimensions.DiscussionBy targeting high-risk populations and providing comprehensive eye exams, both programs detected a large proportion of new glaucoma-related cases (17–19%) – a much larger proportion than that found in the general population (<2%). The educational intervention increased glaucoma knowledge; evidence that it led people to seek eye care was inconclusive.ConclusionsEvaluation findings from the mobile eye health program and the telemedicine program may provide useful information for wider implementation in public health clinics and in optometrist clinics located in retail outlets.     

The Effectiveness of Faith-Based Welfare-to-Work Programs

Abstract

This essay first highlights three major challenges of measuring program effectiveness. It concludes effectiveness can best be conceptualized in terms of program outcomes and that such outcomes-while difficult to measure-can be measured with sufficient accuracy to justify the effort. These considerations are then illustrated by data gathered from 17 Los Angeles welfare-to-work programs and their clients in order to assess the comparative effectiveness of faith-based and secular programs. The essay concludes that no one type of welfare-to-work program was more effective across the board than any other type of program. Instead, the different types of programs seemed to be especially effective in certain specialized areas. The faith-based programs were especially effective in providing welfare recipients with emotional support and a sense of having a sympathetic, understanding base of support. The for-profit providers were especially effective in providing needed training in marketable job skills and help in finding employment. The essay concludes by discussing two public policy implications that flow from this type of program specialization.     

Social Work Admissions and Academic Accommodations for Students with Disabilities

ABSTRACT

The Americans With Disabilities Act of 1990 (ADA) protects students wth disabilities from discrimination by academic and professional programs in higher education. A student with a disability cannot be denied admission to an educational program because of his or her disability if the student is otherwise qualified. This means that a student with a disability who is qualified for an academic or professional educational program cannot be denied admission to a program based solely on the student&s disability. This educational requirement mandated by the ADA applies to all educational fields including social work education. The purpose of this study was to explore how social work programs are dealing with these requirements in their admission and academic accommodation procedures. Twelve social work programs located in the mid-western United States participated in the study. Representatives of each of these programs were given an in-depth interview focusing on their admissions process, academic accommodations, and general topics related to social work education and disabilities. All programs taking part in the research were accredited by the Council on Social Work Education (CSWE). The CSWE is the national accreditation body for social work education within the United States.     

“Where’s the human dignity in that?”: LGBTQIA + people with intellectual disability exploring sexual lives and respectful relationships

Abstract

For people with disabilities, sexuality remains a contested and restricted space. This research investigated how the Sexual Lives & Respectful Relationships (SL&RR) program might be adapted for people with intellectual disabilities who identify as LGBTQIA+. The program included co-developed materials and specifically curated narratives. It was received positively. The facilitators identifying as members of the LGBTQIA+ community was also reported as positive. While an inclusive program might be ideal, participants expressed the view that community attitudes (including those of other people with intellectual disability) might not yet be conducive to integrated human relationships programs.     

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.     

Evaluating Social Integration and Psychological Outcomes for Older Adults Enrolled at a University Intergenerational Program

Abstract

Intergenerational programs are proliferating. The basic assumption of such programs is that pairing older and younger people in various activities and contexts such as educational programs will produce positive developmental benefits for all participants. Educational programs appear to help adults to remain socially active and connected. The University of Valencia (Spain) initiated in 1999–2000 a “University Program for Seniors” for adults age 55 and over, entitled “Nau Gran.”The goal of this paper is to present baseline results on the benefits and impacts for older adults enrolled at the program during 2002. Results from a sample of 212 sophomore and junior older adult participants are presented. The questionnaire included socio-demographic information and items regarding health, social integration, and psychological outcomes. Results indicated that older adults attending courses at the University increased their level of social integration.     

On-Line,Computer-Based,Interactive Simulations: Bridging Classroom and Field

ABSTRACT

This paper briefly describes two educational tutorial programs, and the efforts of this author to evaluate the effectiveness of these computer mediated programs. One interactive program is designed to teach students how to apply crisis concepts to a practice simulation, and the second program is designed to teach students how to assess lethality with a suicidal client. Formal evaluations by students of these two programs and student performance on a follow-up quiz have produced positive results. A comparison study of classes who completed these programs and a class that did not have the training has also produced significant, favorable results.     

Medicaid State Plan Personal Care Services

Abstract

Policymakers face mounting pressures from consumer demand and the 1999 Olmstead Supreme Court decision to extend formal (paid) programs that deliver personal care to the elderly, chronically ill, and disabled. Despite this, very little is known about the largest program that delivers personal care: the Medicaid State Plan personal care services (PCS) optional benefit. This paper presents the latest available national program (participant and expenditure) trend data (1999–2002) on the Medicaid PCS benefit and findings from a national survey of eligibility and cost control policies in use on the program. The program trends show that, over the study period, the number of states providing the Medicaid PCS benefit grew by four (from 26 to 30), and national program participation, adjusted for population growth, increased by 27%. However, inflation-adjusted program expenditures per participant declined by 3% between 1999 and 2002. Findings from the policy survey reveal that between 1999 and 2002 there was a marked decline in the range of services provided, and by 2004, almost half the programs operated a cap on the hours of services provided.     

The Supportive Community

Abstract

This study examines supportive community programs in Israel, which aim to improve the quality of life of the elderly. These innovative programs pool existing resources to provide a benefits package that includes medical services, an emergency call switchboard, a “neighborhood facilitator,” and social activities. Data were collected in 2000-2001 using qualitative and quantitative methods. The program provides specific services to meet needs that otherwise are not adequately addressed. The major contributions of the program reported by the members was increasing their personal security (two-thirds), easing the burden on their children (one-third), and enabling them to remain at home (one-quarter). The supportive community program enriches the variety of services available, thus providing the elderly with the choice of staying within their familiar surroundings of their homes and neighborhoods. This model appears to be both a cost-effective way to facilitate aging in place and a way to meet many of the elderly's essential needs, thereby maintaining their quality of life.