Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

Design of a collaborative monitoring and evaluation system for a community-based nutrition project in rural Bangladesh

Poor diet and undernutrition are common among children living in Bangladesh. To promote appropriate complementary feeding of young children, an economic development (ED) program involving income-generating asset transfer was implemented alongside a social and behavior change (SBC) program. This paper introduces a collaborative monitoring and evaluation (M&E) system in which diverse collaborators (“research group”, “implementation team”, and “coordinators”) facilitate M&E data acquisition by leveraging their comparative advantages. The implementation team built a monitoring system to track the ED (n = 2960) and SBC participants (n=∼10,000) over 12 months. Based on the baseline design and the monitoring records, the collaborators planned an impact evaluation introducing a quasi-experimental design using two cross-sectional surveys and a prospective cohort survey of child feeding and nutritional status. Using various data sources generated from the M&E system, the collaborators will also reveal the program impact pathway through which each intervention component is delivered, received, and utilized alongside the context-specific facilitators and barriers, including the programs’ uptake. The collaborative M&E system enables the sharing of program goals, strengthens collaborators' commitment to the program, and extends the understanding of the program's progress and evaluation activities.     

“We thought it would be easy”: recruitment challenges,successes and best practices for low-income couple populations

Recruitment challenges present ongoing struggles for social science professionals. Especially problematic are recruiting endeavors when target populations are highly stressed, highly mobile, and are comprised of couples rather than individuals. Included in this study is a rationale supporting the need to study recruitment challenges coupled with a literature review discussing recent recruitment research summaries. Based upon a Delphi-type discussion with couples relationship education (CRE) experts, our team constructed a qualitative study to examine recruitment challenges in low-income CRE programs. Fourteen interviews were conducted with “successful” program leaders identified by federal regional managers within the Administration for Children and Families (ACF). Data were analyzed, and results detailing four main “challenge” themes, four “success” themes, and a seven-step best practice model are presented.     

Learning from Experience in Family Support Programs

ABSTRACT

The paper describes three related California family support program initiatives representing field experiments in program development and policy related to reducing adverse outcomes for vulnerable families through replicating known “best practices” models. The ways in which the three initiatives were affected by the context of implementation are used to demonstrate the difficulty both of identifying the “best” program models and the need for leadership at all levels in order to develop programs and policies, which achieve the intended effects.     

Community-based behavioral health interventions: Developing strong community partnerships

The popularity of community-based interventions has experienced a revival over the last two decades. The general theme behind this trend is that greater community involvement provides researchers and practitioners with culturally relevant information to make interventions successful. This involvement is usually solicited through the formation of community coalitions and advisory boards. Although the positive influence of community partnerships is well established, strategies for successfully developing and sustaining these partnerships are less clear. To address this gap, this paper presents evaluation findings from The Making Connections Initiative, a national effort emphasizing community-level prevention strategies for improving the mental health of men/boys. Sixteen grantees were tasked with developing a coalition of community organizations to provide input into the development and implementation of a local prevention initiative. An evaluation of the 16 grantees’ planning activities identified four themes related to developing and sustaining successful community partnerships. This article reports on the strategies that were found most successful across grantees. With the increasing popularity of community-based programs, strategies for successfully partnering with local organizations are particularly important. This study adds to the limited research on best practices for forming community partnerships and should serve as a guide for community-based evaluators and program planners.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

“She Got Spoilt”: Perceptions of Victims of Child Sexual Abuse in Ghana

ABSTRACT

International research has commented on social stigma as a key reason for nondisclosure of child sexual abuse. However, the actual components of this social stigma frequently remain unexplored. The present study deals with perceptions of consequences of child sexual abuse among professionals and laypeople in Ghana (N = 44), employing a bystander perspective. As a qualitative study using a grounded theory framework, it considers these consequences in light of their underlying beliefs about child and adolescent development, particularly in relation to gender-based expectations placed on girls and boys. Consequences of child sexual abuse could be divided into sexual health consequences, beliefs about “destroyed innocence” and beliefs about a “destroyed future,” which were strongly related to the sexual nature of the violence perpetrated. These perceived consequences of child sexual abuse hold implications for what surviving child sexual abuse means on a social level. Implications for practice are discussed on the basis of the data analysis.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

Putting the “S” Back into Sexual and Reproductive Health

ABSTRACT

Sexuality is the most fundamental aspect of our lives and yet it has long been the silent partner in sexual and reproductive health (SRH). Growing awareness of the rights-based approach to health, the commitment of governments and international bodies to promote sexual health and well-being, and the failure of many of the interventions to combat the HIV/AIDS epidemic are helping the reproductive health community put the “S” back into SRH even as the Millennium Development Goals fail to address SRH directly. The disconnect is due to the fragmented, non-holistic approach to reproductive health, short-sightedness in neglecting violations like gender-based violence, a lack of training for students and health care professionals in sexuality, a dearth of indicators and sexuality research and discrimination against vulnerable groups like youth and sexually diverse populations. Recommendations include reaching consensus about the nexus between sexuality and reproductive health and strengthening the commitment to gender equity, empowerment, sexuality and sexual rights among policymakers. Ensuring that explicit and positive references to sexuality are universal, especially in comprehensive sexuality education, youth-focused programs and the integration of HIV and other SRH programs, and strengthening the ability of program managers and service providers to work on sexual health issues are key to successful integration and program sustainability. Strengthening understanding, acceptance and competence in serving sexually diverse populations and improving the research and evaluation specific to sexuality are longer processes, but equally important in the paradigm shift to non-stigmatizing, gender-equitable, rights-based, pleasurable sexual health services.     

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.     

Houston Homeless Street Outreach

Abstract

Several cities in the nation have developed innovative programs to assist the homeless who are “living on their streets.” This paper describes one such program developed in Houston, Texas. The Homeless Street Outreach was a collaborative project of several service agencies in the greater Houston area. The outcome of this outreach project indicates that a coordinated effort can bring about delivery of more comprehensive services to the homeless population. The principles that emerge from the outreach include involvement of the homeless in planning and development, service agencies taking “ownership” of the project, and agency flexibility in delivering services to the homeless.     

Where do we stand? Recent AEA member views on professionalization

The professionalization of evaluation continues to be debated at numerous conferences in the U.S. and abroad. At this time, AEA member views on the potential benefits and negative side effects of professionalization are essential as the discussion evolves. This study provides recent views on major topics in professionalization, including potential benefits, negative side effects, processes, competencies, and procedures. Results from in-depth interviews and an online survey demonstrate that AEA members view potential benefits of professionalization to be stakeholder trust, evaluator reputation and identity, while concerns about a potential negative side effect known as the “narrowing effect” (i.e., some evaluators will be alienated based on their background, competencies, etc.) were expressed by participants. These recent findings can inform the ongoing discussion of professionalization, and suggest new directions for future research on evaluation.     

Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations

This paper offers a framework for using a systems orientation and “foundational theory” to enhance theory-driven evaluations and logic models. The framework guides the process of identifying and explaining operative relationships and perspectives within human service program systems. Self-Determination Theory exemplifies how a foundational theory can be used to support the framework in a wide range of program evaluations. Two examples illustrate how applications of the framework have improved the evaluators’ abilities to observe and explain program effect. In both exemplars improvements involved addressing and organizing into a single logic model heretofore seemingly disparate evaluation issues regarding valuing (by whose values); the role of organizational and program context; and evaluation anxiety and utilization.     

Understanding innovation in health program planning and development

IntroductionA need for innovative public health programs is evident as the field adapts to address changes in health priorities and target populations. The Innovative Teen Pregnancy Prevention Program (iTP₃) was created to support and enable innovation in teenage pregnancy prevention, developing programs to reach the most at risk youth.MethodsA formative evaluation was conducted to understand what innovation means in the context of program development, and examine the process of innovation. Qualitative data was collected through baseline interviews with program development teams, referred to as Innovators, prior to the start of the project period and follow-up interviews conducted at the end of a 12-month funding period. Additional open-ended written responses were collected in the middle of the funding cycle. A thematic analysis with an open-coding scheme was used to identify emergent themes.ResultsInnovators considered programs innovative because of the target population of focus, program delivery mechanism, and/or program development approach. They specifically identified that a “culture” of innovation must be present if new programs are to be developed. Over time, Innovators began to shift their definition of innovation toward unique design processes and ecological approaches.DiscussionThrough creating a culture of innovation and utilizing systems thinking, this project provides important insights in how to develop innovations in public health.     

An Observational Study in Developing an Intergenerational Shared Site Program

Abstract

This article provides strategic insights into developing and evaluating an intergenerational shared site program at The Community Programs Center (CPC) in Port Jefferson, NY. Conducted under a grant from the Administration on Aging, attention is given to the meth odological challenges in conducting intergenerational evaluation research. The Project Evaluator found that the use of videotaping created a host of difficulties for both the staff and participants. Emphasis is given to chronicling the critical importance of developing a multi-pronged approach to gathering data, and the consequences of utilizing graduate students as evaluators. The evaluation component involved comparing two groups of child/elder participants within a day-care setting during an eight-month period of time. Each session was videotaped and later analyzed for changes in certain participant attributes. Based on the observations gathered, it was found that curriculum development, staff commitment, and programmatic flexibility were critical components in creating an intergenerational-shared site program. Also, the teaching orientation of the intergenerational facilitator was found to play a role in fostering communication among the participants. Important recommendations are provided for both intergenerational practitioners and research evaluators who want to replicate the CPC model. Although positive change was observed in select participants, future evaluation efforts must track change and activity outcomes over a greater period of time.     

Comparing rating paradigms for evidence-based program registers in behavioral health: Evidentiary criteria and implications for assessing programs

Decision makers need timely and credible information about the effectiveness of behavioral health interventions. Online evidence-based program registers (EBPRs) have been developed to address this need. However, the methods by which these registers determine programs and practices as being “evidence-based” has not been investigated in detail. This paper examines the evidentiary criteria EBPRs use to rate programs and the implications for how different registers rate the same programs. Although the registers tend to employ a standard Campbellian hierarchy of evidence to assess evaluation results, there is also considerable disagreement among the registers about what constitutes an adequate research design and sufficient data for designating a program as evidence-based. Additionally, differences exist in how registers report findings of “no effect,” which may deprive users of important information. Of all programs on the 15 registers that rate individual programs, 79% appear on only one register. Among a random sample of 100 programs rated by more than one register, 42% were inconsistently rated by the multiple registers to some degree.     

Entering Youth Justice: Comparing the Views of Human Services and Criminology Students

Abstract

The present paper explores how undergraduate students who are relatively new to the field of youth justice conceptualise their possible future work roles. The research compared the views of students who were commencing two undergraduate courses (subjects) concerned with working with young people. One course was taken by human services students and the other was taken by criminology students. Findings indicated that there were significant differences between these two groups in terms of their views about crime causation, as well as how they see the roles of youth justice workers. Students entered their respective programs and youth courses with differing professional orientations. The findings raise some intriguing questions at a time when universities and employers look to the potential benefits of “double degrees” and “joint degrees” that span supposedly complementary professional areas. However, contrasting beliefs about “troubled and troublesome” young people may resist easy accommodation within the same program of professional preparation.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Farm Friends

Abstract

Intergenerational programs have proliferated in community and educational settings, yet research on their effects and impacts lags (Kaplan, 2002; Kuehne, 1998/99), and the benefits and learning that occur for participants are under-theorized (Abrams and Giles, 1999; Smith and Yeager, 1999). Similarly, insufficient research has been reported on the learning that takes place in environmental education programs. In this paper we focus on the relationship that developed between community elders and elementary school-aged children during the first year of an intergenerational environmental education project. Our goal is to better understand these experiences so we can sustain participation and develop programs rich in rewards for elders and children. We are particularly interested in the nature of intergenerational learning that occurs when children are brought together with community elders for environmental education that is integrated into a school program. Granville and Ellis (1999) propose that for a program to be defined as intergenerational, it must show a benefit and value for both generations and “demonstrate an improvement in the quality of life for both, and from that, an improvement in the quality of life for all” (p. 236). We concur with these criteria and designed a program linking community elders with farming backgrounds with one class of 18 grade-seven girls for the purpose of fostering environmental consciousness. In this paper we focus our attention on the learnings of participants in order to understand the mutuallearning that occurs in an intergenerational environmental program.