Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.     

Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

The State of Our Art

Abstract

In the second part of this two-part paper, the author examines intergenerational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, which appeared in the previous issue, conceptual foundations for intergenerational programming were considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross-cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Conceptual and methodological issues in evaluating emergent programs

This paper identifies and discusses conceptual and methodological issues related to five attributes of emergent programs that frequently apply across the spectrum of human services: undefined client populations, inadequate causal evidence relating inputs to outcomes, shifting objectives, identification of criteria for standardizing treatments, and temporal constraints on program development. A case study of the American Red Cross nutrition program is used to illuminate the evaluation problems associated with these issues to mitigate them. Strategies found to be useful in the evaluation of emergent programs include the examination of the relationship among observed client characteristics and needs, external reviews of objectives, identification of client preconditions that support or inhibit performance, examination of normal treatment variation, and a multi-stage process.     

Using Social Construction Therapy with the REM Family

It is estimated that currently one child in five under the age of eighteen is a stepchild and that by h e year 2000, the remarried (REM) family will outnumber all other family forms. Historically, American society defines any form of family fife that does not conform to the traditional nuclear family as problematic. Family researchers have generally been consistent with society's negative views, using a problem-oriented approach to studying the REM family (Hoffman, 1991; Duncan and Brown, 1992; Coleman, Ganong, and Gingich, 1985). In so doing they have focused on documenting the difficulties of he REM family. This paper attempts to rectify these issues by first exploring the typical problems associated with the REM family focusing on social constructionist themes and therapy which serve to shift the focus of a problem-oriented approach toward a search for the strengths and positive characteristics of the family. Social construction focuses on the sociocultural and the individual relationship influences of language, family and culture. In addition, h e paper proposes a six-stage therapeutic model based on Social Constructionist Theory (Gergen and Gergen, 1983). The six stages of therapy with a remarried family are: ( I ) Joining the Family Meaning System, (2) Proposing the Notion of a Socially Constructed Family Meaning System, (3) Learning the Family's Meaning System,(4) Challenging the Family's Meaning System, (5) Amplifying the New Meaning System, and (6) Stabilizing the New Meaning System. It allows the REM family the ability to construct new meanings around their problems in living in order that they might find workable family solutions.     

A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.     

The use of Outcome Harvesting in learning-oriented and collaborative inquiry approaches to evaluation: An example from Calgary,Alberta

The Community Development Learning Initiative (CDLI) in Calgary, Alberta, Canada aims to be a network that brings together neighbourhood residents, community development practitioners and other supporters to learn and act on neighbourhood-based, citizen-led community development projects. In 2013, the CDLI initiated The Evaluation for Learning and Dialogue Project to provide the opportunity for organizations and supporters to work together to establish a shared vision and goals through discussions about evaluation learning and outcomes. It was intended that the project would be a useful learning tool for participating organizations by enabling them to engage in an evaluative methodological process, and record relevant information and to compare and learn from each other’s projects. Outcome Harvesting was chosen as the evaluation methodology for the project. This article reviews critical learning from the project on the use of Outcome Harvesting methodology in the evaluation learning and outcomes of local community development projects, and it provides lessons for other jurisdictions interested in implementing this methodology.     

Das Aggregationsproblem bei Mikro-Makro-Erklärungen

The present paper discusses a component of the micro-macro explanatory model (i.e. the individualistic research program): the question is whether and, if so, how collective properties (or, equivalently, macro variables) can be constructed by properties of individuals (i.e. by aggregating micro variables). This is the aggregation problem. After sketching the procedure of micro-macro explanations and its possible problems the state of research in regard to the aggregation problem is described. Then two kinds of aggregation are analyzed in detail: analytical aggregations (i.e. the micro-to-macro relationship is a logical one) and empirical aggregations (i.e. the micro-to-macro relationship is empirical). It is further discussed whether there exist collective properties that cannot be reconstructed individualistically. Many collective properties emerge by the unintended consequences of individual action. It is argued that the structure of micro-macro explanations do not differ if these consequences are intended or unintended. In the final part it is claimed that there are emergent properties but that they do not invalidate methodological individualism. In this context it is argued that “ontological” arguments are not helpful for solving explanatory problems of the social sciences.     

Political socialization and policy evaluation: The case of youth employment and training program

The detection and assessment of side effects or unintended consequences in policy evaluation are seldom conducted in other than a casual manner. This paper investigates the unintended impact of an employment and training program on the political orientations of participants. Applying a political socialization framework, this study compared the postprogram attitudes toward citizenship of 286 CETA-eligible youth with a randomly assigned control group of 161 youth. Entering minority status and gender as factors and preprogram attitudes toward citizenship as the covariate, the ANCOVA results indicate that political orientations of experimentais, especially program completers, are significantly higher compared with control group members.     

Career Development for At-Risk Youth: A Program Evaluation

At-risk populations pose a major challenge to the counseling profession. This article describes a career development program for at-risk youth and delineates both the process and the outcomes of a combined quantitative and qualitative program evaluation. The results suggest that at-risk youth can benefit significantly from this career development program. A combination of work and curricular experiences seemed to be effective with this population.     

Interpreting interviews

ABSTRACT

Realist evaluation (RE) is increasingly adopted to assess the effectiveness of social programs. RE goes beyond measurement of outcomes, providing insight into how and why programs work by focusing on how participants experience and respond to programs within certain contexts. A RE was conducted on Healthy Start-Départ Santé (HSDS), an early childhood development program in Canada, created to address the prevalence of overweight and obese children. The first step of the RE was to create an initial program theory (IPT) to identify key contexts and mechanisms relating to the success of the program. Realist methodology can be challenging, time consuming, and resource intensive and there are few practical examples of how to conduct a RE. This article aims to add to the realist methodological literature by describing the construction of the IPT, reflecting on the process used, and providing a starting point for future realist practitioners.     

Administrators as information brokers: A managerial perspective on naturalistic evaluation

The managerial perspective on justifying conclusions from naturalistic evaluation is based on how administrators function as brokers who exchange, highlight, and interpret information for others. Information brokering is particularly important within a cognitive perspective on organizations, and consists of process and outcome dimensions. Brokering processes include using evaluative information for conceptualizing, motivating action, and monitoring performance. Brokering outcomes refer to conceptual, instrumental, and symbolic results of information use. The combination of these dimensions results in a variety of substantive and methodological criteria that administrators might use to justify knowledge claims from program evaluation.     

The evidence base for social support interventions in child and family practice: Emerging issues for research and practice

This paper examines the evidence base for interventions which seek to enhance social support resources for families with children at risk of adverse developmental outcomes. Substantive and methodological issues related to the implementation of social support interventions are discussed and a research agenda for the development, testing and evaluation of such interventions is presented.     

Evaluation of a cross-cultural training program for Pakistani educators: Lessons learned and implications for program planning

In this paper, we share the results of a summative evaluation of PEILI, a US-based adult professional development/training program for secondary school Pakistani teachers. The evaluation was guided by the theories of cultural competence (American Psychological Association, 2003, Bamberger, 1999, Wadsworth, 2001) and established frameworks for the evaluation of professional development/training and instructional design (Bennett, 1975, Guskey, 2002, King, 2014, Kirkpatrick, 1967). The explicit and implicit stakeholder assumptions about the connections between program resources, activities, outputs, and outcomes are described. Participant knowledge and skills were measured via scores on a pre/posttest of professional knowledge, and a standards-based performance assessment rubric. In addition to measuring short-term program outcomes, we also sought to incorporate theory-driven thinking into the evaluation design. Hence, we examined participant self-efficacy and access to social capital, two evidenced-based determinants or “levers” that theoretically explain the transformative space between an intervention and its outcomes (Chen, 2012). Data about program determinants were collected and analyzed through a pre/posttest of self-efficacy and social network analysis. Key evaluation findings include participant acquisition of new instructional skills, increased self-efficacy, and the formation of a nascent professional support network. Lessons learned and implications for the design and evaluation of cross-cultural teacher professional development programs are discussed.     

How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.     

The problem of replicability in program evaluation. The component solution using the example of case management

Specificity and replicability of a program are crucial in order to rigorously evaluate the program's effectiveness. However, the traditional approaches to evaluation have ignored the problem that most social and mental health programs consist of vaguely specified interventions and cannot meet minimal standards of replicability. Consequently, the results of effectiveness evaluations of poorly defined programs have little general generalizability. Nevertheless, effectiveness information concerning su ch programs is much needed. One solution to this problem is to focus on program components, examining interventions at a level where operational definitions can be employed and replicability is possible. In this paper, this approach is expanded using the example of case management programs. A component-oriented approach ensures a systematic evaluation of program effectiveness for which the first question for evaluators is whether a program consists of interventions that reliably produce specific immediate outcomes. Then, and only then, is it possible to address questions concerning whether a program will reliably produce those outcomes that are the overall objectives of the program.     

Promoting Well-Being Through Relationship Building: The Role of Smartphone Technology in Foster Care

Mindful of the tremendous risk factors that foster youths encounter, a federal demonstration project sought to increase their protective factors and to mitigate their adverse experiences. The ability to form and maintain healthy relationships is a key protective factor as it helps youths navigate life’s challenges, especially those challenges that result from unresolved loss and grief as a result of being in care. The study was guided by a relational competence framework. The youths were issued smartphones, and their social service providers were given a Web-based application designed to increase communication between the foster youths, their providers, and their mentors. Findings reveal several favorable outcomes associated with the use of the smartphones but also several unintended negative consequences. Recommendations for balancing youths’, caregivers’, and providers’ needs, positions, and rights are discussed. Likewise, the authors discuss the study’s implications for program development that utilizes smartphone technology to increase youths’ ability to experience healthy relationships.     

Implementation and process evaluation of a school-based drug abuse prevention program: Project Towards No Drug Abuse

The present study provides an implementation, process, and immediate outcomes evaluation of the classroom component of Project Towards No Drug Abuse (TND). This project involves development and evaluation of a school-based drug abuse prevention curriculum for continuation high school youth, who are at relatively high risk for drug abuse. Three randomized conditions were evaluated: a standard care, classroom only, and classroom plus school-as-community. The latter condition was an enhanced school-based condition which involved outside-of-classroom meetings and activities. Implementation was high in both program conditions even though this was a higher risk context. Process evaluation data were favorable and did not vary between the two program conditions. Immediate outcomes data (knowledge) was higher in the two program conditions than in the standard care condition. Regarding the classroom program, addition of extra-classroom activities does not appear to alter the quality of delivery of the program.