A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12

Logic models are based on linear relationships between program resources, activities, and outcomes, and have been used widely to support both program development and evaluation. While useful in describing some programs, the linear nature of the logic model makes it difficult to capture the complex relationships within larger, multifaceted programs. Causal loop diagrams based on a systems thinking approach can better capture a multidimensional, layered program model while providing a more complete understanding of the relationship between program elements, which enables evaluators to examine influences and dependencies between and within program components. Few studies describe how to conceptualize and apply systems models for educational program evaluation. The goal of this paper is to use our NSF-funded, Interdisciplinary GK-12 project: Bringing Authentic Problem Solving in STEM to Rural Middle Schools to illustrate a systems thinking approach to model a complex educational program to aid in evaluation. GK-12 pairs eight teachers with eight STEM doctoral fellows per program year to implement curricula in middle schools. We demonstrate how systems thinking provides added value by modeling the participant groups, instruments, outcomes, and other factors in ways that enhance the interpretation of quantitative and qualitative data. Limitations of the model include added complexity. Implications include better understanding of interactions and outcomes and analyses reflecting interacting or conflicting variables.     

Systems thinking

Evaluation is one of many fields where “systems thinking” is popular and is said to hold great promise. However, there is disagreement about what constitutes systems thinking. Its meaning is ambiguous, and systems scholars have made diverse and divergent attempts to describe it. Alternative origins include: von Bertalanffy, Aristotle, Lao Tsu or multiple aperiodic “waves.” Some scholars describe it as synonymous with systems sciences (i.e., nonlinear dynamics, complexity, chaos). Others view it as taxonomy—a laundry list of systems approaches. Within so much noise, it is often difficult for evaluators to find the systems thinking signal. Recent work in systems thinking describes it as an emergent property of four simple conceptual patterns (rules). For an evaluator to become a “systems thinker”, he or she need not spend years learning many methods or nonlinear sciences. Instead, with some practice, one can learn to apply these four simple rules to existing evaluation knowledge with transformative results.     

The Objects of Sociology: A Response to Breslau's "Sociology After Humanism"

Daniel Breslau's essay opens up a valuable space in seeking to align the sociologically impure objects explored in science studies with the practice of a pure sociology. I challenge Breslau's conclusion that the latter can swallow the former and proceed with business as usual. Contrary to Breslau, I argue that confronting head-on the impure objects of science studies can indeed represent a new beginning in sociology as a discipline. I also correct Breslau's misreading of my work as "symmetrical humanism."     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.     

Private thoughts and public actions:: A commentary on articles dealing with ethical issues in evaluation

This commentary represents a personal reaction to seven articles on ethical issues in evaluation. The author of the commentary begins with two assumptions: (1) Professional evaluators might not have the vocabulary and constructs to analyze their thoughts and actions in ethical terms, and (2) professional evaluators have often confronted troubling situations that probably contain ethical issues embedded within them. Consequently, she queries the seven articles individually and collectively in terms of the following questions: 1. What is the range of ethical problems I might encounter as an evaluator; and how do I recognize the ethical characteristics of professional dilemmas? 2. How do I describe troubling evaluation problems I have encountered that seem to have an ethical component? 3. What are some approaches I have used or might use in thinking about a proper course of action? 4. How do I know if, in a given situation, I have behaved ethically? 5. Where do we go from here?     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

Practice‐based Research Networks: Opportunities in Family Therapy?

I would like to continue the discussion of practice‐based research networks (PBRNs). I will provide more detail on what I mean by PBRNs and describe some of the features of existing networks. I will argue that research networks could be a valuable tool for family therapists. But, I also point out that the establishment of the mechanisms and culture necessary for PBRNs to function effectively takes considerable commitment from clinicians and their professional organisations. I will also remount my hobbyhorse and give a more specific account of my thoughts about the use of the internet to provide the necessary infrastructure for these networks.     

Logic and evaluation theory

With this series of papers, evaluators are being called to substantiate the rationale and warrant for their own evaluative actions in ways parallel to how evaluators question the logic of program interventions, both as designed and as implemented. This endeavor is timely, appropriate, and important. In these comments, I raise modest questions about the logical constitution of an evaluation theory and about what is missing from a textual reading alone of such theory.     

The implementation of social choice functions via social choice correspondences: A general formulation and a limit result

A corollary of Maskin's characterization theorem for Nash implementable social choice correspondences is that only trivial social choice functions can be implemented. This paper explores the consequences of implementing non-trivial social choice functions by extending them minimally to social choice correspondences which are implementable. The concept of asymptotic monotonicity is introduced. The main result states that it is not possible to find social choice rules satisfying a mild condition on its range, which is asymptotically monotonic. The implication of this result is that the multiplicity of equilibria problem which is at the heart of Nash implementation theory persists even in the limit as the number of individuals in society tends to infinity. This is true even though the opportunities for an individual to manipulate the outcome disappears in the limit.This paper is an extensively revised version of a chapter of my Ph.D. dissertation submitted to Princeton University in June 1987. I wish to thank my advisor Hugo Sonnenschein for his valuable advice and constant encouragement. I am also grateful to Andrew Caplin, Vijay Krishna, William Thomson, Jean-Luc Vila and two anonymous referees of this journal for their numerous suggestions. All remaining errors are my own responsibility.     

A TOOL FOR TRAINING SUPERVISORS: USING THE SUPERVISION FEEDBACK FORM (SFF)

As the field of supervision and training has evolved, there has been a growing recognition of and appreciation for the skills required to be an effective supervisor or trainer (Liddle, Breunlin, & Schwartz, 1988). This new awareness has pointed to the need for effective methods and tools for training family therapy supervisors. One of the tools I developed to improve my supervision complete to give me feedback on my supervision. This article describes how developing and using an SFF was valuable to me in my training as a supervisor. Based on my experience, I believe the SFF could be a useful tool in training other supervisors.     

Community-Based Program Research

ABSTRACT

At first glance, the worlds of program professionals and evaluators may seem quite separate. However, there are common issues whose resolution will enhance both program development and research on programs. Program professionals and evaluators have a great deal to learn from each other. As that learning occurs, both program delivery and evaluation research will benefit. Both are concerned about matters of validity, whether it pertains to the nature of the program intervention itself or to the nature of the assessment of the program intervention. To reinforce the development of program evaluation in partnership, this paper discusses key points about evaluation research relevant to both program professionals and researchers. These include the contextual influences on a program, the “readiness” of a program for evaluation, and whether the evaluation research “works” for the program. Underlying our discussion is a call for the development of partnerships around research on programs. In that light, this article elaborates the process steps that should be taken to build program evaluation partnerships, including a discussion of what evaluators and program professionals need to know about each other, and their respective values, interests, and professional perspectives.     

"Show me your impact": evaluating transitional justice in contested spaces

This paper discusses some of the most significant challenges and opportunities for evaluating the effects of programs in support of transitional justice - the field that addresses how post-conflict or post authoritarian societies deal with legacies of wide spread human rights violations. The discussion is empirically grounded in a case study that assesses the efforts of the International Development Research Centre (IDRC) and one of its Guatemalan partners to evaluate the effects of a museum exposition that is attempting to recast historic memory and challenge racist attitudes in post-conflict Guatemala. The paper argues that despite the increasing trend to fund transitional justice programs, many international aid donors are stuck in traditional and arguably orthodox paradigms of program evaluation. This is having a negative effect not only upon the administration of aid but also upon how transitional justice research is perceived and valued by local populations. The case study experience indicates that there is no perfect evaluation model or approach for evaluating transitional justice programming - only choices to be made by commissioners of evaluation, evaluators, and those being evaluated. These are profoundly influenced by the extreme politics and moral values that define transitional justice settings as contested spaces in which calls to remember the tragic past must be balanced with aspirations to re-build a hopeful future.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

The evaluation of the PACA regional public health plan: reconciling the managerial, scientific and democratic finalities

For more than a decade now, evaluation has developed considerably in France, thanks in particular to the Société Fran?aise de l'évaluation, whose charter sets out a number of principles designed to guide the work of evaluators. This article examines how the evaluation process surrounding a regional public health plan (referred to as PRSP)--itself being a new instrument for regional planning in France--accords with one of these principles, which specifies that evaluation must be framed according to "a three-fold logic involving public management, democracy and scientific debate." Our analysis shows that while this evaluation was driven primarily by managerial concerns (i.e., assessing the capacity of the plan to structure health policy in a region), it also provided an Opportunity for restoring dialogue with a range of actors by opening up a space of cooperation and consultation. In addition, in order to ensure the legitimacy of the evaluation's conclusions, the knowledge produced by the evaluators had to rest on an irreproachable methodology. This example illustrates how evaluation, in the French tradition, is a process that strives to reconcile the viewpoints and expectations of managers, scientists and the general public; it is also a process that brings out lines of tension and areas of complementariness between these three logics.     

Eschew obfuscation in discussing political correctness

Conclusion Have I added to the constructive debate about PC? You decide. Sometimes it's valuable to provide a fresh perspective, even if it is rejected.One of the principles my journal was founded on was better writing. I think clear writing is crucial in the discussion of difficult issues like PC. While I may have made enemies of these authors, I probably share many values with them—if I could only be sure from their presentation. And I'll bet that they'll look at their future writing with a new eye.Founding Editor ofThe Journal of Social Behavior and Personality, an interdisciplinary peer-reviewed periodical.     

“I want my social worker”. One child's struggles to find an available maternal figure: reflections from a peer supervision group

In this paper, I analyse a piece of social work practice using the reflections provided by a peer supervision (work discussion) group. I describe my work with an eight year child looked after, who telephoned me stating that he wanted to kill himself. I discuss the powerful unconscious processes that contributed to his risk-taking behaviour and my emotional experience of it. I reflect on how my knowledge of psychoanalytic theory enabled me to contain the powerful feelings being projected and to safely manage the situation. I highlight how reflective peer supervision groups are a helpful way to understand the complexities of the lives of children looked after and can help inform decision making.     

Parsons' Concept of "Generalized Media of Social Interaction" and its Relevance for Social Anthropology

The concept of "generalized symbolic media of social interaction" was introduced by Parsons five years ago in his two articles on the concepts of power and influence.¹ I believe that it represents a theoretical development of the first importance, which has received much less attention—particularly in my own field of social anthropology—than it deserves. The purpose of the present paper is to review and criticize the concept itself and to point out its relevance to certain problems in social anthropology.