Using theory of change as a framework for examining community context and philanthropic impact

As part of the process of examining their theory of change (TOC) and reflecting on grant making activity, one Midwestern foundation employed a multipronged strategy to assess 209 community-based mental health grants across seven years of funding. This article details the evaluation approach, which comprised these areas of the TOC: grantees’ use of evidence-based interventions, cultural competency, quality improvement, community collaboration, and use of integrated care. Inductive analyses identified grantees’ use of innovative practices, trauma-informed care, and use of validated instruments. In an iterative exercise spanning multiple years, the foundation and researchers found ways to gather information about community capacity and impact while simultaneously minimizing grantee data collection/reporting burden. Findings revealed that community agencies were improving in their uptake of evidence-based interventions and use of validated instruments. Community grantee use of trauma informed care also improved over the study period. Grantee desire to collaborate and coordinate services within the community was also strong. The research also revealed that in multiple domains (such as technology and non-profit operating costs) grantees needed support in building agency capacity. These findings shed light on which aspects of the foundation’s TOC were being addressed through funding, and which areas of the community needed additional support.     

Value for money: A utilization-focused approach to extending the foundation and contribution of economic evaluation

Value for Money (VfM) is an evaluative question about the merit, worth, and significance of resource use in social programs. Although VfM is a critical component of evidence-based programming, it is often overlooked or avoided by evaluators and decision-makers. A framework for evaluating VfM across the dimensions of economy, effectiveness, efficiency, and equity has emerged in response to limitations of traditional economic evaluation. This framework for assessing VfM integrates methods for engaging stakeholders in evaluative thinking to increase acceptance and utilization of evaluations that address questions of resource use. In this review, we synthesize literature on the VfM framework and position it within a broader theory of Utilization-Focused Evaluation (UFE). We then examine mechanisms through which the VfM framework may contribute to increased evaluation use. Finally, we outline avenues for future research on VfM evaluation.     

Investigating value structure: Using theory-based starting configurations in Multidimensional Scaling (Research Note)

Abstract

Multidimensional Scaling (MDS) has been a central tool of psychological research into value structure for more than two decades. MDS is usually applied in an exploratory way because strict confirmatory analyses are still difficult to realize, due to the manifold problems of how to impose regional constraints on MDS solutions. “Weak confirmatory approaches” (Borg & Staufenbiel, 2007) which use a theory-based starting configuration could be an acceptable compromise. In this paper, we outline how such a configuration can be derived from Schwartz's (1992, p. 14) original model, which is characterized by a strict circular structure. Such a “parsimonious” structure seems particularly suited for multi-method studies. We also apply this configuration to the Spanish value data from the first three rounds of the European Social Survey to illustrate the procedure. Finally, we show how to test whether and to what extent regional adjustments of single items affect the overall fit of the respective MDS solution.     

Embedding value-for-money in practice: A case study of a health pooled fund programme implemented in conflict-affected South Sudan

In recent times, there has been an increasing drive to demonstrate value for money (VfM) for investments made in public health globally. However, there is paucity of information on practical insights and best practices that have helped implementing organisations to successfully embed VfM in practice for programming and evaluation. In this article, we discuss strengths and weaknesses of approaches that been used and insights on best practices to manage for, demonstrate, and compare VfM, using a health pooled fund programme implemented in conflict-affected South Sudan as case study supported by evidence reported in the literature while critiquing adequacy of the available approaches in this setting. An expanded and iterative process framework to guide VfM embedding for health programming and evaluation is then proposed. In doing so, this article provides a very relevant one-stop source for critical insight into how to embed VfM in practice. Uptake and scale-up of the proposed framework can be essential in improving VfM and aid effectiveness which will ultimately contribute to progress towards achieving the Sustainable Development Goals by 2030.     

Theory of change models deepen online learning evaluation

Massive Open Online Courses (MOOCs) have gained traction as resources for professional development. This article presents the method that we used to evaluate a professional development MOOC for postdoctoral trainees that was created by a university consortium in the US. Most approaches to evaluating MOOCs focus on analysis of participation, outcomes from course assignments, self-reported learning outcomes, course completion and user pathways through the online content or clickstream data. Few published evaluations describe in detail how learning happens within online courses and the anticipated medium and longer term cognitive and behavioral outcomes on participants. This work aims to guide those who are designing, implementing and evaluating MOOCs through applying theories of change to focus evaluation on the process of learning. This approach can be used as a complement to traditional approaches for evaluating MOOCs. We described how we worked with the MOOC team building the content to develop a theory of change for each module (or lesson) within the MOOC and used the theory of change to guide evaluation of short and medium term participant cognitive, affective and attitudinal, and behavioral outcomes. Finally, we share lessons learned and suggestions for implementing theories of change in both the design and evaluation phases of MOOC development.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

Workplace flexibility: a model of change

The inception of When Work Works – a theoretical- and research-based model of change in promoting effective and flexible workplaces – dates back to the confluence of a number of factors in the early 1990s. One decade later, this ‘perfect storm’ of factors led to the creation of When Work Works, and two decades later, to this critical assessment of the foundation for and results of this model of change. When Work Works has used partnerships with nationwide business organizations with community chapters in combination with media coverage, educational events, the creation of tools and materials and grassroots methods to communicate well-researched information to employers that show workplace flexibility is not just a favor or perk for employees but can be a powerful strategy for promoting better outcomes for both employers and employees. The program has been responsible for implementing a rigorous award and providing numerous community-based recognition and educational events on workplace effectiveness and flexibility around the country and for assisting many employers to effectively create more flexible and effective workplaces. When Work Works has been designed around eight key principles for creating change on a national level. This paper presents the context in which When Work Works evolved, a detailed discussion of the eight principles of Families and Work Institute's model of change, its results and lessons other organizations may wish to apply in their own change initiatives.     

Evaluating unintended program outcomes through Ripple Effects Mapping (REM): Application of REM using grounded theory

Several evaluation models exist for investigating unintended outcomes, including goal-free and systems evaluation. Yet methods for collecting and analyzing data on unintended outcomes remain under-utilized. Ripple Effects Mapping (REM) is a promising qualitative evaluation method with a wide range of program planning and evaluation applications. In situations where program results are likely to occur over time within complex settings, this method is useful for uncovering both intended and unintended outcomes. REM applies an Appreciative Inquiry facilitation technique to engage stakeholders in visually mapping sequences of program outcomes. Although it has been used to evaluate community development and health promotion initiatives, further methodological guidance for applying REM is still needed. The purpose of this paper is to contribute to the methodological development of evaluating unintended outcomes and extend the foundations of REM by describing steps for integrating it with grounded theory.     

Breaking the taboo: a history of monetary financing in Canada, 1930–1975

Monetary financing – the funding of state expenditure via the creation of new money rather than through taxation or borrowing – has become a taboo policy instrument in advanced economies. It is generally associated with dangerously high inflation and/or war. Relatedly, a key institutional feature of modern independent central banks is that they are not obligated to support government expenditure via money creation. Since the financial crisis of 2007–2008, however, unorthodox monetary policies, in particular quantitative easing, coupled with stagnant growth and high levels of public and private debt have led to questions over the monetary financing taboo. Debates on the topic have so far been mainly theoretical with little attention to the social and political dynamics of historical instances of monetary financing. This paper analyses one of the most significant twentieth‐century cases: Canada from the period after the Great Depression up until the monetarist revolution of the 1970s. The period was a successful one for the Canadian economy, with high growth and employment and manageable inflation. It offers some interesting insights into the relationship between states and central banks and present‐day discussions around the governance of money creation.     

Climate change: a complex societal process; analyzing a problem according to the Compram methodology

Abstract

In this article some issues about the theory of societal complexity are explained: what are complex societal problems and how should these be handled. The problem of climate change is used as an illustration. The Compram methodology is a methodology based on the theory of societal complexity that gives guidelines to handle reallife complex societal problems such as climate change. Handling according to the Compram methodology supports defining and changing a complex societal problem in a sustainable way. In this article the focus of the methodology is on the relation between the steps in the Compram methodology and the problem-handling phases.     

Administrative structure theory and evaluation: An example from the Community Services Administration

Concern about the inability of existing theories of utilization to explain the use (or lack thereof) of evaluation research on policymaking in public agencies leads the authors to postulate an “administrative structure theory.” Utilization is then explained, in large part, by the structure and behavior of administrative agencies. Administrative structure theory rests on four propositions which relate evaluation research to the development and maintenance sub-systems (within the administrative structure). These propositions are illustrated in the context of the recently abolished U.S. Community Services Administration.     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

Responding to ambiguity, responding to change the value of a responsive approach to evaluation

Loosely organised teams and flexible arranged programs offer a new answer to improve the practice of palliative health care. Given the ambiguity of this care concept and the innovative organisational arrangement it is possible for confusion to emerge over the goals, core activities and target groups. In such a context it is not sufficient to determine the degree to which goals have been attained because the search for goals is part of the developmental process. An evaluation of a palliative team in a Dutch health region shows that it is more appropriate to acknowledge ambiguity and to go on to facilitate the handling of ambiguity as a vital element in dynamic learning processes. The utility of a responsive approach to evaluation, methodological considerations and learning experiences are described. The article is educational for workers in the field of health care but also to scholars and practitioners in an evaluation community.     

Service users’ involvement in social work education: focus on social change projects

ABSTRACT

This article presents and analyzes four projects focusing on diverse forms of service users’ involvement in social work training and research in different countries (Israel, Italy, Slovenia and UK). It highlights the value of service user involvement (SUI) to specific social change objectives and to social work education. The conceptual framework focused on the Standpoint Theory, while methodologically participatory action research was applied, and evaluation measures were developed. Key findings, facilitators and limitations to the involvement, students’ views of it and similarities and differences among the four projects are outlined. The challenges embedded in introducing and sustaining social change objectives in a co-production framework within social work education are identified, alongside the added value of meeting them. The differentiated impact the projects had on students is highlighted as well as their significance for health and social care providers were relevant. It is encouraging that in each project SUI was positively valued. The projects indicate the wide range of SUI in the content and format of social work education, as well as its applicability cross-culturally to a range of key issues pertaining to both training and research in social work.     

Ease of retrieval as a source of information in social change judgments

Abstract

Several authors have proposed two alternative sources of information in memory based judgments: the contents retrieved from memory and the ease with which the retrieval process is perceived. Schwarz and colleagues have made it clear that, in certain circumstances, people base their judgments on the subjective experience of ease with which the information is retrieved. The present work studies the extent to which the use of this heuristic could be generalized to judgments of social change (Study 1) and some of the factors that can affect the ability to diagnose the ease of retrieval as a source of information (Study 2). The results obtained confirm the ability to generalize this heuristic to judgments of social change and to establish some conditions that affect the diagnostic- city of this source of information.     

Building evaluative culture in community services: Caring for evidence

An organization with a strong evaluative culture engages in self-reflection, evidence-based learning and experimentation. It sees evidence as essential for managing well, but building such a culture is challenging. Community service organizations seek to provide effective services for their clients. To build an evaluative culture, they need to acquire basic monitoring and evaluation capabilities, be provided with opportunities for using these capabilities and be adequately motivated to care about evidence as a means to improve services to their clients. Leadership along with a phased in approach are key in bringing about these behaviour changes.     

Detecting self-organised behavioural patterns in workplace settings: Towards a lay epistemic theory

Abstract

Several theorists have called for people who work in organisations to become more proficient at discerning the subtle, self-organised patterns that emerge in their organisations. Such theorists argue that the capacity to detect such patterns can enable managers to understand what drives organisational behaviour, and what stands in the way of organisational transformation. This paper presents findings of a study on how some organisational practitioners detect and understand self-organised behavioural patterns that emerge in the workplace. It seeks, therefore, to integrate processes used by successful practitioners with a theoretical foundation grounded in systems and organisational research.     

Mathematical Sociology in Japan: Its Powerful Development and a Problem

Mathematical sociology in Japan was born in the mid-1970s and has actively developed since then. Mathematical sociologists in Japan have studied various topics of mathematical sociology as well as of quantitative sociology. The Japanese Association for Mathematical Sociology (JAMS) was established in 1986. It holds semi-annual conferences and publishes Sociological Theory and Methods, its official journal. Thus, the JAMS is a platform for mathematical sociologists in Japan to present and publish papers, contributing to the institutionalization of mathematical sociology in Japan. It has also co-sponsored five joint conferences with the Section on Mathematical Sociology of the American Sociological Association. Based on these activities, mathematical sociology in Japan could be judged to be vibrant domestically and internationally; it has a bright future. I argue, however, that mathematical sociologists in Japan have tended to confine themselves to areas where mathematical modeling is relatively easy. These areas are not necessarily attractive to sociologists in other fields. I propose that mathematical sociologists in Japan should tackle social phenomena that other sociologists think are critical to sociology so that they further contribute to advances in the discipline.