Causal inferences on the effectiveness of complex social programs: Navigating assumptions,sources of complexity and evaluation design challenges

This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention’s effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

Measuring costs from permitted gaming: concepts and categories in evaluating gambling's consequences

This paper addresses the methodology of cost–benefit analysis as applied to policy alternatives regarding legally sanctioned gambling in its various forms. Existing economic studies regarding the social costs of gambling are reviewed and critiqued. Distinctions are made between definitions of social costs that are defined as actions which result in negative changes in aggregate social wealth (the narrow definition), and those which also include internal nonmarket costs that are borne by individual gamblers and their immediate families and acquaintances (the broader definition). This distinction is important because of its bearing on economic policies that are primarily concerned with economic efficiency versus policies that are more paternalistic, which attempt to protect individuals from self-damage or self-destruction by restricting their ranges of choice. Whether societies choose to prohibit or severely restrict permitted gambling, or allocate substantial resources to mitigate its negative side effects, rests largely on which of these perspectives regarding social costs they find more appropriate. Finally, the issue of social protection through restrictions on the availability of gambling to the entire population, versus a strategy tailored toward identifiable problem gamblers, is discussed.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Power and perceptions in participatory monitoring and evaluation

This paper examines the assumptions that commonly underpin the design of participatory monitoring and evaluation (PM&E) within development programmes through the examination of a case study of a large rural development programme in Uganda. This case study reveals a mismatch between programme assumptions and participant perceptions, which stymied the implementation of PM&E. In this case, PM&E was based on the assumptions that programme and participant goals were compatible, participants were willing to volunteer and engage themselves for the greater good of their communities, and information sharing and communication was fairly free and open. However, farmers within the programme felt that the bureaucratic and accountability requirements of the programme were not their concern, and were acutely aware of power differences between farmers and programme officials, and between farmers of varying status. The key lesson to be drawn from this case is the need for a heightened awareness of power dynamics and political factors in the design of PM&E.     

Dimensions of Internet use: amount,variety, and types

We examine the dimensions of Internet use based on a representative sample of the population of the UK, making three important contributions. First, we clarify theoretical dimensions of Internet use that have been conflated in prior work. We argue that the property space of Internet use has three main dimensions: amount of use, variety of different uses, and types of use. Second, the Oxford Internet Survey 2011 data set contains a comprehensive set of 48 activities ranging from email to online banking to gambling. Using the principal components analysis, we identify 10 distinctive types of Internet activities. This is the first typology of Internet uses to be based on such a comprehensive set of activities. We use regression analyses to validate the three dimensions and to identify the characteristics of the users of each type. Each type has a distinctive and different kind of user. The Internet is an extremely diverse medium. We cannot discuss ‘Internet use’ as a general phenomenon; instead, researchers must specify what kind of use they examine.     

The use of concept mapping for scale development and validation in evaluation

Evaluators often make key decisions about what content to include when designing new scales. However, without clear conceptual grounding, there is a risk these decisions may compromise the scale's validity. Techniques such as concept mapping are available to evaluators for the specification of conceptual frameworks, but have not been used as a fully integrated part of scale development. As part of a multi-site evaluation of family support programs, we integrated concept mapping with traditional scale-development processes to strengthen the creation of a scale for inclusion in an evaluation instrument. Using concept mapping, we engaged staff and managers in the development of a framework of intended benefits of program participation and used the information to systematically select the scale's content. The psychometric characteristics of the scale were then formally assessed using a sample of program participants. The implications of the approach for supporting construct validity, inclusion of staff and managers, and theory-driven evaluation are discussed.     

Problem attribution and intervention. The interpretation of problem causations and solutions in regard of Brickman et al.

Typologies of problem perception are of great importance for the reconstruction of social workers’ professional ideologies. The best-known typology has been provided by Brickman et al. who differentiated between a medical, moral, enlightenment, and compensatory model. The article discusses this typology on a conceptual level and indicates its central flaws. An empirical study is presented in which an alternative attempt to develop a typology is tested. The evaluation refers to a wider spectrum of perceptions of problem causes and solutions than the typology of Brickman et al. The resulting typology comprises a ‘social’, a ‘workfare’, a ‘pathology’, and an ‘education’ type.     

The use of participant observation in evaluation research

In the last several years considerable interest has developed in the use of qualitative methods in evaluation research. Recent works indicate that the tradition of qualitative research has much that will be useful to evaluation researchers. This paper does not seek to cover the range of qualitative methods available to the evaluation researcher but rather is concerned with one methodology in that tradition, participant observation. The paper describes the methodology of participant observation for those who are interested in its characteristics and for those who would like to consider it for evaluation or other types of research. Examples are given of how it has been used and references to additional sources of information are included as well. The paper is organized into three sections: characteristics of participant observation, when it is appropriate to use it, and specific techniques of the method.     

Models,methods and network topology: Experimental design for the study of interference

How should a network experiment be designed to achieve high statistical power? Experimental treatments on networks may spread. Randomizing assignment of treatment to nodes enhances learning about the counterfactual causal effects of a social network experiment and also requires new methodology (ex. Aronow and Samii, 2017a, Bowers et al., 2013, Toulis and Kao, 2013). In this paper we show that the way in which a treatment propagates across a social network affects the statistical power of an experimental design. As such, prior information regarding treatment propagation should be incorporated into the experimental design. Our findings justify reconsideration of standard practice in circumstances where units are presumed to be independent even in simple experiments: information about treatment effects is not maximized when we assign half the units to treatment and half to control. We also present an example in which statistical power depends on the extent to which the network degree of nodes is correlated with treatment assignment probability. We recommend that researchers think carefully about the underlying treatment propagation model motivating their study in designing an experiment on a network.     

Collegial oligarchy and networks of normative alignments in transnational institution building

This paper presents a combined relational and cultural approach to transnational institution building by focusing on a network analysis of a small collegial oligarchy and normative alignments among its peers. To contribute to a theory of institutionalization, we propose hypotheses about whom professionals as institutional entrepreneurs are likely to select as members of their collegial oligarchy, about the role of social networks among them in identifying these leaders, and about the costs of alignments on these leaders’ normative choices. We test these hypotheses using mainly Exponential Random Graph Models (ERGMs) applied to a dataset including network information and normative choices collected at the so-called Venice Forum – a field-configuring event that was central in creating and mobilizing a network of European patent judges for the construction of a new transnational institution, the European Unified Patent Court. We track normative alignments on the collegial hierarchy in this network of judges and their divergent interpretations of the contemporary European patent. Highlighting this under-examined articulation of relational infrastructures and cultural framing in transnational institutionalization shows how Northern European forms of capitalism tend to dominate in this institutionalization process at the expense of the Southern European forms. It also helps reflect on the usefulness of analyses of small networks of powerful players in organizational societies, where power and influence are highly concentrated.     

The use of concept mapping in measurement development and evaluation: Application and future directions

The past decade has seen an increase of measurement development research in social and health sciences that featured the use of concept mapping as a core technique. The purpose, application, and utility of concept mapping have varied across this emerging literature. Despite the variety of uses and range of outputs, little has been done to critically review how researchers have approached the application of concept mapping in the measurement development and evaluation process. This article focuses on a review of the current state of practice regarding the use of concept mapping as methodological tool in this process. We systematically reviewed 23 scale or measure development and evaluation studies, and detail the application of concept mapping in the context of traditional measurement development and psychometric testing processes. Although several limitations surfaced, we found several strengths in the contemporary application of the method. We determined concept mapping provides (a) a solid method for establishing content validity, (b) facilitates researcher decision-making, (c) insight into target population perspectives that are integrated a priori, and (d) a foundation for analytical and interpretative choices. Based on these results, we outline how concept mapping can be situated in the measurement development and evaluation processes for new instrumentation.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

Translation of words and concepts in social policy

This paper describes how optimal matching or sequence analysis can be used to empirically evaluate the use of ideal typologies to describe the lifecourse. In our illustration, the lives of over 250 individuals, originally members of the Boyd Orr study as children in the 1930s, now in early old age (aged 65–75 years) have been reconstructed using sequence data derived from life grid interviews. Individuals are allocated to various ideal types for three distinct pathways or trajectories describing a person’s labour market, relationship and housing histories. Finally, these allocations are examined in relation to an individual’s self‐reported quality of life (QoL) in an attempt to explore the extent to which the experience of structurally advantaged or disadvantaged states are related to a good QoL in later life. Our findings suggest that individuals assigned to structurally advantaged statuses, in each of the trajectories, generally experience higher levels of QoL compared to those assigned to structurally disadvantaged positions. In particular, the single best predictor of high levels of self‐reported QoL is for those who became owner–occupiers late in life (typically council property purchasers). This suggests that the experience of stability during earlier parts of the lifecourse can provide opportunities within a de‐institutionalised lifecourse that we are witnessing at the beginning of the 21st century. The results suggest that the contemporary circumstances of older people are more important for their QoL. Yet, these circumstances are, in part, conditioned by their lifecourse.     

Exploring the embassy sampling strategy for large-scale cross-national study in replicating the normative theory of global public relations

This study explored a new sampling strategy of using embassies as cultural enclaves for large-scale cross-national research in testing the generic status of individual excellence principles in the normative theory of global public relations. Based on survey data, the investigation looked at factorial invariance for six organizational process-related excellence principles across two cultural groups of embassies (N = 113) in Washington, D.C.: individualist and collectivist. The findings suggested that the principle of symmetric internal communication may not be generic whereas the others are universal in factor structure and further put into question the generic status of the notion of symmetry so fundamental to symmetric internal communication.     

The use of evaluation models for decision making: Application to the community anti-crime program

Evaluations are typically designed to provide useful information, including measures of program effectiveness. Although this information is intended to be useful, the general literature on evaluations suggests that instances in which evaluations are explicitly used for decision making are rare. This article discusses the use of decision-theoretic evaluation models, and their applicability in practice as well as in principle. A case study is presented from the evaluation of LEAA's Community Anti-Crime Program, in which a decision analysis based on a decision-theoretic evaluation model influenced a programmatic decision which arose during the conduct of the evaluation.     

Repression,resistance and lifestyle: charting (dis)connection and activism in times of accelerated capitalism

ABSTRACT

Studying the nexus of media and social movements is a growing subfield in both media and social movement studies. Although there is an increasing number of studies that criticize the overemphasis of the importance of media technologies for social movements, questions of non-use, technology push-back and media refusal as explicit political practice have received comparatively little attention. The article charts a typology of digital disconnection as political practice and site of struggle bringing emerging literatures on disconnection, i.e. forms of media technology non-use to the field of social movement studies and studies of civic engagement. Based on a theoretical matrix combining questions of power, collectivity and temporality, we distinguish between digital disconnection as repression, digital disconnection as resistance and digital disconnection as performance and life-style politics. The article discusses the three types of digital disconnection using current examples of protest and social movements that engage with practices of disconnection.

Abbreviations: AFA: Anti-Fascist Action; CHRI: Center for Human Rights in Iran; DDoS: Distributed Denial of Service     

A Typology of Maritally Violent Men and Correlates of Violence in a Community Sample

This study empirically tests A. Holtzworth‐Munroe and G. L. Stuart's (1994) typology of male batterers in a community sample. Latent class analyses based on severity of physical aggression, generality of violence, and psychopathology partially replicated the Holtzworth‐Munroe and Stuart typology by identifying 3 types of violent men: family‐only, medium‐violence, and generally violent/psychologically distressed. Separate groupings of borderline/dysphoric and generally violent/antisocial types were not found. In comparisons of batterer types to each other and to nonviolent men, generally violent/psychologically distressed men differed from other groups on psychological abuse, life stress, marital satisfaction, and attitudes about violence. Types also differed on wives' fearfulness of their husband and injury from marital aggression. Implications of conceptualizing marital violence from a multidimensional typology perspective are discussed.     

A critical examination of key assumptions underlying diversity and social justice courses in social work

This article examines underlying assumptions of Master of Social Work diversity and social justice courses as sites that embody social work’s dual projects of social justice and professionalization. Through a latent content analysis of course syllabi from 27 US-based social work programs, three key assumptions emerged: (1) social workers are members of dominant social groups; (2) cultural competency and anti-oppression are compatible frameworks; (3) self-awareness mitigates oppression. Findings reflect the reification of dominant culture groups in social work and promotion of individual-level skill development over structural change. Implications and recommendations for social work education and future research are discussed.