Causal inferences on the effectiveness of complex social programs: Navigating assumptions,sources of complexity and evaluation design challenges

This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention’s effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.     

Network Canvas: Key decisions in the design of an interviewer-assisted network data collection software suite

Self-reported social network analysis studies are often complex and burdensome, both during the interview process itself, and when conducting data management following the interview. Through funding obtained from the National Institute on Drug Abuse (NIDA/NIH), our team developed the Network Canvas suite of software – a set of complementary tools that are designed to simplify the collection and storage of complex social network data, with an emphasis on usability and accessibility across platforms and devices, and guided by the practical needs of researchers. The suite consists of three applications: Architect: an application for researchers to design and export interview protocols; Interviewer: a touch-optimized application for loading and administering interview protocols to study participants; and Server: an application for researchers to manage the interview deployment process and export their data for analysis. Together, they enable researchers with minimal technological expertise to access a complete research workflow, by building their own network interview protocols, deploying these protocols widely within a variety of contexts, and immediately attaining the resulting data from a secure central location. In this paper, we outline the critical decisions taken in developing this suite of tools for the network research community. We also describe the work which guides our decision-making, including prior experiences and key discovery events. We focus on key design choices, taken for theoretical, philosophical, and pragmatic reasons, and outline their strengths and limitations.     

Models and methods for characterizing the structural parameters of groups

This paper outlines and illustrates the application of loglinear models for characterizing the structural parameters of groups, as conceptualized by Blau (1974, 1977). Models for nominal parameters are those in which an ingroup outgroup distinction is embodied. Models for graduated parameters introduce distance effects. These models appear to be useful supplements to spatial techniques for identifying patterns of differential association, and they have implications for the calculation of proximity estimators in the use of those techniques.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

The analysis of multiple ties in longitudinal egocentric network data: A case study on bidirectional relationships between trust and drug use

We extend multi-level models to examine single egocentric network ties to the joint analysis of paired dynamic ties. Two analytic challenges are addressed. First, inference needs to account for multiple layers of nesting: ties are nested within pairs, pairs are nested within time points, and time points are nested within egos. Second, the focus is on the relationship between two dynamic ties; specification of outcome and predictor may be difficult. Instead, we treat both ties as outcomes. Our approach is used to analyze trust and reported drug use between egos and alters over time in a Bayesian framework.     

The influence of logo design on country image and willingness to visit: A study of country logos for tourism

This study examines the influence of tourism logo design on people's country image and their intention to visit the country being promoted. In an online survey, undergraduate students were exposed to the tourism logos of Australia, Kenya, and Malawi. The results show that students’ evaluations of a country's logo significantly affected their image of the country and their willingness to visit it after controlling for pre-existing knowledge and attitude toward the object country.     

Ethnographic Cartographies: Social Movements,Alternative Media and the Spaces of Networks

Research on social movement networks has been defined by an emphasis on structural determinism and quantitative methodologies, and has often overlooked the spatial dimension of networking practices. This article argues that scholars have much to gain if (1) they move beyond the understanding of networks as organisational and communication structures, and analyse them as everyday social processes of human negotiation and construction, and (2) they pay attention to how networks between different organisations create multiple and overlapping spaces of action and meaning that define the everyday contexts of social movements. Drawing on ethnographic research within the Cuba Solidarity Campaign, this article explores the everyday dimension of political and communication networks. It shows that everyday networking practices are embedded in processes of identification and meaning construction, and are defined by a politics of inclusion and exclusion; introducing the concept of ethnographic cartography, it demonstrates that social movement networks are incorporated into everyday practices and narratives of place-making.     

Phenomenography: the development and application of an innovative research design in higher education research

This article considers the application of phenomenography, as arguably the only research design (so far) to have been developed substantially within higher education research by higher education researchers. It considers its origins, meaning and application, and the issues and critiques that it has raised. It concludes that, while phenomenography has both acknowledged limitations and undoubted achievements, it still appears to have much to offer to higher education research, and its usage might be extended to other fields.     

Social and feminist design in emergency contexts: the Women’s Social Architecture Project,Cox's Bazar,Bangladesh

ABSTRACT

The rapid influx of Rohingya refugees into Cox's Bazar, Bangladesh, has led to the formation of huge camps, built on difficult terrain, short of space and with high population density. All these factors present numerous challenges to agencies seeking to provide latrines, water points, and bathing facilities. Feedback gathered from women and girls highlighted significant challenges around access, safety, privacy, and dignity, including management of personal hygiene and menstruation. Within the humanitarian community, site planning and WASH (water, sanitation and hygiene) infrastructure planning/design processes have mainly been dominated by male architects and engineers, often with gendered assumptions about the user. Acknowledging these challenges, Oxfam implemented the Women’s Social Architecture Project to work with women and adolescent girls, and female architects with a background or interest in social or feminist design and architecture, to add a different perspective into the design and siting of WASH facilities. The project has consisted of formative research with Rohingya women and girls, and design workshops with architects in which Rohingya women have co-created designs that tackle the issues they face regarding WASH facilities. This article provides an account of the progress to date, together with feedback from women and girls and the WASH Sector in Cox's Bazar ahead of the final completion of the project in June 2019.     

A group, a network and the winning of church union in Canada: a case study in leadership

Au cours du premier quart du xxe siècle, un mouvement oecuménique au sein des Eglises protestantes canadiennes a menéà leur fusion, et à la création de ľEglise Unie du Canada. Au moyen de documents ďarchives, j'étudie dans cet article le leadership du mouvement social qui a produit cette fusion. J'analyse ďabord comment la structure du leadership a évolué, du groupe fondateur à un réseau étendu et de faible densité. Les actions du leadership à divers niveaux de ľEglise et de la société sont ensuite examinées, ainsi que les activités dissidentes de plusieurs membres du groupe fondateur.
This paper, based on archival data, is a study of pro-change leadership in a social movement: namely, the ecumenical movement within the Canadian Protestant churches during the first quarter of this century which succeeded in merging several denominations into the United Church of Canada. First, the structure of the leadership is traced from a founding group to a dispersed and enlarged network, then the leadership is displayed in action at various levels of the church and society. Finally, the dissident activity of several of the original group members is investigated.     

The Good,the Bad and the Unhappy: The Cultural Meanings of Newspaper Reporting on Jackpot Winners

Accounts of jackpot winners and big gambling wins are common in Swedish newspapers. Analysis of over 2000 such newspaper articles reveals that their content is structured according to specific themes and cultural topics. Four such topics are identified: wealth as a test of morals and character, the social impact of wealth, the just and good world, and luck and the occult. Culturally structured narratives like these—which elaborate on mythological and moralistic schemes of good and evil, rewards and punishments, and which concern questions of human nature and social values—have a long history in folk tradition and in Christian moral teachings. Jackpot wins provide a discursive realm for moral and existential questions that, to some extent, fills a void left by the decline of traditional folklore and formal religion. Apparently, this discourse stimulates interest in games and constitutes one of the cultural roots of contemporary gambling.     

Sexual Harassment in the Creative Industries: Tolerance,Culture and the Need for Change

The economic, social and cultural contributions of the creative industries are essential elements of many societies and their governments' policies. However, there is growing evidence that precarity, competition and lack of regulation within these industries is exacerbating inequalities with respect to gender, race and class. With a focus on gender and sexual harassment among female workers, this study involved 32 in‐depth interviews with women working in the Netherlands' creative industries. Data were analyzed using content analysis. Findings suggest that sexual harassment is prevalent, and many women considered it to be part of their occupational culture and career advancement. Four factors influenced this phenomenon: competition for work; industry culture; gendered power relations; and the importance of informal networks. Implications include the need for a climate of non‐tolerance, sector‐specific research and guidelines, sensitivity training and further work with unions and professional associations to provide worker protection strategies traditionally undertaken by organizations. The article concludes that effective sexual harassment prevention requires action at the individual, educational, sectoral and governmental levels, beginning with public conversations to convey the message that sexual harassment is never acceptable.     

Making space for ‘the social’: connecting sociology and professional practices in urban lighting design

Lighting is increasingly recognized as a significant social intervention by both lighting professionals and academic social scientists. However, what counts as ‘the social’ is diverse and contested, with consequences for what kind of ‘social’ is performed or invented. Based on a long‐term research programme, we argue that collaboration between sociologists and lighting professionals requires negotiating discourses and practices of ‘the social’. This paper explores the quality and kinds of spaces made for ‘the social’ in professional practices and academic collaborations, focusing on two case studies of urban lighting that demonstrate how the space of ‘the social’ is constrained and impoverished by an institutionalized division between technical and aesthetic lighting. We consider the potential role of sociologists in making more productive spaces for ‘the social’ in urban design, as part of the central sociological task of ‘inventing the social’ (Marres, Guggenheim and Wilkie 2018) in the process of studying it.     

A Comparison of two analytic methods for the identification of neighborhoods as intervention and control sites for community-based programs

Study Objective: Interest in community as the focus of public health interventions is growing. However, choosing intervention and comparison neighborhoods when designing community based programs poses a challenge to program planners. Ideally, intervention neighborhoods should be chosen based upon risk profiles and demonstrated need for the program. Multiple sources of data that tap into neighborhood characteristics might be used to facilitate the selection of intervention and comparison neighborhoods for program implementation and evaluation.Design: We present and compare selected characteristics of two analytic methods that can be used to create perinatal risk profiles of neighborhoods within cities. For our example, we used information from several sources of routinely available data and used census tract level low birthweight as our intervention or outcome variable.Main Results: At the neighborhood level, we found average household wealth of the census tract, proportion of births to women with late or no prenatal care, proportion of teen births per census tract, per capita crime rates, proportion of housing violations, and number of community organizations as being important factors identifying neighborhoods at risk for high rates of low birthweight births. Advantages of both methods are discussed and risk profiles generated from either method can be used not only to identify high risk areas of the city for adverse perinatal outcomes but also for the identification of intervention and comparison neighborhoods for implementation of community based programs.     

Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.     

Untouchable cultures: memory,power and the construction of Dalit selfhood

In the context of the philosophical literature on multiculturalism, I argue in this article that models of cultural identity based entirely on the nonvoluntary possession of a set of cultural characteristics are seriously incomplete. In particular, such models cannot address the need, among some groups, to reconstruct, invent and imagine alternative positive identities as a result of historical injustice, and to fill in the content of ‘culture’ accordingly. As an illustrative case, I survey processes of identity construction among ‘Dalits’, members of former ‘untouchable’ and other lower caste communities in India, with a focus on the role of historical consciousness and existing power relations in the imagination of Dalit culture. Dalit strategies of identity negotiation reveal the understandable need, on the part of the members of this community in progress, to produce a cultural identity that makes sense, psychologically and politically, given who they cannot imagine themselves to be, due to the fact of historical oppression. My analysis does not merely target essentialism, nor is it meant to be deconstructive of identity claims. Rather, I highlight select elements within the negotiation of Dalit identity to illustrate (1) the relevance of real historical relations of discrimination and inequality to the construction of culture; (2) the equivocal character of ‘choice’ within this process; and (3) the emancipatory possibilities provided by imagined narratives of cultural selfhood.     

The postfeminist ideology at work: Endorsing gender essentialism and denying feminine devaluation in the case of design work

Feminist scholars define ‘postfeminism’ as a set of ideas that both endorse and disavow feminism. Recent work documents postfeminism in interviews of women talking about experiences with gender inequality at work. The present study extends existing theorizations by showing how postfeminism obfuscates ongoing inequality at work. To do this I specify postfeminism in the terms of two mechanisms by which work becomes gender typed: gender essentialization and feminine devaluation. By relating the literature on postfeminism to sociological research on the persistence of gender typing of work, I show how ‘postfeminist ideology’ amounts to a double entanglement with gender typed work. In other words, postfeminist ideology drives distinct stances towards gender essentialization and feminine devaluation. I draw on 40 interviews with product designers in the United States to show this presents empirically. Designers essentialize differences between men and women designers, even mobilizing difference to claim greater inclusion of women designers, who are thought to empathize best with women consumers. Yet when essentialization is accompanied by feminine devaluation, designers deny it and it goes unrecognized. Because of this, the postfeminist ideology celebrates essentialized gendered differences, but insists on overlooking devaluation, which is a key element of ongoing gender inequality.     

An approach to the design of housing for Kathmandu, Nepal : Improving utility and facilitating construction

Totally modern design of housing in Nepal causes problems both in its use and construction. The modern designs have arbitrary spatial layouts that are incompatible with the cultural needs of the users. They require many foreign building materials and skills making construction problematic. Whereas modern design can improve performance of building in areas of damp-proofing and thermal control, traditional design provides culturally useful spaces. Traditional housing is easy to build but fails in some technical aspects of building. A synthesis of modern and traditional designs is the most suitable approach to housing design in Nepal and governmental policy should facilitate it by redirecting the education on building technology, by encouraging the production of indigenous materials, by suitable regulatory controls and by incorporating an optimal design in its own housing projects.     

Articulating influence: Toward a research agenda for interpreting the evaluation of soft power,public diplomacy and nation brands

While the terms soft power, public diplomacy (PD) and nation brands have cemented their place in academic discourse during the early 21st century, the evaluation of these activities has not been given anywhere near the same level of attention. When describing how campaigns are evaluated, scholars tend to make assumptions based on the goals or outputs of an initiative rather than on the basis of reliable, empirical data on its results. Strong positivist tendencies within current scholarship usually lead to evaluation being considered in terms of methodology and best practice, typically with the assumption that certain preferred outcomes will be demonstrable if an ideal model is followed. Most significantly, such approaches seem to underplay the interests and objectives that inform and constrain choices surrounding modes of communication and evaluation. I argue here that PD activities are rarely the product of rational choices about communication options, and nor is PD evaluation the result of applying the “best” methodology. Rather, questions of PD and evaluation practices are bound together in complex organizational and power structures that generate pragmatic responses both to the “problem of influence” and the reporting of results. Through use of the concept of articulation, this article outlines a framework for interpreting evaluation practices from a contextualized perspective, which grasps how and why soft power practices assume certain forms.     

Progress,problems and prospects for network research: Investigating relationships in the age of electronic communication technologies

Here we look at the past accomplishments, present difficulties, and future potentials of network research, stressing the unique advantages of investigating the new interactive technologies. A positive assessment of the past scientific accomplishments of network research is tempered by serious epistemological problems: inadequate attention to network theory, network sampling problems that restrict the generalizability of results, an underemphasis upon data-gathering and measurement, and an analytical shortchanging of the message content that flows through links over time. Partial solutions to certain of these difficulties may be found in network research on the new interactive communication systems.