Causal inferences on the effectiveness of complex social programs: Navigating assumptions,sources of complexity and evaluation design challenges

This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention’s effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.     

Open systems evaluation and the logic model: Program planning and evaluation tools

This paper describes two case management projects and efforts to evaluate outcomes. Lessons learned from the first effort suggest that a tool referred to as the logic model and a particular approach to evaluation called the “open systems” model might be useful to planners and program evaluators. The logic model provides a means of presenting conditions a program is intended to address, activities that constitute a program, short term outcomes resulting from program activities and long term impacts. The open systems perspective emphasizes evaluation as a tool to be used to achieve program objectives as opposed to establishing cause and effect relationships. This paper also describes how the logic model and open systems evaluation were used to facilitate the development of an evaluation plan for a project designed to assist homeless, substance abusing, pregnant women.     

QCA as competing or complementary method? A qualitative comparative analysis approach to protest event data

Studies of state repression of protest have focused on theories of threat and weakness, in which states repress movements that threaten state authority or elite interest or movements that lack organizational or political strength. Empirical studies have most often used regression analysis of protest-event datasets. This paper proposes qualitative comparative analysis (QCA) as an alternative approach to protest-event data that retains more qualitative complexity and captures the conjunctural and heterogeneous nature of causation during events. The paper applies both methods to protest data from the United States (1963–1973). While both methods provide strong evidence for the threat and weakness hypotheses, QCA more effectively illustrates how combinations of threat and weakness factors increase the risk of repression. The paper argues that QCA is a viable alternative approach to event data, but it should also be seen as a valuable complementary method that can improve regression-based approaches.     

Can sample size in qualitative research be determined a priori?

There has been considerable recent interest in methods of determining sample size for qualitative research a priori, rather than through an adaptive approach such as saturation. Extending previous literature in this area, we identify four distinct approaches to determining sample size in this way: rules of thumb, conceptual models, numerical guidelines derived from empirical studies, and statistical formulae. Through critical discussion of these approaches, we argue that each embodies one or more questionable philosophical or methodological assumptions, namely: a naïve realist ontology; a focus on themes as enumerable ‘instances’, rather than in more conceptual terms; an incompatibility with an inductive approach to analysis; inappropriate statistical assumptions in the use of formulae; and an unwarranted assumption of generality across qualitative methods. We conclude that, whilst meeting certain practical demands, determining qualitative sample size a priori is an inherently problematic approach, especially in more interpretive models of qualitative research.     

A research design for generalizing from multiple case studies

Many evaluations would be more useful to decision makers if their results were both explanatory and generalizable. In view of the limitations of traditional methods in meeting both these goals at the same time, the authors describe a research design explicitly intended for generalizing about the range of applicability of “explanatory patterns” (within a designated “target” population of cases). The paper presents an overview of the main features of a “multiple case study design,” shorthand for a multiple site, structured case study design. It then discusses the nature of explanatory patterns, how case study investigators pursue and recognize valid patterns, and how an analyst can apply the same logic to cross-site analysis in order to make valid inferences about the limiting conditions under which particular explanatory patterns apply to particular populations.     

The Impact of Societal-Level Institutional Logics on Hybridity: Evidence from Nonprofit Organizations in England and France

We examine how societal-level institutional logics impact the way in which hybridity develops in nonprofit organizations using international, comparative and qualitative case studies of community regeneration organizations in England and France. The research applies theoretically based conjectures about types of hybridity to empirical data generated from 20 interviews, document analysis and observation in five nonprofits in the city of Lyon and five in Sheffield. We find that the French nonprofits are ‘blended’ hybrids that integrate state and community institutional logics, while ‘assimilated’ hybrids combining state, community and market logics are found in the English cases. Undertaking contextually situated analysis of institutional logics generates new knowledge on the influences on nonprofits’ rules, practices and narratives, so improving the level of knowledge about, and capacity to manage, this sector.     

Roadmap for a communication maturity index for organizations—Theorizing,analyzing and developing communication value

This study employed a constitutive (CCO) communication approach to advance the dynamic concept of communication maturity for the purpose of theorizing, analyzing, and developing communication value on an organizational level. A literature review resulted in six theoretical areas related to communication maturity: understanding, function, organization, prerequisites, competence and practices including assessments of communication. These were analyzed in a qualitative comparative case study including 85 key stakeholders from eleven organizations. Findings illustrated the relevance of the theoretical areas to stakeholders’ perceptions of their respective organizations’ communication maturity. A first version of a Communication Maturity Index including four levels of maturity; immature, emerging, established, and mature is proposed. Based on the qualitative analysis, organizations’ communication maturity levels varied from emerging to mature. Findings of this study are limited to participating organizations and interviewed stakeholders. The CMI can be further developed in quantitative studies to investigate the constitutive role of communication in organizations and to be used in practice to develop higher levels of communication maturity.     

A Multi‐method Approach to the Study of School Class Size Differences

This paper describes a multi‐method approach to data collection used in a large‐scale seven‐year longitudinal study of the effect of class size differences on classroom processes. There were five main types of process and the paper concentrates on the results from one of these: teaching interactions with pupils. It was felt that integration of results from quantitative and qualitative methods would help reconcile inconsistencies and limitations in previous research on this topic. Quantitative information was collected from systematic classroom observations and teacher completed time estimates to address relationships between class size and teacher time allocation, and teacher and pupil behaviour in class. But we also wanted a more qualitative assessment of relationships between class size and teaching through the use of methods that captured practitioners’ experiences, and through detailed case studies. Examples of integrated results from these four methods are given, along with some reflections on lessons learned.     

Multiple Segment Factorial Vignette Designs

The multiple segment factorial vignette design (MSFV) combines elements of experimental designs and probability sampling with the inductive, exploratory approach of qualitative research. MSFVs allow researchers to investigate topics that may be hard to study because of ethical or logistical concerns. Participants are presented with short stories about hypothetical characters. These stories comprise 2 – 5 segments in which randomly assigned dimensions are introduced sequentially. Researchers control the occurrence and timing of when dimensions are presented. Respondents are asked questions after each segment. MSFVs may include written, oral, and visual stimuli. These efficient designs elicit quantitative data about causal relations and qualitative data within complex, multidimensional contexts. We discuss how to develop MSFV and suggest ways to avoid problems.     

How do Spanish disability support offices contribute to inclusive education in the university?

This article analyzes the functioning of disability support offices and their contribution to inclusive education in seven Spanish universities from the perspective of staff. Using a qualitative methodology, interviews with office staff were conducted, and data were analyzed through an inductive coding system. The results are organized around five themes: characteristics of disability support offices, staff training, functions performed by different services, barriers and opportunities identified by office staff, and proposals to improve attention given to disabled students. Information gathered leads to the conclusion that the work carried out in disability support offices must receive support from universities, as these offices are a key element for the access and retention of students with disabilities in the university and for the successful completion of their studies.     

A comparative study on Italian new communities abroad: the cases of Tunisia and South Africa1

This paper presents two case studies about Italian new communities abroad: one Tunisian and one South African. These case studies were conducted – in a larger international research project on 14 countries – with different types of interview: focus interviews, in-depth interviews, focus groups, etc. The mixed interview modes brought to light how – with a different research strategy (methodological triangulation): quantitative and qualitative data – our complex social experience can be inquired into, in particular migration processes. The authors decided to utilise semi-standardised and non-standardised techniques. This paper discusses how the mixed research logic – utilised in both of these cases – is very productive in terms of knowledge. It seems that focus group interviewing techniques (and others) have been used progressively more and more over the past years as both a self-contained method for conducting qualitative research and in combination with other research methods. In the authors’ opinion, mixed-methods research (or multiple research strategies) represents a smaller corpus of literature that is getting increased attention. The essays in our paper are grouped along four themes: (a) economic development and globalisation; (b) perceptions and discourses; (c) the new migration phenomenon; and (d) fragmentation of identity. The authors believe that rebuilding a morceau de vie of Italian emigration helps to better analyse the role of the communities of immigrants in the construction of national processes. As a result of this, their survey introduces reflections on theoretical—methodological logic and concurs with placing such aspects in a global and social structural dimension. However, the universal and social structural dimension can be extended to the specific case. Its contribution in the first part investigates a variety of linkages of migration with broader processes of social–economic development and social globalisation. That global patterns of migration and the processes of globalisation are linked in various ways has long been recognised. The second part of the paper stresses the meaning of perceptions and discourses linked with the migration phenomenon and underlines the importance of an accurate methodological approach in order to provide new sociological points of view and appropriate research strategies.     

Housing Concepts For and By the Elderly: From Subjects of Design to a Design Resource

ABSTRACT

The perception of elderly residents as a homogeneous group is challenged by the diversification of lifestyles and emphasis on individuality in housing. Recently, there has been a rise in Finland in housing projects initiated by groups of elderly people, where the seniors themselves are seeking to fill in qualitative gaps in the offering of housing. This article presents a comparative analysis of four senior housing concepts that represent both a producer-driven (“for the elderly”) and a resident-driven (“by the elderly”) approach. The study shows that elderly people can be a resource for residential development and that this necessitates a different “design logic” than in conventional senior housing.     

Investigating the Effects of Pre‐school Provision: Using Mixed Methods in the EPPE Research

The Effective Provision of Pre‐school Education (EPPE) project is a longitudinal study of young children between the ages of 3 and 7 years. Research began in 1997 and both quantitative and qualitative methods were used to explore the effects of pre‐school education on children’s attainment and social/behavioural development at entry to school, and of any continuing effects on such outcomes at the end of Key Stage 1 (age 7). The sample included over 3,000 children and 141 pre‐school centres drawn from six different types of providers. In addition, a home sample of children who had not attended a pre‐school centre was included. In order to identify the processes that might influence centre effectiveness, observational data were collected and centre managers interviewed. In‐depth case studies of 12 more effective centres were used to explore and illuminate the processes, particularly in relation to pedagogy, associated with better child outcomes. A mixed method approach was adopted to simultaneously answer both confirmatory and exploratory questions, to support user engagement and provide triangulation. The paper argues that the mixed method approach supported the development of stronger research inferences.     

Narrating inequality,eliding empire

Piketty’s call for a historically informed, global analysis of inequality is timely, as is the need for a corresponding transformation of our current politics. However, I believe there to be a fundamental flaw in his analysis which reproduces a Eurocentric approach to understanding global inequality. The key issue is that Piketty organizes his historical comparative analysis in terms of inequality within nations. Yet, the polities he is discussing were rarely just nations over the long durée. Rather, they were imperial formations constituted by a colonizing state and the territories and populations that were incorporated. His approach separates the logic of what he calls the modern proprietarian regime of inequality from enslavement and colonialism when both were integral to it. In contrast, I argue for them to be seen as necessarily interconnected with a lasting legacy in contemporary configurations of global inequality.     

“Paradigms lost”: Some thoughts on choosing methods in evaluation research

Patton agrees with us concerning the need for a catholic approach to method choices in evaluation. However, Patton criticizes us for setting up a straw man when we argued that quantitative and qualitative methods are not inevitably linked to different “paradigms” and that allegiance to a particular paradigm does not force the evaluator into a choice between mutually exclusive qualitative and quantitative methods. The first part of this article shows how some of the past literature creates the impression both of an inevitable method-paradigm linkage and of a forcedchoice between qualitative and quantitative paradigms. It is argued that such impressions could have pernicious consequences for evaluation practice by artificially restricting method choices. In the final section, we consider some of the difficulties of making method choices and suggest that the purposefully developed strengths of different methods should not be completely ignored.     

Predictors of children in foster care being adopted: A classification tree analysis

When returning home is not a viable option, adoption is the primary means by which youth in substitute care achieve permanency. Therefore, understanding the factors that impact adoption is critical to both evaluating and improving the child welfare system. Prior research has mostly studied “main effects” in regard to adoption rates and has not explored the ways in which youth and foster family variables might interact in complex ways to predict adoption. This study uses a classification tree analysis approach known as Optimal Data Analysis (ODA) to predict probability of adoption in 2003 using Adoption and Foster Care Analysis and Reporting System (AFCARS) data. 30,000 adopted youth and 30,000 non-adopted youth were randomly selected for participation in the study. Similar to previous studies, univariate analyses revealed that age, foster parent race/ethnicity, foster parent marital status, and number of previous placements all predicted probability of adoption. However, going further, several combinations of individual variables (multivariate ODA) improved prediction accuracy (e.g., age × structure of foster family × number of previous placements). Results also suggested that the impact of the state on adoption rates varied when essentially controlling for youth and family variables, supporting concerns regarding the crude use of AFCARS data when comparing states.     

Understanding Mills via Mill-type methods: An application of qualitative comparative analysis to a study of labor management in southern textile manufacturing

A number of recent writers have advocated a computer-based approach to the analysis of qualitative data that leans on John Stuart Mill's method of agreement and the indirect method of difference. Those discussions, however, remain fairly abstract and provide little insight into how analyses based on Mill-type methods unfold and their relative costs and benefits. In this paper we describe the logic and use of one prominent Mill-type software program called QCA (Qualitative Comparative Analysis) in the context of an analysis of evidence drawn from twenty-two interviews with managers in textile plants. Our substantive aim is to understand labor-management practices in Southern textile manufacturing plants; our methodological aim is to illustrate the strengths and weaknesses of the Mill-type logic in action.     

The dangers of political naiveté and excessive complexity in evaluating delinquency prevention programs

The current demand for the assessment of programs designed to modify delinquent behavior may actually decrease the likelihood of achieving important long range goals. This article argues that the traditional experimental-control group procedures used in the evaluation of delinquency prevention programs are rarely successful, generate strains which have negative consequences, create hostility toward attempts to gain knowledge, and fail to create superior programs in the future. Furthermore, complex programs increase the likelihood of stress more than simple ones. If sophisticated research skills were directed toward the analysis of data gathered from larger numbers of cases and from more readily available data sources, many of these negative consequences could be minimized. The final theme is that the way in which findings are reported should be treated as an important factor in its own right. Objective, “hard-nosed” assessment may in fact damage promising projects without altering ineffective but entrenched programs.     

Social Survey Research1 in Japanese Sociology

Abstract: In this paper, we look at the history of social survey development in Japanese sociology. First, the history of social research in Japan before World War II is explored. Second, the introduction of survey research to Japan during the American occupation after World War II is examined, and third, the present state and roles of social survey research in Japanese sociology is discussed. Social research was introduced as an administrative tool for the government. Sociology and social research were developed under British empiricism and American pragmatism, but Japanese academia has been based on a metaphysical approach. Social research introduced as a practical tool long had difficulty in being accepted by Japanese academia. For this reason, most sociologists in universities did not use social survey research for practical purposes, but pursued qualitative methodologies for analyzing data to gain academic prestige even after Social Stratification and Mobility (SSM) and Sabro Yasuda's research projects spread social survey methods in the field of Japanese sociology. Such academics did not think that findings acquired through qualitative case studies had to be confirmed through quantitative data to serve a practical purpose, nor did they believe that quantitative data could be better understood when examined along side qualitative data. Social survey methods have been opposed by those who have favored case‐study analysis methods in Japanese sociology. Needless to say, this opposition is fruitless. I propose that professional sociologists in Japanese universities should use social survey research for practical problems more frequently. This is the best way to establish sociology and social research as a science in Japanese society.