Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.     

Validity in an evaluation of Healthy Families Florida—A program to prevent child abuse and neglect

Evidence-based continuums have become a guide for identifying the level of evidence in evaluations of home visiting programs conducted to determine their effectiveness in preventing child abuse and neglect. While randomized controlled trials are required for the highest levels of evidence, quasi-experimental designs have also been specified as an appropriate alternative. Using a quasi-experimental evaluation of a home visiting program to prevent child abuse and neglect that adheres to the Healthy Families America model, this paper describes and illustrates how types of validity can be improved. More specifically, we address how threats to internal validity can be identified and reduced through statistical techniques; how construct validity may be strengthened using state records to measure the outcomes; and how external validity is affected by including or excluding study participants. After applying a variety of statistical adjustments to reduce selection bias, we found that the outcomes favored the home visiting program and increased after accounting for covariates that contributed to child abuse and neglect. This was true across the statistical techniques (traditional covariate and propensity score adjustment) used. For evaluations using quasi-experimental designs, recommendations relevant to the illustrations in the paper are also presented.     

Ethical practice and evaluation of interventions in crime and justice. The moral imperative for randomized trials

In considering the ethical dilemmas associated with randomized experiments, scholars ordinarily focus on the ways in which randomization of treatments or interventions violates accepted norms of conduct of social science research more generally or evaluation of crime and justice questions more specifically. The weight of ethical judgment is thus put on experimental research to justify meeting ethical standards. In this article, it is argued that just the opposite should be true, and that in fact there is a moral imperative for the conduct of randomized experiments in crime and justice. That imperative develops from our professional obligation to provide valid answers to questions about the effectiveness of treatments, practices, and programs. It is supported by a statistical argument that makes randomized experiments the preferred method for ruling out alternative causes of the outcomes observed. Common objections to experimentation are reviewed and found overall to relate more to the failure to institutionalize experimentation than to any inherent limitations in the experimental method and its application in crime and justice settings. It is argued that the failure of crime and justice practitioners, funders, and evaluators to develop a comprehensive infrastructure for experimental evaluation represents a serious violation of professional standards.     

Let’s be honest: A review of experimental evidence of honesty and truth-telling

Honesty toward strangers can be considered an important norm of any given society. However, despite burgeoning interest in honesty among experimenters, the heterogeneous nature of prior experimental designs obfuscates our understanding of this important topic. The present review of 63 economic and psychological experiments constitutes the first attempt to compare findings across a range of honesty experiments. Our findings across experimental designs suggest the robust presence of unconditional cheaters and non-cheaters, with the honesty of the remaining individuals being particularly susceptible to monitoring and intrinsic lying costs.     

Sozialer Status und prosoziales Handeln: Ein Quasi-Experiment im Krankenhaus

Sociological, social psychological and economic research on the nexus between socioeconomic status and prosociality has so far provided contradictory findings. Some studies suggest that actors with a high socioeconomic status act more egoistically than actors with a lower socioeconomic status. Other studies find the opposite to be true. In contrast to previous research, which has worked with one-dimensional measures for socioeconomic status, this study examines prosocial behavior among occupational groups that have regular real-life contact in their workspace. About 150 hospital employees (physicians, nursing staff and nursing students) participated in experiments on altruistic giving in dictator games. The findings are surprisingly strong and clear-cut: Actors with higher social status act more prosocial than low-status actors. Furthermore, we find hardly any in-group effects, which have been repeatedly postulated. Our findings support the claim that high status promotes prosocial behavior. Also, they indicate that the inconclusive and in part contradictory findings provided by previous research stem to a considerable degree from deficient measures of social status and problematic experimental designs.     

WAGES,EMPLOYMENT, AND STATISTICAL DISCRIMINATION: EVIDENCE FROM THE LABORATORY

We report results from laboratory experiments designed to examine statistical discrimination. Our design expands upon existing research by generating data both on wage contracts and unemployment rates of directly competing worker groups. We find some evidence for statistical wage discrimination against workers having an identical expected productivity but a higher productivity variance. However, those same subjects are less likely to be unemployed, suggesting that our employer‐subjects view hiring choice and wage contracts as substitutable. A clear implication is that field data discrimination estimates based on wages alone may overestimate the true impact of such discrimination. (JEL C90, J71)     

Cohabitors’ Reasons for Living Together,Satisfaction with Sacrifices,and Relationship Quality

The purpose of our study was to identify both negative and positive associations of cohabitation with relationship quality. Using a sample of 280 cohabitors, we examined how reasons for cohabitation (i.e., spending time together, testing the relationship, and convenience) are associated with relationship quality (i.e., commitment, satisfaction, ambivalence, and conflict) as moderated by satisfaction with sacrifices. Results showed that a higher score on spending time together as a reason for cohabitation was linked with greater commitment and satisfaction and lower ambivalence and conflict, even when cohabitors reported lower satisfaction with sacrifices. In contrast, a higher score on testing the relationship as a reason for cohabitation was linked with more ambivalence regardless of the level of satisfaction with sacrifices. Finally, a higher score on convenience as a reason for cohabitation was linked with lower commitment, including when cohabitors reported lower satisfaction with sacrifices. Collectively, our results are important in demonstrating the positive and negative aspects of cohabitation in association with relationship quality and when satisfaction with sacrifices moderates such associations.     

Natural experiment methodology for research: a review of how different methods can support real-world research

In particular research domains, the randomized control trial (RCT) is considered to be the only means for obtaining reliable estimates of the true impact of an intervention. However, an RCT design would often not be considered ethical, politically feasible, or appropriate for evaluating the impact of many policy, programme, or structural changes common in public health research. As such, researchers must use alternative yet robust research methods for determining the impact of such interventions. The evaluation of natural experiments (i.e. an intervention not controlled or manipulated by researchers), using various experimental and non-experimental design options can provide an alternative to the RCT. The following review highlights (a) the importance of evaluating natural experiments; (b) design considerations associated with evaluating natural experiments; (c) methods for reducing bias in natural experimental studies; and (d) the potential benefits of targeted systems to enable natural experiments in emerging priority domains moving forward.     

An analysis of the efficacy of infant intervention programs with organically handicapped children

Investigations (N = 49) which utilized pre-experimental, quasi-experimental or true experimental designs to assess the effectiveness of interventions with organically impaired infants are reviewed and analyzed. Each of the designs employed are examined according to how well they protected against threats to internal validity. Plausible alternative explanations for obtained outcomes are illustrated. It was concluded that the majority of studies, because of their failure to control for threats to internal validity, yielded results which are fundamentally uninterpretable. The use of methodologically more rigorous designs in evaluation studies is recommended to assess the efficacy of intervention efforts designed to prevent or ameliorate the adverse effects of constitutional defects.     

Maximizing both external and internal validity in longitudinal true experiments with voluntary treatments: The “combined modified” design

Human services interventions are most rigorously evaluated with true experimental designs in longitudinal experimental field trials (LEFTs). However, differential self-selection or attrition often pose a serious threat to the LEFTs internal validity. This threat can be largely overcome by describing all conditions in advance to prospective subjects and securing their agreement to participate in and complete whichever condition is selected at random by a Lottery. This solution, however, in turn then poses the external validity problem that the program's effects on those who would participate in a Lottery may well be different than its effects on those who would participate in any single condition. In the present paper, we describe a new design, termed the Combined Modified Design, which assesses and overcomes this problem. This new design, in which a modified version of the Randomized Invitation Design (in which only one condition, assigned at random, is described to a potential subject, but outcome measures are obtained on everyone) is combined with the Lottery LEFT, is illustrated with a hypothetical example.     

Relative Deprivation and Collective Protest: An Impoverished Theory?*

Recently, social psychological theories of social movements such as relative deprivation (RD) have come under heavy censure. An examination of the theoretical literature on RD indicates that many of the critics’remarks seem to be predicated on an inadequate and superficial interpretation of RD. A detailed examination of many of the studies purporting to operationalize RD reveals, however, that many of the criticisms retain their pertinence when evaluating the empirical work. At this point it seems premature to declare RD moribund. Instead, directions for future studies that may lead to a more accurate appraisal of RD are recommended, since both social psychological and mobilization theories are necessary for a comprehensive understanding of collective behavior. Ghetto dwellers have much to resent about the way the outside world treats them: poor jobs, unemployment, unfair practices on the part of many employers, high rents for unsatisfactory housing, inadequate schools and health and sometimes brutal police work, the poor performance and sharp practices of many businesses aiming at ghetto customers, as well as a host of major and minor expressions of prejudice and discrimination which may confront a member of the black minority as he goes about his everyday social traffic in American society. Although such circumstances do not hit every member of the community with equal force, they provide each ghetto dweller with some basis for discontent, and probably they all play some role in the accumulation of grievances which may finally result in a rising. (Hannerz, 1971:324)     

The citizen-consumer hybrid: ideological tensions and the case of Whole Foods Market

Ethical consumer discourse is organized around the idea that shopping, and particularly food shopping, is a way to create progressive social change. A key component of this discourse is the “citizen-consumer” hybrid, found in both activist and academic writing on ethical consumption. The hybrid concept implies a social practice – “voting with your dollar” – that can satisfy competing ideologies of consumerism (an idea rooted in individual self-interest) and citizenship (an ideal rooted in collective responsibility to a social and ecological commons). While a hopeful sign, this hybrid concept needs to be theoretically unpacked, and empirically explored. This article has two purposes. First, it is a theory-building project that unpacks the citizen-consumer concept, and investigates underlying ideological tensions and contradictions. The second purpose of the paper is to relate theory to an empirical case-study of the citizen-consumer in practice. Using the case-study of Whole Foods Market (WFM), a corporation frequently touted as an ethical market actor, I ask: (1) how does WFM frame the citizen-consumer hybrid, and (2) what ideological tensions between consumer and citizen ideals are present in the framing? Are both ideals coexisting and balanced in the citizen-consumer hybrid, or is this construct used to disguise underlying ideological inconsistencies? Rather than meeting the requirements of consumerism and citizenship equally, the case of WFM suggests that the citizen-consumer hybrid provides superficial attention to citizenship goals in order to serve three consumerist interests better: consumer choice, status distinction, and ecological cornucopianism. I argue that a true “citizen-consumer” hybrid is not only difficult to achieve, but may be internally inconsistent in a growth-oriented corporate setting.     

Gender differences in young adults’ inclination to sacrifice career opportunities in the future for family reasons: comparative study with university students from Nairobi,Madrid, and Reykjavik

This article addresses the question of to what extent young people show an inclination to accept some sacrifice in their career progression in the future in order to reach a better work–family balance. Data come from a survey conducted among a sample of 2383 university students who attended three universities: University of Nairobi, University of Iceland, and Complutense University of Madrid. After building a set of indicators about career and family involvement aspirations of respondents, and after conducting a statistical and regression analysis, this research shows that young women (on average) still have a greater predisposition than young men to make sacrifices in the future in their working careers in order to achieve a better work–family balance. Moreover, having a high degree of leadership aspirations and belonging to an egalitarian household tend to reduce the inclination to sacrifice career opportunities, whereas having a high inclination to be involved in childcare in the future and having the perception of a future work–family conflict tend to increase it. Gender attitudes have a differential effect on female and male students: having traditional gender attitudes tends to increase the inclination to sacrifice career opportunities in the case of female students and reduce it in the case of male students.     

Context and interpretation in laboratory experiments: The case of reciprocity

The existing literature acknowledges that a mismatch between the experimenter’s and the subjects’ models of an experimental task can adversely affect the interpretation of data from laboratory experiments. We primarily focus on experiments designed to test a hypothesis by comparing behavior across two or more games and highlight the drawbacks of using a between-subjects design or a within subjects design. An alternative design for laboratory experiments is proposed which may alleviate this concern especially in studies of social preferences. We argue that the proposed design does not introduce any conceptual problems in addition to those that are already present in the between and within subjects designs and may alleviate some of the problems inherent to these designs in certain cases. We also use the proposed design to answer some questions that have attracted continued attention in the literature on social preferences in general and reciprocity in particular.     

Relational Sacrifices about Intimate Behavior and Relationship Quality for Expectant Cohabitors

ABSTRACT

Using interdependence theory, we examined how relational sacrifices specific to intimacy were associated with positive (commitment, satisfaction, love, and maintenance) and negative (ambivalence and conflict) relationship quality among expectant, unmarried cohabitors (n = 69 individuals), because this group may struggle relationally. We examined how often individuals made sacrifices for their partners and their partners’ perceived awareness of their sacrifices (i.e., individuals’ perceptions that their partners are aware that intimate sacrifices had been made for them) for both members of the couple using Actor Partner Independence Models. We controlled for children from previous relationships, unplanned pregnancy, gender, education, and race/ethnicity. Results showed that frequency of intimate sacrifices was not associated with any aspects of relationship quality. Instead, as expected, perceived partner awareness of intimate sacrifice was associated with greater commitment, satisfaction, love, and maintenance for individuals; lower ambivalence for individuals; and lower conflict for both individuals and their partners. These results suggest that for expectant cohabitors, perceived partner awareness of intimate sacrifices is central to achieving more positive, and less negative, relationship quality.     

On the relationship between true preference and actual choice

Two different reasons are considered for the divergence of actual choice from true preferences, with the latter assumed fully transitive. These are the presence of finite perception and incomplete expression. The two have a dual correspondence, with one yielding true preference more stringent, the other less stringent, than actual choice. In each case, actual choice can be rationalised; to yield a closer approximation to true preference for finite perception choice, given the alternatives over which choice is made, and to yield a minimal transitive ordering for incomplete expression.     

A validity scale for the Sharp consumer satisfaction scales.

Because of the apparently inflated scores commonly reported for consumer satisfaction with mental health services, we developed a 5-item validity scale for the SHARP questionnaire. The scale discriminated significantly (P less than .001) between those former clients who offered suggestions for improvement of services and those who did not (N = 38). In another experiment, the scale significantly improved the ability to predict the 46 clients' true scores from their obtained scores via stepwise regression (p less than .01). Weights were obtained from that sample of former clients, which significantly reduced the deviation of the obtained satisfaction scores from the true scores, with all p less than .05. In two successive experiments (N = 49 and 26), weights were applied to new samples, significantly reducing deviation from the criterion scores for 5 of the 6 scales (p less than .05), and new weights were then derived. A final set of weights was derived from the combined samples of the three experiments, (N = 121) which significantly improved prediction of the criterion and lowered the mean on all scales, with all p less than .05.     

Evaluation research in social work

Evaluation research in social work is becoming more varied and is showing signs of greater sophistication. It is taking on new forms and functions in such innovations as single subject designs and developmental research. Field experiments have begun to look more like scientific undertakings aimed at advancing knowledge and less like bad marriages between tight designs and loose programs. Agencies have made a start toward making evaluation a part of routine operations through the introduction of accountability systems and other evaluation mechanisms. In so doing they may be turning away from occasional, splashy projects to a more humdrum, but probably more efficient and useful approach to evaluation.     

Pratique sacrificielle et activité scientifique: Enquête ethnographique dans un laboratoire de physiologie

Experimental physiologists use the word “sacrifice” to describe the putting to death of laboratory specimens. Drawing on observations of laboratory practices and on religious sacrifices described by sociologists, the argument is made that, beyond being a mere metaphor, “sacrifice” refers to a practice that activates a specific set of meanings. In this way, experimentation can be conducted while, at the same time, treating laboratory animals with respect.