Validity in an evaluation of Healthy Families Florida—A program to prevent child abuse and neglect

Evidence-based continuums have become a guide for identifying the level of evidence in evaluations of home visiting programs conducted to determine their effectiveness in preventing child abuse and neglect. While randomized controlled trials are required for the highest levels of evidence, quasi-experimental designs have also been specified as an appropriate alternative. Using a quasi-experimental evaluation of a home visiting program to prevent child abuse and neglect that adheres to the Healthy Families America model, this paper describes and illustrates how types of validity can be improved. More specifically, we address how threats to internal validity can be identified and reduced through statistical techniques; how construct validity may be strengthened using state records to measure the outcomes; and how external validity is affected by including or excluding study participants. After applying a variety of statistical adjustments to reduce selection bias, we found that the outcomes favored the home visiting program and increased after accounting for covariates that contributed to child abuse and neglect. This was true across the statistical techniques (traditional covariate and propensity score adjustment) used. For evaluations using quasi-experimental designs, recommendations relevant to the illustrations in the paper are also presented.     

A Review of Data Based Studies Addressing the Affects of Homosexual Parenting on Children's Sexual and Social Functioning

Part of the intellectual and political movement to provide homosexuals equal rights and opportunities under the law extends to homosexuals and bisexuals seeking custody of their biological children after divorce and/or custody of an adopted child. Family courts take into consideration parents' sexual orientation when considering custody decisions. To date, the data based research on children reared by homosexual parents is sparse. A search of the published literature identified fourteen data based studies which addressed some aspect of homosexual parenting and its effects on children. Each study was evaluated according to accepted standards of scientific inquiry. The most impressive finding was that all of the studies lacked external validity, and not a single study represented any sub-population of homosexual parents. Three studies met minimal or higher standards of internal validity, while the remaining eleven presented moderate to fatal threats to internal validity. The conclusion that there are no significant differences in children reared by lesbian mothers versus heterosexual mothers is not supported by the published research data base.     

Where local and national evaluators meet: unintended threats to ethical evaluation practice

The ethical work of program evaluators is based on a covenant of honesty and transparency among stakeholders. Yet even under the most favorable evaluation conditions, threats to ethical standards exist and muddle that covenant. Unfortunately, ethical issues associated with different evaluation structures and contracting arrangements have received little attention in the evaluation research literature. This article focuses on the unintended ethical threats associated with multitiered evaluation contexts. After briefly reviewing the various frames through which evaluation theory and ethics are commonly viewed, we discuss ethical challenges associated with multitiered evaluation designs including examples drawn from our evaluation projects. The article concludes with specific findings and recommendations for evaluators, grantors, grantees, and researchers.     

Shame, Embarrassment, and Formal Sanction Threats: Extending the Deterrence/Rational Choice Model to Academic Dishonesty

Following the lead of Grasmick and Bursik (1990), we propose that significant others and conscience both function as agents of social control to deter illicit behavior in a manner similar to that provided by the law and more formal, bureaucratic organizations. All three–conscience, significant others, and formal organizations–entail potential threats and costs (i.e., shame, embarrassment, and formal sanctions). Actors consider these potential threats and costs, and assess the certainty and severity of them, in deciding whether or not to engage in criminal or deviant behavior. In the research reported here, data from a sample of undergraduate students are employed to examine the effects of perceived certainty and severity of shame, embarrassment, and formal sanction threats of the self-reported prevalence of involvement in a variety of forms of academic dishonesty. The results suggest rather limited support for the deterrence/rational choice model, nearly all of which is due to the influence of conscience (shame).     

The theory-driven approach to validity

Current approaches for dealing with validity tend to focus on one type of validity and play down the importance of others. For example, internal validity is treated by Campbell and Stanley as “sine qua non,” whereas, Cronbach judges external validity to be prime importance in research. However, these single validity oriented approaches may achieve one type of validity at the expense of other types of validity.As an alternative, we propose a theory-driven approach to validity. Our central argument is that a model or theory should be formulated in a program evaluation and the modeling process should include the identification of potential threats to validity in research. In this sense, the purpose of applying a technique such as randomization is to strengthen rather than to replace the model or theory. That is, program evaluation should be model or theory-oriented instead of research design oriented. Randomization can be used in any research if it is applicable, but we need rely less on randomization as a safeguard to internal validity if the threats in a research can be adequately identified and controlled.Since the structural relationships among the relevant variables and intervening processes are carefully modeled, the theory-driven approach avoids the pitfalls of black-box evaluation and provides better understanding of the causal mechanisms underlying the relationships between treatment and effects. Unlike randomized experiments which aim only at internal validity, a theory-driven approach provides a general framework to deal with internal, external, construct and statistical conclusion validity.     

Program evaluation of services for the homeless : Challenges and strategies

The rapid growth of the homeless population in North America has given rise to a sharp increase in the number of task forces and studies aimed at setting up either programs for the homeless, or policies for controlling the phenomenon. Whenever there is a need for program evaluations in this process, numerous methodological and strategic challenges have to be faced. Due to the characteristics of the clientele, evaluation studies of services for the homeless call for innovative approaches. This paper reviews the research strategies (types of evaluations, designs, and indicators) that have been developed to assess programs for chronic inebriates and mentally ill homeless. The findings from previous evaluations are then summarized and their consequences for evaluation practice are considered.     

Assessing the impact of a local housing intervention program

Little reported research exists on the assessment of net impact of the large variety of housing intervention programs which have proliferated in the last twenty years. This paper evaluates one such program, which constructs new housing in inner-city neighborhoods of a Midwestern city, using a quasi-experimental a-posteriori design. Data were collected by survey technique from four types of homeowners, three of which served as constructed control groups for the program treatment. Regression analyses indicate that the program had an effect upon neighborhood satisfaction and satisfaction with the home, but no effect upon desire to move or spending for remodelling. Since the design was weak on internal validity, a number of threats to the validity of the results are systematically tested and, for the most part, rejected. The results are also found to be comparable in direction to those of a national evaluation of the Urban Homestead Demonstration Program.     

Using and interpreting analogue designs

Researchers in rehabilitation counseling and disability studies sometimes use analogue research, which involves materials that approximate or describe reality (e.g., written vignettes, videotaped exemplars) rather than investigating phenomena in real-world settings. Analogue research often utilizes experimental designs, and it thereby frequently possesses a high degree of internal validity. Analogue research allows investigators to exercise tight control over the implementation of the independent or treatment variable and over potentially confounding variables, which enables them to isolate the effects of those treatment variables on selected outcome measures. However, the simulated nature of analogue research presents an important threat to external validity. As such, the generalizability of analogue research to real-life settings and situations may be problematic. These and other issues germane to analogue research in vocational rehabilitation are discussed in this article, illustrated with examples from the contemporary literature.     

Stalking by former intimates: verbal threats and other predictors of physical violence

Scant empirical research has been conducted on the relationship between threats and violence. The purpose of this analysis is to assess the link between verbal threats of violence and actual physical violence against former intimate victims of stalking. The researcher interviewed 187 female former intimate stalking victims, asking respondents about various characteristics of their experiences. Responses to questions pertaining to threats; history of violence; stalkers' drug and alcohol use; frequency of phone calls, "following," and letters during stalking; and victims' age and education were analyzed as possible predictors of three violence-related dependent variables: (a) whether or not violence occurred, (b) the number of violent incidents during stalking, and (c) physical injury during stalking. The results of linear and logistic regression models reveal that, regardless of the measure of violence, there is an independent, moderate, and statistically significant correlation between verbal threats and subsequent violence. Regression coefficients for drug abuse and alcohol abuse were also statistically significant, but only in predicting physical injury during stalking.     

Internal validity in rehabilitation research

In experimental research, internal validity is the extent to which the researcher can conclude that the manipulation of the independent variable caused change in the dependent variable. Without causal inference, researchers and research consumers would not be able to meaningfully interpret the results of experiments. This article discusses (a) threats to internal validity in rehabilitation research an (b) experimental procedures that can aid rehabilitation researchers in maximizing the internal validity of their investigations.     

Panel accretion and external validity in adolescent substance use research

This study investigates the relationship of panel accretion to panel attrition and examines its impact on threats to external validity in adolescent substance use research. Panel accretion involves the addition of new participants to a panel design after initial baseline data has been obtained. Accretion occurs in longitudinal studies in which data is routinely gathered on all participants, as is often the case in prevention or epidemiological research on adolescent substance use. Accretion, attrition, and panel findings for three, 2-year panel samples in grades 7 through 9 are examined. The results indicate that accretion and attrition samples have an isomorphic relationship to one another on a range of dependent measures when drawn from the same population. The results are discussed in terms of their implications for adolescent substance use research, the examination of validity threats due to panel bias, and understanding the ecological validity of research findings.     

From comparative advantage to damage control: Clarifying strategic issues using swot analysis

The principal ingredient of strategic planning is SWOT analysis—the assessment of organizational strengths, weaknesses, opportunities, and threats. SWOT can be either a powerful management tool or a superficial and even misleading exercise. Much depends on the care with which decision makers integrate their analyses of internal and external factors affecting the future of the organization. This article presents a simple yet powerful framework for identifying and clarifying four types of strategic issues facing nonprofit organizations.     

Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.     

THERAPEUTIC INTERVENTIONS IN LIFE-THREATENING SITUATIONS*

Therapists who encounter life-threatening behavior in their clinical practice are at risk in providing interventions which do not take into account the systemic effects of their interventions. Specifically, therapists are likely to engage in “more of the same” behavior with clients who present with suicidal or homicidal threats. This paper presents two case studies that employed therapeutic methods derived from the systemic therapy of the Palo Alto group. In these case studies, the author illustrates the importance of the therapist's ability to speak the client's language in formulating an intervention that has the potential for reducing the likelihood of actual violence. The first case illustrates the use of the client's world view about her situation to assist her in dealing with domestic violence. The second case presented describes the use of systemic interventions in the case of a young woman known to have made homicidal threats. The author discusses the appropriateness, as well as the ethical issues, of the techniques described. Also offered, is a discussion clarifying the theoretical underpinnings of the clinical work presented and an amplification of some of the ideas originally presented by the Palo Alto group.     

IDENTIFYING SEPARATION THREATS IN FAMILY THERAPY

Separation anxieties are a part of normal family life, whether they arise from the progress of children toward independence or from accidental crises, which threaten the integrity of the family (Argles & Mackenzie, 1970). Family conflict tends to heighten such anxieties, and moreover they are often deliberately or unconsciously exploited, for a variety of reasons. This exploitation usually takes the form of separation threats, either explicit or implied in certain types of family transactions. Children are especially vulnerable to such threats, and experience them more commonly than may be supposed (Newson & Newson, 1968).     

THREAT AND PUNISHMENT IN PUBLIC GOOD EXPERIMENTS

Experimental studies of social dilemmas have shown that while the existence of a sanctioning institution improves cooperation within groups, it also has a detrimental impact on group earnings in the short run. Could the introduction of pre‐play threats to punish have enough of a beneficial impact on cooperation, while not incurring the cost associated with actual punishment, so that they increase overall welfare? We report an experiment in which players can issue non‐binding threats to punish others based on their contribution levels to a public good. After observing others' actual contributions, they choose their actual punishment level. We find that threats increase the level of contributions significantly. Efficiency is improved, but only in the latter periods. However, the possibility of sanctioning differences between threatened and actual punishment leads to lower threats, cooperation, and welfare, restoring them to levels equal to or below the levels attained in the absence of threats. (JEL C92, H41, D63)     

A user-focused model for the utilization of evaluation

It is proposed that the present low level of utilization of evaluation findings is traceable in part to their failure to address directly the information needs of a clearly specified decision maker. An alternative model proceeding from such specification is proposed here, with evaluation closely interwoven with the on-going innovation process. The model suggests a number of implications for the organizational role of evalutors, for the design of evaluations, and for directions for methodological development. In particular, we suggest that evaluation designs be assessed against possible “threats to utility” as well as against the traditional “threads to validity.”     

Are Population and Land Use Changes Perceived as Threats to Sense of Place in the New West? A Multilevel Modeling Approach

The Intermountain West's rapid changes in population growth and land use may be welcome to some, but others perceive such changes as threats to sense of place. The objective of this study is to assess whether New West and Old West contextual variables predict how agricultural landowners view threats to agricultural lifestyles and sense of place. We analyze survey data collected from 2,270 agricultural landowners in Colorado and Wyoming utilizing a multilevel regression model (MLM). We posit that this analytical approach is effective for evaluating hierarchal New West or Old West economic configurations that may otherwise be difficult to observe. Our study specifically examines whether population pressures threaten agricultural lifestyles in the amenity‐based New West or in Old West economic regimes with proclivity toward large‐scale agricultural production. Our results show that landowners in farming‐dependent counties and in high‐amenity areas express greater concern than other landowners surveyed about increases in population growth that could threaten an agricultural way of life. Furthermore, we demonstrate that these perceptions relate to whether individuals reside in New West or Old West counties. In summary, some of the contextual variables of New West and Old West economic structures predict whether individuals perceive population growth and land use changes as threats to sense of place.     

An evaluation of analysis options for the one-group-per-condition design. Can any of the alternatives overcome the problems inherent in this design?

This article addresses the analytic problems associated with a design in which one identifiable group is allocated to each treatment condition and members of those groups are measured to assess the intervention. Such designs are often called quasi-experiments if the groups are not randomized to conditions and group-randomized trials if the groups are randomized. They present special problems, and previous reports have argued against their use in efficacy or effectiveness trials. Even so, this design still appears with surprising frequency. This article presents the results from a new simulation study that underscores the analytic problems associated with this design.     

Peer Relationships and Indonesian Muslim Adolescents' Religiosity and Religious Coping: Selection and Influence

The religious similarity of adolescents and their friends can arise from selection or influence. Prior studies were limited because of confounds that arose from the ethnic and religious heterogeneity of the samples and the use of cross-sectional designs. SIENA was used in this two-year longitudinal study of 825 Indonesian Muslim high school students (445 girls; mean age = 16.5 years) to assess peer selection and influence as these pertained to religiosity and religious coping. The analyses yielded significant influence but not selection effects for both religiosity and religious coping. This study is an important methodological advance over prior research and although limited by correlational data, nevertheless, provides evidence that adolescents influence their peers' religiousness.