Measurement biases in user involvement research

Considerable research has been conducted to demonstrate user involvement's effect on information system success. User involvement and system success typically have been measured by asking users for their perceptions of these variables. This paper reports on a field study conducted to investigate the possibility that this approach to measuring study variables tends to overstate the benefits of user involvement. The link of user involvement to user satisfaction is found to be significantly weaker when user involvement is assessed by systems analysts than when it is self reported. Further, this difference is found to be greater for systems with few users than for systems with many users. The findings suggest that common method variance and self-serving bias may have overstated the apparent benefits of user involvement in past research on information systems. Suggestions for future research are presented.     

Project success' effects on substance use-related attitudes and behaviors: a randomized controlled trial in alternative high schools

Using a randomized controlled effectiveness trial, we examined the effects of Project SUCCESS on a range of secondary outcomes, including the program's mediating variables. Project SUCCESS, which is based both on the Theory of Reasoned Action and on Cognitive Behavior Theory, is a school-based substance use prevention program that targets high-risk students. We recruited two groups of alternative high schools in successive academic years, and randomly assigned schools in each group to either receive the intervention (n = 7) or serve as a control (n = 7). Students completed surveys prior to and following the administration of the program, and again 1 year later. Although participation in Project SUCCESS significantly increased students' perceptions of harm resulting from alcohol and marijuana use, students in the control group reported greater increases in peer support. We also found conflicting evidence in two opposing trends related to students' perceptions of the prevalence and acceptability of substance use. Therefore, the effects of Project SUCCESS on substance use-related beliefs and behaviors must be considered mixed.     

Improving the sensitivity of needs assessment for substance abuse prevention planning: the measurement of differential severity of consequences for individual substance types

There is an absence of systematic, comparative research examining the negative consequences that are experienced as a result of using specific substances. Further, techniques typically used for needs assessment (i.e., prevalence proportions) do not take into account the probability of experiencing a negative consequence as a result of using specific substances. An approximated severity index is proposed that: a) takes into account the probability of experiencing negative consequences as a result of using specific substances; and b) is comparable across substances. Data from the NSDUH and the ADSS are used to demonstrate these techniques. The findings suggest that substances typically considered priorities based on prevalence proportions are not the same substances that have a high probability of causing negative consequences. The policy implications of these findings are discussed.     

The effects of Project SUCCESS on student academic performance: a quasi-experimental study

Project SUCCESS (PS) is a substance use prevention program that targets indicated high school students. We used archival data to explore the program's effects on students' academic achievement and disciplinary problems. It is essential to demonstrate such effects, if prevention curricula are to survive in schools that face multiple competing demands for class time. Using a quasi-experimental design that compared students exposed to the program to those not exposed in five schools, we examined the effects of PS on grades, absenteeism, and disciplinary problems. Using longitudinal analyses that linked individual students' program exposure data to their academic performance, we found that PS had a positive impact on both grades and absenteeism for two years following exposure. We note that these findings are particularly fortuitous, as PS was not designed to affect these outcomes. We recommend that future evaluations of school-based prevention curricula routinely include academic performance.     

A short-term, quasi-experimental evaluation of D.A.R.E.'s revised elementary school curriculum

We present the short-term results of a quasi-experimental evaluation of the revised D.A.R.E. (Drug Abuse Resistance Education) curriculum. Study outcomes examined were D.A.R.E.'s effects on three substances, namely students' lifetime and 30-day use of tobacco, alcohol, and marijuana, as well as their school attendance and academic performance. The study comprised students in 17 urban schools, each of which served as its own control; 5th graders in the 2006-2007 school year constituted the comparison group (n = 1490), and those enrolled as 5th graders in the 2007-2008 school year constituted the intervention group (n= 1450). We found no intervention effect on students' substance use for any of the substance use outcomes assessed. We did find that students were more likely to attend school on days they received D.A.R.E. lessons and that students in the intervention group were more likely to have been suspended. Study findings provide little support for the implementation and dissemination of the revised D.A.R.E. curriculum.     

The Economic Benefits of Marriage and Family Strengthening Programs

Marriage and family strengthening programs have historically had small magnitude effects on changing relationship outcomes. The present study explores the possibility that although these statistical effects are small, they can be shown to represent meaningful financial impacts. Secondary data from 2092 control and 2042 intervention couples who were married with children and participating in the Supporting Healthy Marriage (SHM) project were analyzed. Intervention participants had the opportunity to receive standardized curricula (e.g., PREP) and marital counseling. Similar to meta-analytic findings, conventional analysis of these data found only a modest impact on proximal relationship outcomes (e.g., satisfaction). Offering counseling in conjunction with curricula, although costlier and only demonstrating marginally significant improvements in averting divorce, appears to offer a substantial financial return on investment. The implications of these findings are discussed.