Counselor confounds in evaluations of vocational rehabilitation methods in substance dependency treatment

Evaluation research on vocational counseling in substance dependency treatment should distinguish between the effects of counselors and counseling methods on clients' employment outcomes. Three experimental designs permit investigation of possible confounds between these types of effects: (a) nested designs (each counselor delivers one counseling method so counselors are nested under methods), (b) crossed designs (each counselor delivers all counseling methods so counselors are crossed with methods), and (c) no-treatment control group designs (experimental group's counselors all deliver the same method). Each design is optimal for one stage of evaluation research. No-treatment control group designs are best for exploratory evaluations of new types of vocational counseling. Nested designs are best for outcome evaluations of different types of interventions. Crossed designs are best for (causal) process evaluations of counseling methods of demonstrated efficacy. Despite the importance of methodological issues and problems, vocational rehabilitation in substance dependency treatment has a greater need for stronger interventions than better evaluation designs.     

Methodological strategies for evaluating youth gang prevention programs

This article reflects on common challenges and lessons learned during the evaluation of gang prevention programs based on case studies of three federally funded Canadian programs. Elements of evaluation design, implementation, data analysis and reporting of results are discussed. More specifically, the article highlights issues that occur when evaluating community projects focused on preventing extreme risks for violence and the complexity of working in potentially dangerous and/or unstable work environments. Topics covered include the problem with quasi-experimental designs, model fidelity adherence, program documentation, client recruitment and retention, and data collection. Recommendations are provided to improve evaluations of youth gang prevention programs and similar community-based interventions that focus on the prevention of youth violence.     

Quasi-experimental evaluation using confirmatory procedures: A case study of an economic and social empowerment intervention to reduce girls’ vulnerability to HIV in rural Mozambique

PurposeOur goal was to determine whether an economic and social empowerment intervention implemented in Zambézia Province, Mozambique reduced girls’ vulnerability to HIV. We use this experience to discuss challenges of evaluating real-world interventions.MethodsTwo rounds of data were collected from 885 girls, 13–19 years, for this clustered, non-equivalent (two-stage) cohort trial. We used multi-level exact matching and difference-in-differences estimation to estimate intervention effects on two outcomes: girls’ knowledge of gender-based violence and school attendance.ResultsEstimates of two outcomes analysed indicated no statistically significant intervention effects. Preliminary analysis of data from the intervention group revealed this study was unable to obtain accurate measures for five outcomes related to HIV vulnerability.ConclusionsAlthough our study did not find evidence of impact on the a priori selected outcomes, we report on our experience implementing this robust methodologic design and describe how the challenges encountered in this program setting affected our ability to attain results. We recommend prospective evaluation designs with random allocation be accommodated early during planning. When not possible, quasi-experimental studies should collect data from large samples. To reduce measurement bias, biological endpoints such sexually transmitted infections should serve as primary outcomes for programs intending to reduce sexual behaviors.     

Evidence from the field: Findings on issues related to planning,implementing and evaluating gender-based programs

An Initiative of the United States Department of Health and Human Services’ Office on Women's Health (OWH), Coalition for a Healthier Community (CHC), supports ten grantees across the U.S. in the implementation of gender-based health interventions targeting women and girls. A national evaluation is assessing whether gender-focused public health systems approaches are sustainable and cost effective in addressing health disparities in women and girls. To inform the evaluation, a systematic examination was conducted of literature in both the public and private sector designed to track, assess, understand, and improve women's health, public health systems approaches, and the cost-effectiveness and sustainability of gender-based programs. A two-person team assured the quality of the results following the review of abstracts and full-text articles. Of 123 articles meeting eligibility criteria (See inclusion criteria described in Section 2.2 below), only 18 met inclusion criteria specific to a focus on a systems approach, cost-effectiveness and/or sustainability. Studies assessing systems approaches suggested their effectiveness in changing perceptions and increasing knowledge within a community; increasing involvement of local decision-makers and other community leaders in women's health issues; and increasing community capacity to address women and girls’ health. Further evaluation of the cost-effectiveness and sustainability of gender-based approaches is needed.     

A user-focused model for the utilization of evaluation

It is proposed that the present low level of utilization of evaluation findings is traceable in part to their failure to address directly the information needs of a clearly specified decision maker. An alternative model proceeding from such specification is proposed here, with evaluation closely interwoven with the on-going innovation process. The model suggests a number of implications for the organizational role of evalutors, for the design of evaluations, and for directions for methodological development. In particular, we suggest that evaluation designs be assessed against possible “threats to utility” as well as against the traditional “threads to validity.”     

Pre-, post-, and longitudinal evaluation of juvenile justice education

This article describes two stages of the Juvenile Justice Educational Enhancement Program's pre-, post-, and longitudinal evaluation research. Pilot studies were used to explore how to design statewide research of pre- and postassessment scores and community reintegration outcomes. Preliminary findings suggest that higher performing educational programs produce greater educational gains as measured by academic achievement tests, credits earned, and pupil progression rates. The findings also indicate that these programs have more students returning to school and lower recidivism rates. Building on the pilot studies, refinements were made to the research designs to enable more comprehensive statewide evaluation. Current research includes collection of pre- and postassessment scores from official sources on approximately 16,000 juvenile justice youths. In addition, a research design has been developed to examine program effectiveness by measuring community reintegration variables. Multiple data sources, including official and self-reported data on family, school, employment, and subsequent crime involvement, will be used in the longitudinal study.     

The program effect of batterer programs in three cities

Recent meta-analyses and experimental designs of batterer program evaluations suggest little or no program effect. This finding may be compromised by a variety of analytical issues. Instrumental variable analysis addresses some of these issues, especially the relationship of program dropout to batterer reassault. This method of analysis was, therefore, used to test for program effect in a multi-site evaluation. The sites were three well-established batterer programs using a gender-based, cognitive-behavioral approach (n = 640). Completing a batterer program reduced the likelihood of reassault by 44% to 64%, depending on the specification used. Completing a 3-month program appeared to be as effective as completing a 5 1/2- or 9-month program. This moderate effect is an accomplishment considering the problems associated with the program participants and the inconsistency in the criminal justice system at the research sites. The findings remain tentative because of weak instrumental variables for reassault, but do confirm the need for more complex analyses of program effect.     

Teenage parent programs. A synthesis of the long-term effects of the New Chance Demonstration, Ohio's Learning, Earning, and Parenting Program, and the Teenage Parent Demonstration.

This article synthesizes the long-term findings from three major evaluations of programs that began in the late 1980s and were designed to improve the self-sufficiency of teenage mothers on welfare. Although each of the programs emphasized a different approach, an important story emerges. Economic outcomes for the mothers improved over time, and the interventions had some positive effects, particularly for the women who began these programs while they were enrolled is school. However, the interventions did not affect fertility, and the data on outcomes for the mothers' children raise concern.     

Looking Inside the “Black Box”

Abstract

The continued devolution of social welfare systems and services in the U.S. resultsin high stakes program evaluations in the field of family support and early intervention. Programs are expected to utilize evidence-based interventions and to demonstrate effectiveness. A look at implementation helps to differentiate between theories that do not work and programs that are not effective. Methods for identifying program implementation are needed. In a 17-site program evaluation, the author and her colleagues developed a methodology for measuring implementation and demonstrated the effects of differential implementation in understanding program outcomes.     

Mixed methods,materialism and the micropolitics of the research-assemblage

We assess the potential for mixing social research methods, based upon a materialist and micropolitical analysis of the research-assemblage and of what individual research techniques and methods do in practice. Applying a DeleuzoGuattarian toolkit of assemblages, affects and capacities, we document what happens when research methods and techniques interact with the events they wish to study. Micropolitically, many of these techniques and methods have unintended effects of specifying and aggregating events, with the consequently that the knowledge produced by social inquiry is invested with these specifications and aggregations. We argue that rather than abandoning these social research tools, we may use the micropolitical analysis to assess precisely how each method affects knowledge production, and engineer the research designs we use accordingly. This forms the justification for mixing methods that are highly aggregative or specifying with those that are less so, effectively rehabilitating methods that have often been rejected by social researchers, including surveys and experiments.     

NEIGHBORHOOD DISORDER AND INDIVIDUAL COMMUNITY CAPACITY: HOW INCIVILITIES INFORM THREE DOMAINS OF PSYCHOSOCIAL ASSESSMENT

This article examines contextual models to bring together the disorder and community capacity perspectives, since both are grounded in social (dis)organization theory and cumulative causation. We analyze how individual and neighborhood characteristics, social and physical disorder, and crime affect three individual community capacity outcomes: city quality of life, neighborhood safety, and household moving intentions. The “broken windows” downward spiral suggests that neighborhood incivilities may decrease multiple psychosocial assessments, or, individual community capacities. Consistent with prior research, we find that social and physical disorder decreases all three outcomes. Second, we find that both disorders also mediate neighborhood effects, including socioeconomic status and residential stability. Third, these direct and indirect disorder effects are not altered by prior victimization or neighborhood crime rates. Reducing disorder will, in turn, improve three distinct domains and geographic scales of individual community capacity, and can also reduce the adverse effects of other local area capacity deficits.     

Validity in an evaluation of Healthy Families Florida—A program to prevent child abuse and neglect

Evidence-based continuums have become a guide for identifying the level of evidence in evaluations of home visiting programs conducted to determine their effectiveness in preventing child abuse and neglect. While randomized controlled trials are required for the highest levels of evidence, quasi-experimental designs have also been specified as an appropriate alternative. Using a quasi-experimental evaluation of a home visiting program to prevent child abuse and neglect that adheres to the Healthy Families America model, this paper describes and illustrates how types of validity can be improved. More specifically, we address how threats to internal validity can be identified and reduced through statistical techniques; how construct validity may be strengthened using state records to measure the outcomes; and how external validity is affected by including or excluding study participants. After applying a variety of statistical adjustments to reduce selection bias, we found that the outcomes favored the home visiting program and increased after accounting for covariates that contributed to child abuse and neglect. This was true across the statistical techniques (traditional covariate and propensity score adjustment) used. For evaluations using quasi-experimental designs, recommendations relevant to the illustrations in the paper are also presented.     

Coalition for a Healthier Community: Lessons learned and implications for future work

The Coalition for a Healthier Community (CHC) initiative was implemented to improve the health and well-being of women and girls. Underpinning CHC is a gender-based focus that uses a network of community partners working collaboratively to generate relevant behavior change and improved health outcomes. Ten programs are trying to determine whether gender-focused system approaches are cost-effective ways to address health disparities in women and girls. Programs implemented through coalitions made up of academic institutions, public health departments, community-based organizations, and local, regional, and national organizations, are addressing health issues such as domestic violence, cardiovascular disease prevention, physical activity, and healthy eating. Although these programs are ongoing, they have made significant progress. Key factors contributing to their early success include a comprehensive needs assessment, robust coalitions, the diversity of populations targeted, programs based on findings of the needs assessments, evaluations taking into consideration the effect of gender, and strong academic–community partnerships. A noteworthy impact of these programs has been their ability to shape and impact public, social, and health policies at the state and local levels. However, there have been challenges associated with the implementation of such a complex program. Lessons learned are discussed in this paper.     

Evaluating theory-based evaluation: information, norms, and adherence

Programmatic social interventions attempt to produce appropriate social-norm-guided behavior in an open environment. A marriage of applicable psychological theory, appropriate program evaluation theory, and outcome of evaluations of specific social interventions assures the acquisition of cumulative theory and the production of successful social interventions - the marriage permits us to advance knowledge by making use of both success and failures. We briefly review well-established principles within the field of program evaluation, well-established processes involved in changing social norms and social-norm adherence, the outcome of several program evaluations focusing on smoking prevention, pro-environmental behavior, and rape prevention and, using the principle of learning from our failures, examine why these programs often do not perform as expected. Finally, we discuss the promise of learning from our collective experiences to develop a cumulative science of program evaluation and to improve the performance of extant and future interventions.     

Using intervention-oriented evaluation to diagnose and correct students’ persistent climate change misconceptions: A Singapore case study

The evaluation of classroom-based educational interventions is fraught with tensions, the most critical of which is choosing between focusing the inquiry on measuring the effects of treatment or in proximately utilizing the data to improve practice. This paper attempted to achieve both goals through the use of intervention-oriented evaluation of a professional development program intended to diagnose and correct students’ misconceptions of climate change. Data was gathered, monitored and analyzed in three stages of a time-series design: the baseline, treatment and follow-up stages. The evaluation itself was the ‘intervention’ such that the data was allowed to ‘contaminate’ the treatment. This was achieved through giving the teacher unimpeded access to the collected information and to introduce midcourse corrections as she saw fit to her instruction. Results showed a significant development in students’ conceptual understanding only after the teacher's decision to use direct and explicit refutation of misconceptions. Due to the accessibility of feedback, it was possible to locate specifically at which point in the process that the intervention was most effective. The efficacy of the intervention was then measured through comparing the scores across the three research stages. The inclusion of a comparison group to the design is recommended for future studies.     

The complexity of roles in community action projects: The example of the evaluation of ‘alternatives’

Evaluation of community interventions is a special form of evaluation involving many roles that are key to accomplishing both the intervention and the accompanying research. The present paper describes the roles involved in the evaluation of a community intervention (‘Alternatives’) aimed at safer use of alcohol and depressant medications by seniors.The three core roles were the community developer who provided leadership in the implementation of the project, the researcher/evaluator, and the community committee that sanctioned and facilitated the project. Other roles on the project included: the project coordinator, the nurse-interviewers who collected the survey data for the evaluation, volunteers from seniors groups, local health service providers, the media, the funders, and others on the project team who provided specific technical expertise.The paper describes the competing interests among these roles as well as conflicts that arose and how these conflicts were handled. Although specific roles are described, the types of roles and the potential for competing interests and conflict are generally relevant to community level evaluations.     

A systematic review of interventions to improve post-secondary educational outcomes among foster care alumni

In their transition to adulthood, youth who have aged out of foster care struggle in a variety of areas. One area of concern is achieving success in post-secondary education. In response, universities and community colleges around the country offer support programs for these youth, in order to assist them in gaining a college education. The purpose of this study was to utilize a systematic review process to uncover the current status of what is known about campus support programs for foster care alumni in two areas—student views of their program experiences and outcomes of interventions to improve post-secondary educational success. We adapted the Participants, Interventions, Comparison groups, Outcomes, and Study designs (PICOS) model in conducting the systematic review. A search of six electronic databases and other gray literature sources yielded seven articles. Results revealed valuable insight from students regarding the ways in which program implementation was beneficial, and highlighted important strategies future programs should consider. Further, findings of outcome studies are favorable. However, additional outcome evaluations are needed in order to validate and confirm their impact.     

Methodological issues in the measurement of birth preparedness in support of safe motherhood

Behavior change interventions focusing on birth preparedness for pregnant women, their husbands, and adults in the community are common components of community-oriented Safe Motherhood programs in developing countries. Few studies have examined the effectiveness of these interventions, and existing studies are flawed due to study and sample design. This article highlights methodological issues that are often overlooked when measuring indicators of birth preparedness among multiple audiences for program evaluation purposes in household-based surveys. Solutions are proposed to address each of these problems in an effort to improve future research.     

Identifying and measuring unintended outcomes

Unintended outcomes of governmental actions have received little attention from evaluators. The argument made here is that outcome evaluations should routinely include efforts to identify and measure unintended outcomes. A systems perspective is presented which treats governmental actions as disequilibrating intrusions into reacting systems that produce an array of outcomes, only some of which are intended. Intended and unintended outcomes can be valued monetarily, in terms of human rights, or both; specifically, outcomes having monetary value alter both the level and distribution of income while outcomes with rights value alter the level and distribution of rights. More needs to be done on unintended outcomes. The systems approach offered here requires further elaboration. Design modification that increase the chances of identifying and measuring unanticipated outcomes are needed, especially for designs based on non-equivalent groups and time-series data. Instrumentation is needed for designs based on non-equivalent groups and time-series data. Instrumentation is needed that will more accurately and sensitively register unanticipated outcomes. In the meantime, evaluators should pay more attention to theoretical predictions, the experiences of those involved with programs similar to the ones being evaluated, intercultural differences, and to unintended outcomes that originate within government itself.