Comparing rating paradigms for evidence-based program registers in behavioral health: Evidentiary criteria and implications for assessing programs

Decision makers need timely and credible information about the effectiveness of behavioral health interventions. Online evidence-based program registers (EBPRs) have been developed to address this need. However, the methods by which these registers determine programs and practices as being “evidence-based” has not been investigated in detail. This paper examines the evidentiary criteria EBPRs use to rate programs and the implications for how different registers rate the same programs. Although the registers tend to employ a standard Campbellian hierarchy of evidence to assess evaluation results, there is also considerable disagreement among the registers about what constitutes an adequate research design and sufficient data for designating a program as evidence-based. Additionally, differences exist in how registers report findings of “no effect,” which may deprive users of important information. Of all programs on the 15 registers that rate individual programs, 79% appear on only one register. Among a random sample of 100 programs rated by more than one register, 42% were inconsistently rated by the multiple registers to some degree.     

Similarities and differences in program registers: A case study

Researchers, clinicians, and other professionals are increasingly in need of cost-effective, evidence-based programs and practices. However, these individuals may lack the time and, for some, the required expertise to search for and identify such interventions. To address this concern, several online registers that list or categorize programs according to their empirical evidence of effectiveness have been established. Although these registers are designed to simplify the task of selecting evidence-based interventions, the use of distinct review processes and standards by each register creates discrepancies in final program classifications, which can pose a challenge for users. The present case study highlights three programs that have been evaluated by more than one register and have received similar or different classifications. Reasons for inconsistencies are discussed, and several recommendations for evaluating organizations and register users are provided to enhance the functionality and ease of use of online program registers.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals.Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.     

A qualitative exploratory study of substance abuse prevention outcomes in a heterogeneous prevention system

The prevention of abuse of alcohol and other drugs is a concern for parents, policy-makers, educators, and social service professionals. Prevention programs are sponsored by many different types of social and educational agencies using a variety of intervention strategies. This article reports a study of a sample of such programs in the state of Nevada. The overall prevention system in the state espouses a "risk and resiliency" approach to prevention. Focus group methodology was used to study perception of outcomes of these programs from the viewpoints of various program stakeholders (youth participating in the programs, parents of participants and program staff). Analysis of the qualitative data yielded findings about potential outcomes as well as implicit program theories. Implications for future planning efforts as well as further evaluation efforts are discussed.     

A large-scale initiative to disseminate an evidence-based drug abuse prevention program in Italy: Lessons learned for practitioners and researchers

Across developed countries, experimentation with alcohol, tobacco, and other drugs often begins in the early adolescent years. Several evidence-based programs have been developed to prevent adolescent substance use. Many of the most rigorously tested and empirically supported prevention programs were initially developed and tested in the United States. Increasingly, these interventions are being adopted for use in Europe and throughout the world. This paper reports on a large-scale comprehensive initiative designed to select, adapt, implement, and sustain an evidence-based drug abuse prevention program in Italy. As part of a large-scale regionally funded collaboration in the Lombardy region of Italy, we report on processes through which a team of stakeholders selected, translated and culturally adapted, planned, implemented and evaluated the Life Skills Training (LST) school-based drug abuse prevention program, an evidence-based intervention developed in the United States. We discuss several challenges and lessons learned and implications for prevention practitioners and researchers attempting to undertake similar international dissemination projects. We review several published conceptual models designed to promote the replication and widespread dissemination of effective programs, and discuss their strengths and limitations in the context of planning and implementing a complex, large-scale real-world dissemination effort.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

Using propensity scoring to estimate program-related subgroup impacts in experimental program evaluations

This article discusses the use of propensity scoring in experimental program evaluations to estimate impacts for subgroups defined by program features and participants' program experiences. The authors discuss estimation issues and provide specification tests. They also discuss the use of an overlooked data collection design--obtaining predictions that program intake staff make about applicants' likely program assignments and experiences--that could improve the quality of matched comparison samples. They demonstrate the effectiveness of this approach in producing credible subgroup findings using data from a large-scale experimental evaluation of Job Corps, the nation's largest federal education and training program for disadvantaged youths.     

Applied behavioral intervention in rural school settings: Problems in research design,implementation, and evaluation

This article discusses a number of theoretical, practical, and methodological difficulties encountered in the design, implementation, and evaluation of treatment programs conducted in the natural environment. In order to illustrate how these difficulties might be overcome, a case example is provided. Contingency contracts were used to modify the assaultive and disruptive behaviors of a 6-year-old child attending a preschool program for emotionally and physically handicapped children. Treatment effectiveness was evaluated through the use of a multiple baseline design and maintenance of behavior change was assessed in periodic follow ups of the child for 1 year after his admittance to public school. The importance of, planning for the generalization of newly acquired behaviors to different environments is stressed and suggestions for facilitating this process are discussed.     

Empirically based conversion factors for calculating couple-years of protection

Couple-years of protection (CYP) is one of several commonly used indicators to assess international family planning efforts. It has been the subject of much debate, relating in part to the specific conversion factors used to translate the quantity of the respective contraceptive methods distributed to a single measure of protection. This article outlines a comprehensive effort to revisit those conversion factors based on the best available empirical evidence. In most instances, the analysis supports previously established standard conversion factors. However, there are two notable departures. Fewer condoms and spermicides are recommended for each CYP (120 vs. 150), primarily because coital frequency among condom users is lower than previously assumed. Furthermore, for sterilization, the authors recommend the use of country or region-specific conversion factors. Every program evaluation indicator has strengths and weaknesses, and the best program evaluation efforts use a variety of indicators. If CYP is used to evaluate programs, however, the authors believe that the conversion factors presented reflect the best available evidence.     

The employment service and vacancy durations

In the literature, there are few examples of studies analyzing the impact of labor market programs on vacancies. This article presents the results of a study of the impact of personnel increase at Swedish employment offices on vacancy durations. The evaluation method in the study is quasi-experimental. The main results of the study are as follows: (a) The increase of employment office staff members reduced their vacancy durations by approximately 2 days, and the probability that a vacancy is cancelled from the register is 1.17 times higher for the program offices; and (b) this effect is too small for the increase of staff members to represent a socially efficient use of resources and for a positive outcome in a public finance context.     

Alternative solutions to the problem of selection bias in an analysis of federal residential drug treatment programs

In an evaluation of prison-based residential drug treatment programs, the authors use three different regression-based approaches to estimating treatment effects. Two of the approaches, the instrumental variable and the Heckman approach, attempt to minimize selection bias as an explanation for treatment outcomes. Estimates from these approaches are compared with estimates from a regression in which treatment is represented by a dummy variable. The article discusses the advantage of using more than one method to increase confidence in findings when possible selection bias is a concern. Three-year outcome data for 2,315 federal inmates are used in analyses where the authors separately examine criminal recidivism and relapse to drug use for men and women. Statistical tests lead the authors to conclude that treatment reduces criminal recidivism and relapse to drug use. The treatment effect was largest when the inference was based on the Heckman approach, somewhat smaller when based on the instrumental variable approach, and smallest when based on the traditional dummy variable approach. Treatment effects for females were not statistically significant.     

Towards a faith-based program theory: a reconceptualization of program theory

A meta-program theory is proposed to overcome the limitations and improve the use of program theory as an approach to faith-based program evaluation. The essentials for understanding religious organizations, their various programs, and faith and spirituality are discussed to support a rationale for developing a faith-based program theory that captures the complex nature of religious program phenomena. Implications for public policy are examined. New concepts based on structural approach and four program theories are formulated for program analysis. Several theoretical concepts for faith-based program evaluation to guide methods for describing patterns that reflect the religious beliefs and values are suggested.     

Qualitative cost–benefit evaluation of complex,emergent programs

This paper discusses a methodology used for a qualitative cost–benefit evaluation of a complex, emergent program. Complex, emergent programs, where implementation varies considerably over time and across sites to respond to local needs and opportunities, present challenges to conventional methods for cost–benefit evaluation. Such programs are characterized by: ill-defined boundaries of what constitutes the intervention, and hence the resources used; non-standardized procedures; differing short-term outcomes across projects, even within the same long-term goals; and outcomes that are the result of multiple factors and co-production, making counter-factual approaches to attribution inadequate and the use of standardized outcome measures problematic. The paper discusses the advantages and limitations of this method and its implications for cost–benefit evaluation of complex programs.     

ATOD-TV: A Case Study on Developing Interactive Multimedia

ABSTRACT

ATOD-TV (Alcohol, Tobacco, and Other Drugs-TV), an interactive multimedia program funded by the National Institute on Drug Abuse, was designed to teach the general public about the importance of substance abuse research and treatment. This article examines the process used to develop the program and argues that its success is due more to careful planning and formative evaluation than the use of sophisticated tools and techniques.     

Using evaluation to improve program quality based on the BELL model

Building Educated Leaders for Life (BELL) is a national not-for-profit organization whose mission is to increase the educational achievements, self-esteem, and life opportunities of elementary school children living in low-income urban communities. BELL has been engaged in formal evaluation, internally and externally, for more than five years and has built internal evaluation capacity by investing in a specialized full-time evaluation team. As part of a continuous program improvement model of evaluation, BELL uses the data to refine program implementation and replicate successful elements of the services and operations. In this chapter, the authors highlight best practices from the field by outlining BELL's approach to using evaluation data for continuous program improvement. Key strategies include (1) carefully identifying intended users of the evaluation throughout the organization and among its external stakeholders, then working closely with intended users throughout the evaluation process, ensuring full engagement at every step of the process; (2) reporting findings in a readable, user-friendly format and timing the reporting so that it is aligned with programmatic decision making and planning cycles; and (3) making and supporting explicit recommendations for the next program cycle, where intended users have agreed to recommendations and ownership is assigned. BELL's successful use of data for improvement is evidenced by the consistently strong outcomes for the students it serves as well as increased efficiency and satisfaction related to service delivery that has supported the replication of BELL's programs nationally.     

The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.