Collaborative research in child welfare: a rationale for rigorous participatory evaluation designs to promote sustained systems change

Expansion of the child welfare evidence base is a major challenge. The field must establish how organizational systems and practice techniques yield outcomes for children and families. Needed research must be grounded in practice and must engage practitioners and administrators via participatory evaluation. The extent to which successful practices are transferable is also challenged by the diversity of child welfare systems. The U.S. Department of Health and Human Services' Children's Bureau funded Quality Improvement Centers (QICs) that were designed to promote collaborative, multisite research that would address these evaluation needs. This article, based on the findings of a regional and a national QIC, describes the challenges facing research collaboration and the strategies for achieving success.     

Constructing quantitative implementation scales from categorical services data. Examples from a multisite evaluation

Like measures of outcome, measures of implementation are most useful and analytically powerful when measured at client-level and are quantitative. However, high-quality, individual-level, quantitative service utilization data can be expensive, intrusive, or otherwise impractical to obtain. Cruder data--for example, presence versus absence of a given service--are often more feasible to collect, as well as more likely to be available. Consequently, evaluators can benefit by finding ways to better exploit such data at the analysis phase to compensate for shortcomings at the collection phase. This article documents one such instance in which this was done. Specifically, it describes how quantitative, client-level implementation scales were derived from qualitative (categorical) data and used to support a cross-site synthesis of implementation and outcome analyses in a multisite evaluation. It also suggests additional scenarios in which quantitative implementation scales might be derived form qualitative services data.     

A strategy for accurate collection of incremental cost data for cost-effectiveness analyses in field trials. Pawtucket's minimal contact cholesterol education intervention

With many community field trials or education interventions, the cost-effectiveness analyses are not given a high priority. However, this type of evaluation is important for purposes of future adoption of the intervention. The accurate measurement of costs can best be served by prospective collection of data. This article describes a methodology for collection of cost data that coincides with the intervention implementation. This cost analysis strategy has seven discrete steps. The Minimal Contact Education for Cholesterol Change study is used as an example of the use of this strategy. This intervention provides cholesterol education at six different levels of intensity at four different sectors. The intensity levels vary along a continuum from very little education input to a maximum level of intervention that might be practical in a screening setting. The cost-effectiveness analysis component of the study will identify the incremental cost-effectiveness of each intervention along the continuum.     

Analysis approaches to community evaluation

Analysis approaches to the evaluation of community interventions must be sensitive to a wide variety of analytic contaminants that may bias the statistical assessment of changes in outcome measures. These contaminants include model misspecifications related to failures to control for community-specific time trends, temporal autocorrelated errors in equations, spatial autocorrelated errors among geographic units, and other failures of unit independence otherwise indexed by estimated intraclass correlations. Although an enormous amount of progress has been made toward the solution of many of these analytic problems over the past years, the contemporary evaluator of community interventions is left with a number of unenviable design and analysis choices; choices that inevitably force an assessment of the relative threats of different sources of error to the internal and external validity of the evaluation. This article describes the choices made for the evaluation of the Community Trial Project outcome data.     

Data integration in the evaluation of juvenile justice education

This article discusses the Juvenile Justice Educational Enhancement Program's evaluation of juvenile justice education through the integration of multiple data sources. Validation of best practices and the corresponding effect of these practices on community reintegration require the development of a comprehensive database of program-level and individual-level indicators from several sources. These sources include the Florida Department of Education, the Florida Department of Juvenile Justice, the Florida Department of Law Enforcement, the Florida Department of Corrections, and the Juvenile Justice Educational Enhancement Program's own educational quality assurance and pre-/posttest data. This article examines the methodological, political, and bureaucratic obstacles encountered in this effort. Evaluation research analyses are also explored with an emphasis on the meaning of outcome measures relative to causal issues of time order, the modeling of events, and analytic strategy. The article provides a framework for future juvenile justice educational evaluation and a critical examination of problems encountered in integrating data sources for the purpose of outcome evaluation.     

Pre-, post-, and longitudinal evaluation of juvenile justice education

This article describes two stages of the Juvenile Justice Educational Enhancement Program's pre-, post-, and longitudinal evaluation research. Pilot studies were used to explore how to design statewide research of pre- and postassessment scores and community reintegration outcomes. Preliminary findings suggest that higher performing educational programs produce greater educational gains as measured by academic achievement tests, credits earned, and pupil progression rates. The findings also indicate that these programs have more students returning to school and lower recidivism rates. Building on the pilot studies, refinements were made to the research designs to enable more comprehensive statewide evaluation. Current research includes collection of pre- and postassessment scores from official sources on approximately 16,000 juvenile justice youths. In addition, a research design has been developed to examine program effectiveness by measuring community reintegration variables. Multiple data sources, including official and self-reported data on family, school, employment, and subsequent crime involvement, will be used in the longitudinal study.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

An evaluability assessment of a West Africa based Non-Governmental Organization's (NGO) progressive evaluation strategy

While program evaluations are increasingly valued by international organizations to inform practices and public policies, actual evaluation use (EU) in such contexts is inconsistent. Moreover, empirical literature on EU in the context of humanitarian Non-Governmental Organizations (NGOs) is very limited. The current article focuses on the evaluability assessment (EA) of a West-Africa based humanitarian NGO's progressive evaluation strategy. Since 2007, the NGO has established an evaluation strategy to inform its maternal and child health care user-fee exemption intervention. Using Wholey's (2004) framework, the current EA enabled us to clarify with the NGO's evaluation partners the intent of their evaluation strategy and to design its program logic model. The EA ascertained the plausibility of the evaluation strategy's objectives, the accessibility of relevant data, and the utility for intended users of evaluating both the evaluation strategy and the conditions that foster EU. Hence, key evaluability conditions for an EU study were assured. This article provides an example of EA procedures when such guidance is scant in the literature. It also offers an opportunity to analyze critically the use of EAs in the context of a humanitarian NGO's collaboration with evaluators and political actors.     

Building a worldwide evaluation community: past, present, and future

This article describes recent efforts to build a worldwide evaluation community. It notes events that led to an almost 200 percent growth in the number of regional and national evaluation organizations in slightly more than one year. Profiles are provided on the present status of many national and regional evaluation organizations that participated in the W.K. Kellogg Foundation-sponsored international evaluation meeting in Barbados, West Indies. Afterwards, the authors engage in a scenario planning exercise to picture alternatives of what the future of international evaluation might look like.     

An evaluation of retail outlets as part of a community prevention trial to reduce sales of harmful legal products to youth

Communities across the nation have become increasingly concerned about inhalant use and use of harmful legal products among youth because of increasing prevalence rates and deleterious health consequences from abusing these products. The increasing concern of communities about inhaling and ingesting legal products has been coupled with increasing awareness and concern about ability of youth to access and abuse a variety of other legal retail products. There are few examples of scientifically designed community prevention projects that seek to reduce youth abuse of such legal products. This article describes a community prevention trial that is designed to reduce sales of inhalants and other harmful legal products to youth and demonstrates how the retailer component of the trial can be rigorously evaluated. It also shows how data from youth purchase attempts can complement survey data from retailers.     

Development of a participatory monitoring and evaluation strategy

This paper describes the process of developing a participatory monitoring and evaluation strategy for a Kenyan youth-based NGO. The iterative nature of the study including the process of narrowing down indicators to measure and methods to monitor/evaluate these is well documented. A discussion on the extent to which the process achieved participation and was empowering for the participants reflects on existing power relationships and cultural context of Kenya and points to the need to create opportunities for youth where they engage with the broader community. Lessons that emerge out of the study focus on the importance of prioritizing monitoring and evaluation, the potential of youth to carry out effective monitoring and evaluation, and the need for researchers to engage respectfully with communities and participants.     

The development and field testing of a vocational education evaluation model

Reported is the development and field testing of an evaluation model used to evaluate the vocational education programs at the Metropolitan Community Colleges (MCC) of Kansas City, Missouri which can be implemented on vocational programs at other institutions. The model evaluated a vocational program on the following six criteria: (1) program's relationship to job market profile; (2) program's level of community support; (3) program's success in meeting vocational aspirations of clientele; (4) program's success in terms of student performance; (5) program 's cost-effectiveness; and (6) program 's success in reaching the handicapped and disadvantaged. The evaluation model provides a strategy and a set of procedures for assessing the quality of existing vocational education programs of study at any post-secondary institution. The model can be easily and inexpensively implemented, thus providing a data base which is useful for administrative planning and decision making.     

The complexity of roles in community action projects: The example of the evaluation of ‘alternatives’

Evaluation of community interventions is a special form of evaluation involving many roles that are key to accomplishing both the intervention and the accompanying research. The present paper describes the roles involved in the evaluation of a community intervention (‘Alternatives’) aimed at safer use of alcohol and depressant medications by seniors.The three core roles were the community developer who provided leadership in the implementation of the project, the researcher/evaluator, and the community committee that sanctioned and facilitated the project. Other roles on the project included: the project coordinator, the nurse-interviewers who collected the survey data for the evaluation, volunteers from seniors groups, local health service providers, the media, the funders, and others on the project team who provided specific technical expertise.The paper describes the competing interests among these roles as well as conflicts that arose and how these conflicts were handled. Although specific roles are described, the types of roles and the potential for competing interests and conflict are generally relevant to community level evaluations.     

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals.Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

Data-Driven Organizing: A Community–University Partnership to Address Vacant and Abandoned Property

Vacant and abandoned properties adversely affect the physical, mental, social, and economic health and well-being of individuals, families, and communities. This article describes a community–university partnership that combined a multimethod data collection strategy with a novel community-based participatory intervention research model (i.e., data-driven organizing) to address the vacancy problem. The project assessed the conditions of over 1,500 properties in an economically disadvantaged, predominantly African American neighborhood in Pittsburgh, Pennsylvania, and mobilized residents to use an existing policy mechanism to ameliorate the impact of property vacancy in the community.     

Hospital-to-community transitions. A bridge program for adolescent mental health patients

Making the transition from the hospital to a community setting can be extremely challenging for patients with acute mental health conditions. Transitional services have been created to help patients overcome difficulties associated with this transition. Nurses frequently play an integral role in the success of these services. By providing patients with individualized support during such transitions, nurses act as clinical liaisons and directly contribute to an increase in positive patient and system-level outcomes. This article describes a transitional service called the Bridge Program, designed to help adolescents make a successful transition from the hospital to the community. An overview of the Bridge Program is provided, and the results of an evaluation of this program are presented. Results suggest that the Bridge Program contributes to a decrease in the length of hospital stays and improves continuity of care for patients and their families.