The opportunities and challenges of multi-site evaluations: Lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals.Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.     

‘Parenting teenagers’ programmes: a UK perspective

The parenting of teenagers is an area of political and public debate at present, and many different forms of support are now available for this group of parents. There has been a particular increase in recent years in the number of group-based parenting programmes provided for parents of teenagers. This paper describes research which (i) surveyed the range and nature of ‘parenting teenagers’ courses currently available in the UK, and (ii) identified key characteristics of these courses, including aims, value bases, materials, methods, parents attending, outcomes, and evaluation. A number of key issues are identified, and recommendations for future practice, policy and research in this important area are made.     

Methodological challenges in the implementation and evaluation of social welfare policies

As social reality is quite elusive, even regarding seemingly well-recognized everyday concepts and objects, there are always methodological challenges underlying assessments and evaluations of implementation policies. The present article addresses this area of concern by presenting the results of a rereading of an empirical study of elderly home care services. Our results reveal the emergence of a dissolution of common and professional key concepts and objects in these welfare services to a degree that challenges both the implementation policy and the evaluation of policy. We claim that this has methodological implications for evaluation of implementation policies in general.     

Key considerations for logic model development in research partnerships: a Canadian case study

Community-academic partnership research is a fairly new genre of community-based participatory research. It has arisen in part, from recognition of the potential role of alliances in the development and translation of applied knowledge and the elimination of health disparities. This paper reports on the learning process of academic and community members who worked together in developing a logic model for a research program focusing on partnerships with vulnerable populations. The Partners in Community Health Research is a 6-year training program that seeks to combine research, training, and practice through the work of its "learning clusters". As these types of partnerships proliferate, the articulation and exploration of clear models will assist in their implementation. The authors, coming from both academia and community agencies, present a logic model meant to facilitate program management. Key considerations in the model's development are discussed in the context of an ongoing research partnership; namely, the complexity of the research partnership, power and accountability, alignment with health promotion policy, and the iterative nature of program design. Recommendations challenge academics, policy-makers, service providers, and community members to reflect on the elements needed to support and manage research partnerships and the tools necessary to ensure continued collaboration.     

Economic evaluation of continuing care interventions in the treatment of substance abuse: recommendations for future research

The chronic and relapsing nature of substance abuse points to the need for continuing care after a primary phase of treatment. This article reviews the economic studies of continuing care, discusses research gaps, highlights some of the challenges of conducting rigorous economic evaluations of continuing care, and offers research guidelines and recommendations for future economic studies in this emerging field. Rigorous economic evaluations are needed by health care providers and policy makers to justify the allocation of scarce resources to continuing care interventions. The adoption of cost-effective continuing care services can reduce long-term consequences of addiction, thereby potentially increasing overall social welfare.     

Applying national survey results for strategic planning and program improvement: The National Diabetes Education Program

Since the 1970s, the federal government has spearheaded major national education programs to reduce the burden of chronic diseases in the United States. These prevention and disease management programs communicate critical information to the public, those affected by the disease, and health care providers. The National Diabetes Education Program (NDEP), the leading federal program on diabetes sponsored by the National Institutes of Health (NIH) and the Centers for Disease Control and Prevention (CDC), uses primary and secondary quantitative data and qualitative audience research to guide program planning and evaluation. Since 2006, the NDEP has filled the gaps in existing quantitative data sources by conducting its own population-based survey, the NDEP National Diabetes Survey (NNDS). The NNDS is conducted every 2–3 years and tracks changes in knowledge, attitudes and practice indicators in key target audiences. This article describes how the NDEP has used the NNDS as a key component of its evaluation framework and how it applies the survey results for strategic planning and program improvement. The NDEP's use of the NNDS illustrates how a program evaluation framework that includes periodic population-based surveys can serve as an evaluation model for similar national health education programs.     

Evaluation in the community mental health centers program: A bold new reproach?

The Federal Community Mental Health Centers Program (CMHC)-from 1963 to 1981-was heralded as a revolution in mental health care. Championed by many, and severely criticized by others, the actual impact of the program on the nation's mental health remains unclear. The authorization to evaluate the CMHC Program came originally from congressional legislation (PL 90-174), and later from the policies and regulations of NIMH under a series of Federal laws, notably PL 94-63. From 1976-1980, two dominant evaluation strategies were prevalent: funds expended by NIMH each year for studies of CMHC services or program-wide evaluations, and a much larger expenditure by CMHCs to conduct their own, independent evaluations following federal guidelines. As the Center's Program was turned over to the states in the form of block grants (PL 97-35), a group of professionals involved with setting and carrying out federal CMHC evaluation policy of both varieties met in public forum to debate the impact of these two evaluation approaches. While some participants cited gains in evaluation technology and impact upon local management of CMHCs, others found the lack of a coordinated and systematic approach to evaluating the CMHC Program to have been an opportunity missed. The impact of CMHC evaluation efforts are also discussed in terms of their major contribution to the field of evaluation research as a whole.     

Obstacles to gaining ethical approval for a multi-centre study of family support

The NHS has emphasised the need for rigorous evaluations of preventative home-visiting support (Bull and others, 2004) in building a public health evidence base. Interdisciplinary studies involving the NHS in any capacity are subject to ethical review by NHS Research Ethics Committees. However, the current process of ethical review does not respond flexibly to research conducted by non-NHS organisations. Through documentary analysis, this paper describes the experience of a non-clinical research evaluation of a family support service offered during pregnancy and the difficulties encountered in gaining ethical approval. Implications for social science research with minimal NHS involvement are discussed and recommendations for improvements suggested.     

Monitoring and evaluation design of Malawi’s Right Foods at the Right Time nutrition program

Child stunting is a public health problem in Malawi. In 2014, the Government of Malawi launched the Right Foods at the Right Time (RFRT) program in Ntchisi district delivering nutrition social and behavior change communication, a small-quantity lipid-based nutrient supplement to children 6–23 months, and nutrition sensitive activities. Monitoring and evaluation (M&E) systems are key aspects of successful program implementation. We describe these and the methodology for an impact evaluation that was conducted for this program. Two monitoring systems using traditional and electronic platforms were established to register and track program delivery and processes including number of eligible beneficiaries, worker performance, program participation, and to monitor input, output, and outcome indicators. The impact evaluation used comparative cross-sectional and longitudinal designs to assess impact on anthropometric and infant and young child feeding outcomes. Three cross-sectional surveys (base-, mid-, and end-line) and two longitudinal cohorts of children followed in 6-month intervals from 6 to 24 months of age, were conducted in sampled households in the program and a neighboring comparison district. Additional M&E included qualitative studies, a process evaluation, and a cost-effectiveness study. The current paper describes lessons from this program’s M&E, and demonstrates how multiple implementation research activities can inform course-correction and program scale-up.     

The complexity of roles in community action projects: The example of the evaluation of ‘alternatives’

Evaluation of community interventions is a special form of evaluation involving many roles that are key to accomplishing both the intervention and the accompanying research. The present paper describes the roles involved in the evaluation of a community intervention (‘Alternatives’) aimed at safer use of alcohol and depressant medications by seniors.The three core roles were the community developer who provided leadership in the implementation of the project, the researcher/evaluator, and the community committee that sanctioned and facilitated the project. Other roles on the project included: the project coordinator, the nurse-interviewers who collected the survey data for the evaluation, volunteers from seniors groups, local health service providers, the media, the funders, and others on the project team who provided specific technical expertise.The paper describes the competing interests among these roles as well as conflicts that arose and how these conflicts were handled. Although specific roles are described, the types of roles and the potential for competing interests and conflict are generally relevant to community level evaluations.     

How Academic Research Shapes Labor and Social Policy

How academic research affects labor and social policy is viewed through a program evaluation framework that highlights the difficulties of determining the causal impact of such research on public policy. The effect is illustrated by examples. My conclusion is that academic research can have a modest to substantial impact on policy. Its impact is enhanced if it has a number of key characteristics: high quality; reputable researchers involved; synthesized and translated into a language understood by policy makers, the general public, and the media; credible champions who will broker and defend it, in the political process or in the public realm; timeliness; and, political acceptability.     

Tailored program evaluation: Past, present, future

This paper discusses measurement issues related to the evaluation of computer-tailored health behavior change programs. As the first generation of commercially available tailored products is utilized in health promotion programming, programmers and researchers are becoming aware of the unique challenges that the evaluation of these programs presents. A project is presented that used an online tailored health behavior assessment (HBA) in a worksite setting. Process and outcome evaluation methods are described and include the challenges faced, and strategies proposed and implemented, for meeting them. Implications for future research in tailored program development, implementation, and evaluation are also discussed.     

Evaluation of a research mentorship program in community care

This article describes the results of a qualitative case study evaluating a research mentorship program in community care settings in Ontario, Canada. The purpose of the program was to build evaluation and research capacity among staff of community care agencies through a mentorship program. Data were collected through in-depth, semi-structured interviews, focus group interviews, and written evaluations. Three themes were identified: taking hold, fostering hold, and holding on. Mentees took hold of new evaluation and research skills. Factors fostering hold included positive mentoring relationships and participation in relevant research projects. Mentees struggled to hold on to, and apply their skills within a community care context of challenges such as constrained resources and a mandate to provide client-centered care. The lessons learned include having trainees participate in relevant, time-limited evaluation and research projects, and early implementation of mechanisms to sustain integration of trainees' knowledge and skills within community care agencies.     

Swimming upstream: The challenges and rewards of evaluating efforts to address inequities and reduce health disparities

Racial and ethnic disparities in the health of Americans are widespread and persistent in the United States despite improvements in the health of Americans overall. Increasingly, strategies for reducing disparities have focused on addressing the factors that contribute to – if not fundamentally underlie – health disparities: social, economic, and environmental inequities, which limit access to resources and cause unhealthy exposures. As public health shifts to interventions that seek to improve the circumstances of disproportionately affected populations and achieve equity through policy change, alternative methods to evaluate these efforts are also required. This paper presents an example of such approaches to addressing asthma disparities through Regional Asthma Management and Prevention's (RAMP) programmatic efforts and an evaluation of these activities. The paper describes RAMP's targets and strategies, as well as the specific evaluation methods applied to each, including activity tracking, observations, surveys, key informant interviews, and case studies. Preliminary evaluation findings are presented, as are lessons learned about the efficacy of the evaluation design features – both its strengths and shortcomings. Findings discussed are intended to contribute to the growing literature that provides evidence for the application of emerging approaches to evaluation that reflect non-traditional public health and support others interested in expanding or replicating this work.     

Building evaluation capacity: definitional and practical implications from an Australian case study

Internationally, evaluation capacity-building activities have mushroomed as demands have increased for government-funded programs to demonstrate that they are effective and efficient. Despite this, there is a lack of clarity in the way in which evaluation capacity-building is defined and conceptualized. The current paper presents a case study of a national evaluation capacity building exercise that we are in the midst of conducting in Australia, and discusses the findings in relation to definitional, conceptual and practical issues. Specifically, we describe an evaluation capacity building exercise involving over 100 mental health projects, detailing the methods that we employ, some of the challenges that we have faced, and the benefits we feel we are achieving. Our key message is that definitions of evaluation capacity-building should not only make reference to equipping organizations to routinely conduct evaluations, but should also stress the varied uses to which evaluation findings can be put. In addition, such definitions should acknowledge some of the valuable by-products of evaluation capacity building activities, such as the development of shared understandings of the program or project being evaluated.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.