Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

Examining implementation and preliminary performance indicators of veterans treatment courts: The Kentucky experience

Veterans’ Treatment Courts (VTCs) are posited as a solution to offer rehabilitation for veterans involved in the criminal justice system. Despite the pervasive implementation of VTCs, there is little research focused specifically on VTC implementation and outcomes, which are based on other problem-solving court models such as drug court. The current study presents qualitative process evaluation data from key stakeholders (n = 21) and veteran participants (n = 4) to show accomplishments, challenges, and lessons learned during first-year implementation at two VTC sites. Quantitative performance data is also presented on veteran participants (n = 19) served during the first year to show: types of services, monitoring, judicial interaction, sanctions/therapeutic responses, and rewards, as well as preliminary data on recidivism. Qualitative data, from both key stakeholders and veteran participants, suggests that offering rehabilitation via various program components, services/referrals, and accountability are critical to the success of the VTC. Data also provides valuable lessons learned for VTC implementation including communication, collaboration, information/protocols, and resources. Performance data shows that a variety of services are utilized and that frequent judicial interaction, drug testing, and sanctions are cornerstones of the VTC. Implications and future directions for research are discussed.     

Monitoring and evaluating transition and sustainability of donor-funded programs: Reflections on the Avahan experience

PurposeIn low and middle-income countries, programs funded and implemented by international donors frequently transition to local funding and management, yet such processes are rarely evaluated. We reflect upon experience evaluating the transition of a large scale HIV/AIDS prevention program in India, known as Avahan, in order to draw lessons about transition evaluation approaches and implementation challenges.ResultsIn terms of conceptualizing the transition theory, the evaluation team identified tensions between the idea of institutionalizing key features of the Avahan program, and ensuring program flexibility to promote sustainability. The transition was planned in three rounds allowing for adaptations to transition intervention and program design during the transition period. The assessment team found it important to track these changes in order to understand which strategies and contextual features supported transition. A mixed methods evaluation was employed, combining semi-structured surveys of transitioning entities (conducted pre and post transition), with longitudinal case studies. Qualitative data helped explain quantitative findings. Measures of transition readiness appeared robust, but we were uncertain of the robustness of institutionalization measures. Finally, challenges to the implementation of such an evaluation are discussed.ConclusionsGiven the scarceness of transition evaluations, the lessons from this evaluation may have widespread relevance.     

SASA! Together: An evolution of the SASA! approach to prevent violence against women

One in three women globally will experience intimate partner violence (IPV) with devastating consequences for individual survivors, their families and communities. While prevalence remains high, violence against women is not inevitable and community mobilization approaches have emerged as particularly promising for transforming the gender inequitable norms and practices that underlie violence. The SASA! Activist Kit to Prevent Violence Against Women (SASA!), developed by Raising Voices in 2008, provides a theory-based approach for mobilizing communities to transform power imbalances between women and men through critical discussion and positive action. In this article, we provide the rational for revising SASA! after ten years of program learning and formal research. We aim to contribute to the knowledge base around what works to prevent IPV by describing the core enhancements in the revised version--called SASA! Together—and linking these changes to Raising Voices’ program learning and broader advancements in the field. In addition, we reflect on how current debates—such as how best to “scale up” violence prevention programs—were considered and resolved in SASA! Together. The paper concludes by sharing lessons learned that may provide guidance for future revisions development and revisions of evidenced-based programs.     

Multisectoral approaches to addressing youth violence

Youth violence is a public health concern that has devastating consequences for communities and requires a collaborative community response. While public health approaches underscore the importance of collaborative efforts to address public health concerns, violence prevention programs are often siloed and lack a multisectored response that includes the education system, criminal justice system, local government, university, and service providers. This article explores the successes and lessons learned from the emergence of a multisectoral community-based initiative to address youth violence in southeastern North Carolina. The model for this initiative evolved from the Harlem Children’s Zone and includes identifying existing resources and coordinating a continuum of support and services to increase availability of after-school, summer enrichment, life-skills, and parenting programs to reduce youth violence. Implications of applying multisectoral community-based interventions are discussed in terms of capacity building, promotion of well-being, and research.     

Methodological strategies for evaluating youth gang prevention programs

This article reflects on common challenges and lessons learned during the evaluation of gang prevention programs based on case studies of three federally funded Canadian programs. Elements of evaluation design, implementation, data analysis and reporting of results are discussed. More specifically, the article highlights issues that occur when evaluating community projects focused on preventing extreme risks for violence and the complexity of working in potentially dangerous and/or unstable work environments. Topics covered include the problem with quasi-experimental designs, model fidelity adherence, program documentation, client recruitment and retention, and data collection. Recommendations are provided to improve evaluations of youth gang prevention programs and similar community-based interventions that focus on the prevention of youth violence.     

New roles for families in child welfare: Strategies for expanding family involvement beyond the case level

This paper examines the strategies and approaches child welfare agencies used to integrate meaningful family involvement into their service delivery systems, under the Children's Bureau Improving Child Welfare Outcomes Through Systems of Care demonstration initiative. Through a series of retrospective interviews with child welfare agency staff, systems of care project staff, family members actively involved in implementing the systems of care initiative, and local program evaluators, researchers investigated the family involvement planning and capacity building activities of child welfare agencies during the initiative's implementation. Findings indicate that child welfare agencies' capacity building efforts primarily centered on human resource development, focusing on three areas: program staffing, family engagement, and agency buy-in. These findings illustrate the importance of developing the capacity of child welfare agency staff and family members before fully implementing family involvement programs and activities. Although more research is needed to document the impact of family involvement, the lessons learned from these grant communities' experiences provide critical information and can inform development of policies and practices to help child welfare and other child- and family-serving systems promote and implement meaningful and sustainable family involvement.     

Evaluation of a research mentorship program in community care

This article describes the results of a qualitative case study evaluating a research mentorship program in community care settings in Ontario, Canada. The purpose of the program was to build evaluation and research capacity among staff of community care agencies through a mentorship program. Data were collected through in-depth, semi-structured interviews, focus group interviews, and written evaluations. Three themes were identified: taking hold, fostering hold, and holding on. Mentees took hold of new evaluation and research skills. Factors fostering hold included positive mentoring relationships and participation in relevant research projects. Mentees struggled to hold on to, and apply their skills within a community care context of challenges such as constrained resources and a mandate to provide client-centered care. The lessons learned include having trainees participate in relevant, time-limited evaluation and research projects, and early implementation of mechanisms to sustain integration of trainees' knowledge and skills within community care agencies.     

Evaluating the Effectiveness of Faith-Based Welfare Agencies

Abstract

This study examines the methodological challenges associated with conducting research on faith-based organizations. The arguments advanced are based on a group of comparative case studies conducted by the authors on secular and faith-based providers in three different social service domains: (1) transitional housing, (2) parent education, and (3) residential substance abuse treatment programs. All case studies utilized the same research protocol. The study identifies the lessons learned from comparative case study research on faith-based organizations when a similar protocol is implemented to examine a variety of social service domains and provider types.     

Fostering Neighborhood Involvement in Workforce Development

ABSTRACT

One of the challenges in welfare reform implementation is to provide neighborhood outreach and assistance to unemployed individuals. This case study tells the story of three neighborhoods as they implemented a public-private neighborhood jobs pilot initiative (NJPI). The Rockefeller Foundation, in partnership with Alameda County Social Services Agency, supplied seed money for the NJPI to develop “one-stop” employment resource centers. The article describes how the neighborhoods implemented the NJPI, as well as common strengths and challenges, and lessons learned from the NJPI experience.     

The opportunities and challenges of multi-site evaluations: Lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.     

Food Landscapes: Cooking,Community Service and Art-Making with Teens

ABSTRACT

In this article, we describe Food Landscapes, an interdisciplinary program involving youth who live in an urban community identified as a food desert. Food Landscapes combined socially engaged art-making, cooking, community engagement, and community service to open dialogue with youth about food justice and equity, the meaning of community, and the formation of meaningful relationships between youth and community members who have disabilities. We describe the program and discuss the theory and model bases for the program and lessons learned during program implementation.     

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.     

Conceptualizing on-campus support programs for collegiate foster youth and alumni: A plan for action

Increasingly, post-secondary education institutions are implementing supplemental support programs to assist foster youth and alumni (i.e., those formerly in foster care) matriculating into higher education. Despite the promise of these academic support programs in helping young people achieve their educational goals, the empirical educational research literature related to these support programs is nominal. This study employed Concept Mapping (CM) with a convenience sample of 51 foster youth/alumni in one southeastern state to explicate a conceptual framework for the development of campus supports for collegiate foster youth/alumni, and examine priority areas (e.g., importance and feasibility). CM is a mixed-method research approach that employs non-metric multidimensional scaling and hierarchical cluster analyses to analyze qualitative data. As a result, visual representations of the data are computed. The final cluster map yielded an eight cluster solution: Campus Awareness, Advocacy, Data Tracking, Pre-College Supports, Fostering Family Connections, Academic Financing, Campus Life, and Peer/Mentor Supports. Foster youth/alumni in this study perceived the Pre-College Supports cluster as most feasible, while the Advocacy cluster was rated at the least feasible. The Academic Financing cluster was rated as the most important; the Campus Life cluster was rated as the least important. After a review of pertinent literature, this paper explicates CM methodology as applied to the current study, reports results, and discusses lessons learned as they apply to child welfare research and practice.     

An introduction to methodological challenges in the evaluation of eHealth research: Perspectives from the Health e-Technologies Initiative

In February 2002 the Health e-Technologies Initiative (HETI), a program office of the Robert Wood Johnson Foundation^®, was created to advance discovery of scientific knowledge regarding the effectiveness of interactive eHealth applications. This article is the introduction to a series of seven articles written by grantees of HETI which address challenges, lessons learned, and proposed solutions as researchers implement eHealth projects. From this body of work it is clear that the overall process of conducting evaluation research in eHealth requires careful and detailed planning, recognition of the heightened sensitivity of IRBs, and institutions around the electronic collection and communication of personal health information, and a combination of tenacity and creativity to address the inevitable thorny methodological challenges to eHealth research. Use of established guidelines to help standardize the evaluation process, where feasible, is recommended.     

Valuing and embracing complexity: How an understanding of complex interventions needs to shape our evaluation capacities building initiatives

This paper describes some of the main challenges of evaluating complex interventions, as well as the implications of such challenges for evaluation capacity building. It discusses lessons learned from a case study of an evaluation of Dancing with Parkinson’s, an organization that provides dance classes to people with Parkinson’s disease in Toronto, Canada. These implications are developed from a realist evaluation lens. Key lessons include the need to develop skills to understand program mechanisms and contexts, recognize multiple models of causality, apply mixed method designs, and ensure the successful scaling up and spread of an intervention.     

Development of a New Approach to Guided Self-Help via the Internet: The Swedish Experience

ABSTRACT

This article describes the development and empirical status of guided Internet-delivered self-help. The treatment approach combines the benefits of bibliotherapy with book-length text materials and the support given online via web pages and e-mail. Interactive features such as online registrations, tests, and online discussion forums are also included. Cognitive behavior therapy (CBT) guided the research and clinical implementations of this approach, as it lends itself more easily to the self-help format compared with other presently available psychotherapy approaches. We include an overview of the research, current issues and research in service delivery, lessons learned through a program of research, and directions for future investigations.