Fostering CHANGE: Lessons from implementing a multi-country,multi-sector nutrition-sensitive agriculture project

Using agriculture to improve nutrition is an approach growing in popularity, with programs becoming increasingly complex and multisectoral. While there is an active line of research assessing the impacts of such programs, little has been written about the process of successfully implementing them. As such, this paper uses a multisectoral nutrition-sensitive agriculture program implemented in four African countries as a case study to address key challenges in and lessons learned from implementation. We highlight the overall flexibility of nutrition-sensitive agriculture but also the need to adapt certain aspects to the particular context, as well as the opportunities for cross-context learning (and the limits to this). Integrating rigorous evaluation into such complex programs and forging diverse cross-sectoral partnerships offer both rewards and challenges, upon which we reflect. Main lessons learned from the program include the importance of carefully sequencing interventions, retaining flexibility in implementation, allowing for considerable time for cross-sector integration and coordination, and considering community impacts when designing research.     

Strategies for Creating MIS Technology to Improve Social Work Practice and Research

SUMMARY

This paper illustrates the potential for Management Information System (MIS) technology to integrate information collection, management and reporting within a single program or network of organizations. Properly devised and created, MIS applications can improve administration, service delivery and practice evaluation. Three strategies are offered to guide the design and development of MIS software. This paper is based on lessons from the production and implementation of MIS software that serves as a management and evaluation tool for a nationwide policy demonstration. Data from the MIS have helped to shape state and federal policy.     

Food Landscapes: Cooking,Community Service and Art-Making with Teens

ABSTRACT

In this article, we describe Food Landscapes, an interdisciplinary program involving youth who live in an urban community identified as a food desert. Food Landscapes combined socially engaged art-making, cooking, community engagement, and community service to open dialogue with youth about food justice and equity, the meaning of community, and the formation of meaningful relationships between youth and community members who have disabilities. We describe the program and discuss the theory and model bases for the program and lessons learned during program implementation.     

Examining implementation and preliminary performance indicators of veterans treatment courts: The Kentucky experience

Veterans’ Treatment Courts (VTCs) are posited as a solution to offer rehabilitation for veterans involved in the criminal justice system. Despite the pervasive implementation of VTCs, there is little research focused specifically on VTC implementation and outcomes, which are based on other problem-solving court models such as drug court. The current study presents qualitative process evaluation data from key stakeholders (n = 21) and veteran participants (n = 4) to show accomplishments, challenges, and lessons learned during first-year implementation at two VTC sites. Quantitative performance data is also presented on veteran participants (n = 19) served during the first year to show: types of services, monitoring, judicial interaction, sanctions/therapeutic responses, and rewards, as well as preliminary data on recidivism. Qualitative data, from both key stakeholders and veteran participants, suggests that offering rehabilitation via various program components, services/referrals, and accountability are critical to the success of the VTC. Data also provides valuable lessons learned for VTC implementation including communication, collaboration, information/protocols, and resources. Performance data shows that a variety of services are utilized and that frequent judicial interaction, drug testing, and sanctions are cornerstones of the VTC. Implications and future directions for research are discussed.     

The relocation of nursing home residents: An illustration of the advantages gained by planning a new program and designing an implementation evaluation together

This paper describes a collaboration between an evaluation team and the management and staff responsible for relocating nursing home residents to a new, state-of-the-art facility. Both the relocation plan and the implementation evaluation of this plan are detailed in order to illustrate the link between program planning and evaluation. I argue that combining program planning and implementation evaluation activities is of value to a wide range of organizations contemplating the development and implementation of a new program. Then I draw upon the research utilization literature to discuss how the collaborative relationship between the evaluation team and management contributed to the success of both program planning and program evaluation activities.     

Looking Inside the “Black Box”

Abstract

The continued devolution of social welfare systems and services in the U.S. resultsin high stakes program evaluations in the field of family support and early intervention. Programs are expected to utilize evidence-based interventions and to demonstrate effectiveness. A look at implementation helps to differentiate between theories that do not work and programs that are not effective. Methods for identifying program implementation are needed. In a 17-site program evaluation, the author and her colleagues developed a methodology for measuring implementation and demonstrated the effects of differential implementation in understanding program outcomes.     

How local health departments work towards health equity

BackgroundHealth inequities are exacerbated when health promotion programs and resources do not reach selected populations. Local health departments (LHDs)¹ have the potential to address health equity via engaging priority populations in their work. However, we do not have an understanding of what local agencies are doing on this front.MethodsIn the summer of 2016, we collaborated with informants from thirteen LHDs across North Carolina. Via semi-structured interviews, the research team asked informants about their LHD’s understanding of health equity and engaging priority populations in program planning, implementation, and evaluation.FindingsAll informants discussed that a key function of their LHD was to improve the health of all residents. LHDs with a more comprehensive understanding of health equity engaged members of priority populations in their organizations’ efforts to a greater extent than LHDs with a more limited understanding. Additionally, while all LHDs identified similar barriers to engaging priority populations, LHDs that identified facilitators more comprehensively engaged members of the priority population in program planning, implementation, and evaluation.ConclusionsLHDs are ideally situated between the research and practice worlds to address health equity locally. To promote this work, we should ensure LHDs hold an understanding of health equity, have the means to realize facilitators of health equity work, and recognize the complex context in which health equity work exists.     

Abortion: A Technique for Working through Grief

Abstract

The successful implementation by a large public university health service of a satellite clinic involves a number of processes and funding issues. The authors discuss the development of such a program at a small community college in Appalachia. The first 4 years' experience in operating the satellite clinic created a number of valuable management lessons that they believe could assist others with similar goals for extending their services.     

A participatory evaluation framework in the establishment and implementation of transdisciplinary collaborative centers for health disparities research

ObjectiveWe describe the formulation and implementation of a participatory evaluation plan for three Transdisciplinary Collaborative Centers for Health Disparities Research funded by the National Institute of Minority Health and Health Disparities.MethodsAlthough different in scope of work, all three centers share a common goal of establishing sustainable centers in health disparities science in three priority areas – social determinants of health, men’s health research, and health policy research.ResultsThe logic model guides the process, impact, and outcome evaluation. Emphasis is placed on process evaluation in order to establish a “blue print” that can guide other efforts as well as assure that activities are being implemented as planned.ConclusionWe have learned three major lessons in this process: (1) Significant engagement, participation, and commitment of all involved is critical for the evaluation process; (2) Having a “roadmap” (logic model) and “directions” (evaluation worksheets) are instrumental in getting members from different backgrounds to follow the same path; and (3) Participation of the evaluator in the leadership and core meetings facilitates continuous feedback.     

Investigating feasibility and fidelity of the Parenting Young Children program in Sweden

Specific aimsThis study explored feasibility and fidelity of an education program for parents with intellectual disability. The Australian Parenting Young Children intervention was translated and adapted for support workers in the Swedish social services.MethodFidelity was examined using implementation diaries tracking frequency of implementation of program activities and teaching approaches ((over 14 months, completed by 27 support workers from 15 municipalities). Eighteen of these support workers and their managers (n = 12) completed questionnaires examining feasibility using Roger’s concepts of program compatibility and complexity.FindingsResults showed that implementation of Parenting Young Children was feasible, with overall positive attitudes to Evidence Based Practice, good program compatibility, low perceived program complexity, and high satisfaction with implementation support. The fidelity measures suggest problems in how the program is used by some support workers: parents received less than the recommended number of sessions and activities such as observation of parents and follow up, were infrequently used by support workers.DiscussionThe findings raise questions about how program support and education should be provided. Difficulties in defining fidelity within the flexibility of the Parenting Young Children program are also discussed.     

Valuing and embracing complexity: How an understanding of complex interventions needs to shape our evaluation capacities building initiatives

This paper describes some of the main challenges of evaluating complex interventions, as well as the implications of such challenges for evaluation capacity building. It discusses lessons learned from a case study of an evaluation of Dancing with Parkinson’s, an organization that provides dance classes to people with Parkinson’s disease in Toronto, Canada. These implications are developed from a realist evaluation lens. Key lessons include the need to develop skills to understand program mechanisms and contexts, recognize multiple models of causality, apply mixed method designs, and ensure the successful scaling up and spread of an intervention.     

An innovative medical and dental hygiene clinic for street youth: Results of a process evaluation

Canada has a noteworthy reputation for high quality health care. Nonetheless, street youth are one of our most vulnerable yet underserved populations. Consequently, a medical and dental clinic was created in downtown Ottawa, Ontario to respond to their needs. The purpose of this study is to describe a process evaluation of the clinic during its first year of operation with a focus on program fidelity, dose, reach, and satisfaction. A mixed methods approach was used involving interviews with providers, focus groups with street youth, analysis of Electronic Medical Record (EMR) data, and supplemental information such as document reviews. The evaluation identified areas that were working well along with challenges to program implementation. Areas of concerns and possible solutions were presented to the management team that then helped to plan and make improvements to the clinic. Our evaluation design and working relationship with clinic management promoted the integration of real-time evidence into program improvements.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Fostering Neighborhood Involvement in Workforce Development

ABSTRACT

One of the challenges in welfare reform implementation is to provide neighborhood outreach and assistance to unemployed individuals. This case study tells the story of three neighborhoods as they implemented a public-private neighborhood jobs pilot initiative (NJPI). The Rockefeller Foundation, in partnership with Alameda County Social Services Agency, supplied seed money for the NJPI to develop “one-stop” employment resource centers. The article describes how the neighborhoods implemented the NJPI, as well as common strengths and challenges, and lessons learned from the NJPI experience.     

Using a logic model to evaluate the Kids Together early education inclusion program for children with disabilities and additional needs

Despite clear evidence that learning and social opportunities for children with disabilities and special needs are more effective in inclusive not segregated settings, there are few known effective inclusion programs available to children with disabilities, their families or teachers in the early years within Australia. The Kids Together program was developed to support children with disabilities/additional needs aged 0–8 years attending mainstream early learning environments. Using a key worker transdisciplinary team model, the program aligns with the individualised package approach of the National Disability Insurance Scheme (NDIS).AimThis paper reports on the use of a logic model to underpin the process, outcomes and impact evaluation of the Kids Together program.MethodsThe research team worked across 15 Early Childhood Education and Care (ECEC) centres and in home and community settings. A realist evaluation using mixed methods was undertaken to understand what works, for whom and in what contexts. The development of a logic model provided a structured way to explore how the program was implemented and achieved short, medium and long term outcomes within a complex community setting.Discussion and conclusionKids Together was shown to be a highly effective and innovative model for supporting the inclusion of children with disabilities/additional needs in a range of environments central for early childhood learning and development. The use of a logic model provided a visual representation of the Kids Together model and its component parts and enabled a theory of change to be inferred, showing how a coordinated and collaborative approached can work across multiple environments.     

The State of Our Art

Abstract

In the second part of this two-part paper, the author examines intergenerational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, which appeared in the previous issue, conceptual foundations for intergenerational programming were considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross-cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

Evaluation of a comprehensive school physical activity program: Be a Champion!

BackgroundComprehensive School Physical Activity Programs (CSPAPs) are widely supported to increase physical activity (PA) in schools, but little has been reported to inform a comprehensive evaluation effort. The purpose of this report is to describe a comprehensive evaluation and monitoring strategy for CSPAP implementation.MethodsThe System for Observing Staff Promotion of Activity and Nutrition (SOSPAN), System for Observing Student Movement in Academic Routines and Transitions (SOSMART), and System for Observing Play and Leisure Activity (SOPLAY) were employed to record student and staff behaviors (2642 total scans). To assess policies, practices, and environments, a CSPAP Policies and Practices questionnaire was used, which included components of the School Health Index, the School PA Policy Assessment, the School Environment and Policies Survey, and the School Health Policies and Practices Study. Youth PA (minutes/day) was assessed using accelerometers.ResultsWorking with school sites to ascertain the number of opportunities that are provided for PA and the amount of PA students engage in is challenging but feasible. The use of systematic observation, accelerometers, and questionnaires can provide a comprehensive representation of policies, practices, and behaviors to aid in establishing targeted PA action items.ConclusionsDeveloping an evaluation framework that was rigorous enough to assess the physical, social, and policy environment of a school in order to answer the overarching research questions of the study but flexible enough to be feasibly streamlined for utilization by a school implementation team proved to be challenging but possible.     

Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.