Implementing Single Session Family Consultation: A Reflective Team Approach

Setting up and implementing a single session family consultation in a specialist child and adolescent mental health setting, which for a long time has been an organisation and culture that has undertaken a traditional detailed child and family assessment and treatment, brings with it challenges. This paper will look at the history of single session consultation within the field of child and family services and, in particular, child and family services in mental health. It will also look at the development and implementation of a single session therapy program, and associated research on the effectiveness of the program, at the Alfred Child and Youth Mental Health Service. Clinical examples will assist in illustrating the program and its model and structures. Lastly, it will review the research data and compare this with other research data in the field.     

Evaluation of a complex, multisite, multilevel grants initiative

The Safe Schools/Healthy Students (SS/HS) national evaluation seeks to assess both the implementation process and the results of the SS/HS initiative, exploring factors that have contributed to or detracted from grantee success. Each site is required to forge partnerships with representatives from education, mental health, juvenile justice, and law enforcement, coordinating and integrating their efforts and working together to contribute to comparable outcomes (e.g., reduced violence and alcohol and drug use, improved mental health services). The evaluation uses multiple data collection techniques (archival data, surveys, site visits, interviews, and focus groups) from a variety of sources (project directors, community partners, schools, and students) over several years. Certain characteristics of the SS/HS initiative represent unique challenges for the evaluation, including the absence of common metrics for baseline, outcome data, and lack of comparison group. A unifying program theory was required to address these challenges and synthesize the large amounts of qualitative and quantitative information collected. This article stresses the role of program theory in guiding the evaluation.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

Improving productivity in child welfare programs

The project met its objective of finding ways to improve the efficiency of a child welfare program. The agency will continue an intensive evaluation of the program to reassess the procedures' impact on productivity. The project found that, with adaptation, work-methods techniques developed in industry can be useful to child welfare programs. Critical elements in the success of a productivity improvement project include full, visible support by the agency's policy makers and top management; early intensive involvement of affected personnel; wide implementation of recommended improvements; and continued review of their effectiveness. Although no single work-methods project can solve all problems connected with an agency's way of carrying out its work, the process developed by the Texas Department of Human Resources can be applied usefully by both public and private child welfare agencies as they seek ways to improve the efficiency of their programs.     

Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

The opportunities and challenges of multi-site evaluations: Lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.     

From plan to action: Adapting evaluation to serve the developmental needs of a newly-funded multidisciplinary research center

The Southeastern Coastal Center for Agricultural Health and Safety (SCCAHS) is one of many newly-funded federal research centers, housing five multidisciplinary research projects and seven pilot projects, and serving a multi-state region. In the early stages of such a complex project, with multiple teams separated by geography and disciplines, the evaluation program has been integral in connecting internal and external stakeholders at the center and project levels. We used a developmental evaluation (DE) framework to respond to the complex political environment surrounding agricultural health and safety in the southeast; to engage external stakeholders in guiding the center’s research and outreach trajectories; to support center research teams in a co-creation process to develop logic models and tailored indicators; and to provide timely and feedback within the center to address communications gaps identified by the evaluation program. By using DE principles to shape monitoring and evaluation approaches, our evaluation program has adapted to the dynamic circumstances presented as our center’s progress has been translated from a plan in a grant proposal to implementation.     

Organizational capacity building: addressing a research and practice gap

The purpose of this article is to address the gap between evaluation research, and the practice of capacity building with nonprofits. This study describes a 5-year capacity building initiative with grassroots organizations including a longitudinal evaluation of the implementation and outcomes achieved. Formative processes yielded many lessons that were used to improve the capacity building model of services. The results show that the majority of groups met a priori expectations for participation success. Organizational staff valued technology, consultants, and program funding the most. Increases were found in board membership and perceptions of visibility of the organization were enhanced. Executive directors reported greater awareness of needs and improved management knowledge. These small organizations fill many unmet needs and more capacity building evaluation studies are needed to understand the mechanisms that support their efforts and the impact on their sustainability.     

Using process evaluation to determine effectiveness of participatory ergonomics training interventions in construction

The construction industry continues to experience high rates of musculoskeletal injuries despite the widespread promotion of ergonomic solutions. Participatory ergonomics (PE) has been suggested as one approach to engage workers and employers for reducing physical exposures from work tasks but a systematic review of participatory ergonomics programs showed inconclusive results.. A process evaluation is used to monitor and document the implementation of a program and can aid in understanding the relationship between the program elements and the program outcomes. The purpose of this project is to describe a proposed process evaluation for use in a participatory ergonomic training program in construction workers and to evaluate its utility in a demonstration project among floor layers.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Looking Inside the “Black Box”

Abstract

The continued devolution of social welfare systems and services in the U.S. resultsin high stakes program evaluations in the field of family support and early intervention. Programs are expected to utilize evidence-based interventions and to demonstrate effectiveness. A look at implementation helps to differentiate between theories that do not work and programs that are not effective. Methods for identifying program implementation are needed. In a 17-site program evaluation, the author and her colleagues developed a methodology for measuring implementation and demonstrated the effects of differential implementation in understanding program outcomes.     

Implemented as intended? Recording family worker activities in a Families First program

In evaluating family preservation services, it is important not only to study the service outcomes and the family characteristics, but also what actually happens during the treatment. This requires a program model. This article describes how a program model works, prescribes how workers should carry it out, and describes how researchers should measure the program's characteristics. The authors use data from Families First in The Netherlands to test the model. Results show that the method of the program meets the specified characteristics. The results are important for treatment, policy, education, and evaluation research.     

Dosage effects on school readiness: evidence from a randomized classroom-based intervention

Variations in the dosage of social interventions and the effects of dosage on program outcomes remain understudied. This study examines the dosage effects of the Chicago School Readiness Project, a randomized, multifaceted classroom-based intervention conducted in Head Start settings. Using a principal score matching method to address the issue of selection bias, the study finds that high-dosage levels of teacher training and mental health consultant class visits have larger effects on children's school readiness than the effects estimated through intention-to-treat (ITT) analyses. Low-dosage levels of treatment are found to have effects that are smaller than those estimated in ITT analyses or to have no statistically significant program effects. Moreover, individual mental health consultation services provided to high-risk children are found to have statistically significant effects on their school readiness. The study discusses the implications of these findings for research and policy.     

Treatment services received in the CASAWORKS for Families program

This article presents information on treatment services received by women participating in an initial multistate evaluation of CASAWORKS families. Results indicated most women received services to address medical, employment, basic needs, alcohol and drug, family, and psychiatric problems during the first six months of the program. The clients also had frequent contact with their case managers and were retained in the program for an average of 222 days. Considerable variation was observed across sites in the percentage of clients who received various services and the number of sessions they received. In Cox regressions, shorter retention in the program was predicted by referral to program from Child Protective Services or parole/probation, social conflicts, employment, and marijuana use at baseline, whereas a history of suicide attempts was associated with longer retention. Longer retention was associated with better alcohol use outcomes but was unrelated to employment or drug use outcomes.     

Evidence-based Medicine and Child Mental Health Services

There is increased focus on using research evidence to guide clinical practice in the provision of child mental health services, to allocate resources and to make policy decisions. Emphasis has been placed on randomised controlled trials, which are clearly vital. However, given the complexity of children's lives and the multiple systems relevant to their care, it is important to consider alternative research designs and to think creatively about outcomes. In the context of difficulties with random allocation a number of quasi-experimental designs are described that may more closely reflect actual clinical work. In the context of ecological theory, the importance of taking a systems level approach to evaluation is highlighted, with examples from the US. Finally, the ways in which qualitative outcomes measures can be used to sensitively describe the effects of treatment are outlined, and their potential for giving voice to children and families as consumers of mental health services. These suggestions aim to lessen the tension between the requirements of the scientific investigations of treatment effectiveness and the realities of clinicians' daily experiences.     

The implementation of the healthy cities project in developing countries: lessons from Chittagong

The Chittagong Healthy City Project was carried out in late 1994 in Chittagong, Bangladesh. This paper presents findings of an evaluation of the project based upon internationally generated process indicators related to the institutional aspects of the project. The following issues are discussed with regard to project implementation: the institutional organization of local authorities, institutions' conceptual understanding of the project, formal insertion of the project into public authorities' activities, institutional leadership of the project, central-local relations, the lack of interministerial coordination, the project's office, international projects, and community organization. Giving consideration to these issues may help program planners detect problems in forthcoming projects prior to their implementation.     

IMPLEMENTATION OF PARENT MANAGEMENT TRAINING AT THE NATIONAL LEVEL: THE CASE OF NORWAY

This article describes early aspects of the nationwide implementation of an evidence‐based program (EBP) in Norway and the design for studying program fidelity over time. The EBP is the Parent Management Training, Oregon Model (PMTO). The project is a combination of a ‘top down’ initiative at the governmental and ministry level, and a ‘bottom up’ initiative from practitioners seeking effective interventions in the prevention and remediation of behavior problems in children and youth. The main components of the implementation strategy were to (a) establish a national implementation and research center; (b) provide for regional and local participation at county and municipal levels; (c) establish a comprehensive therapist recruitment, training, and maintenance program; (d) create a network for collaboration, supervision, and quality control; (e) conduct clinical outcome research; and (f) study the implementation process. Following the training of three successive generations of PMTO specialists, a study was designed to examine how individual, family, interventionist, organizational, and community characteristics influence program adoption and implementation, as well as how these factors impact upon how PMTO specialists provide the intervention to families with competent adherence to the model. Within the framework of the collaborative North American and Norwegian project, challenges and facilitators in the process of the continental crossing of the PMTO model are discussed.     

The evaluation and monitoring of a Medicaid second surgical opinion program.

The evaluation of newly implemented and controversial programs often present real problems for the evaluator. When a program is accompanied by a legislatively mandated evaluation, the issues become even more acute. The evaluator, wanting to impact on policy, must strike a balance between the adherence to principles of scientific inquiry, yet be responsive to the needs of policymakers. This paper describes the evaluation and follow-up monitoring of a recently established second opinion program for Medicaid recipients. The program was to terminate in 17 months unless extended by the Wisconsin Legislature. The evaluation found, among other things, marked reductions in the use of surgical services after implementation of the second opinion program. Based in large part on the evaluation, the program was extended with a provision for continued monitoring. The paper discusses the factors that contributed to the evaluation having a major impact on the decision to extend the program.     

Scaling up a community-led health promotion initiative: Lessons learned and promising practices from the Healthy Weights for Children Project

The increase in overweight and obesity among children has emerged as an important public health issue. This trend has highlighted the need for accessible and novel approaches to support healthy weights for children and their families to prevent childhood obesity. The purpose of this article is to describe the iterative development and scale-up of a community-led, national-level project to promote healthy weights among Canadian children and families who may be experiencing vulnerabilities. In this project, the Healthy Together program was designed to engage families in an interactive program to support healthy lifestyles. The program also provides a platform for creating supportive environments for healthful lifestyles through practice and policy change. Based on a process evaluation, we describe the iterative development of Healthy Together from Phase 1 through 3 to shed light on processes shaping implementation and scale-up of the program. Lessons learned during each phase were used to refine the program and further expansion. Indicators of successful scale-up include the Healthy Together program’s cross-jurisdictional reach and promising evaluation results in real-world conditions. The practice-based program scaling approach provides practical guidance for planning and implementing similar health promotion programs in diverse communities.