Workforce training in family therapy and systemic practices: An evaluation framework and case study

Internationally, while the interdisciplinary field of family therapy and systemic practices is well established and evidence for therapeutic impact is advanced, evaluation of training and measurement of its impact lags behind. This paper addresses this gap, utilising a single case study at the Bouverie Centre, the largest family therapy and workforce development service in Australia. We describe (a) the scope of workforce training and implementation at the Centre, (b) the development of our training research and evaluation framework, and (c) standardised data collection methodology, including a pre-post training reflective survey. Collectively, this work comprises our current evaluation framework through which we will subsequently examine training impact. Future research and practice recommendations are offered to address challenges to workforce training evaluation and progress efficacy of family therapy training.     

A comprehensive health service evaluation and monitoring framework

ObjectiveTo develop a framework for evaluating and monitoring a primary health care service, integrating hospital and community services.MethodA targeted literature review of primary health service evaluation frameworks was performed to inform the development of the framework specifically for remote communities. Key principles underlying primary health care evaluation were determined and sentinel indicators developed to operationalise the evaluation framework. This framework was then validated with key stakeholders.ResultsThe framework includes Donabedian's three seminal domains of structure, process and outcomes to determine health service performance. These in turn are dependent on sustainability, quality of patient care and the determinants of health to provide a comprehensive health service evaluation framework. The principles underpinning primary health service evaluation were pertinent to health services in remote contexts. Sentinel indicators were developed to fit the demographic characteristics and health needs of the population. Consultation with key stakeholders confirmed that the evaluation framework was applicable.ConclusionData collected routinely by health services can be used to operationalise the proposed health service evaluation framework. Use of an evaluation framework which links policy and health service performance to health outcomes will assist health services to improve performance as part of a continuous quality improvement cycle.     

Multilevel network interventions: Goals,actions, and outcomes

COVID-19 has resulted in dramatic and widespread social network interventions across the globe, with public health measures such as distancing and isolation key epidemiological responses to minimize transmission. Because these measures affect social interactions between people, the networked structure of daily lives is changed. Such largescale changes to social structures, present simultaneously across many different societies and touching many different people, give renewed significance to the conceptualization of social network interventions. As social network researchers, we need a framework for understanding and describing network interventions consistent with the COVID-19 experience, one that builds on past work but able to cast interventions across a broad societal framework. In this theoretical paper, we extend the conceptualization of social network interventions in these directions. We follow Valente (2012) with a tripartite categorization of interventions but add a multilevel dimension to capture hierarchical aspects that are a key feature of any society and implicit in any network. This multilevel dimension distinguishes goals, actions, and outcomes at different levels, from individuals to the whole of the society. We illustrate this extended taxonomy with a range of COVID-19 public health measures of different types and at multiple levels, and then show how past network intervention research in other domains can also be framed in this way. We discuss what counts as an effective network, an effective intervention, plausible causality, and careful selection and evaluation, as central to a full theory of network interventions.     

Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

The process of evaluating a capacity-building support initiative for HIV positive South Africans

This paper describes the process of evaluating a capacity-building initiative involving HIV positive South Africans. The evaluation was conducted within a participatory evaluation framework, which aims at empowering participants through social action. The rationale is that an understanding and acceptance of evaluation results promotes commitment to act. Forty four members of the Kudu Support Group and 23 non-members participated in the evaluation. Methods included in-depth interviews, a survey, repeated measures of the General Health Questionnaire-28 (GHQ-28), a SWOT analysis, interviews with health workers, field notes, and participatory observation. The capacity-building initiative resulted in empowerment through paid employment for support group members, the establishment of a vegetable garden, increased skills and efficacy, access to resources and networks, a perceived reduction of stigma, perceived improvement of social relations, and improved scores on the GHQ-28. Lessons that emerge out of this study are the importance of integrating evaluation as a part of ongoing activities throughout the research project; to ensure that methods and results are understandable and useful to for the participants; that the evaluator has an in-depth understanding of the project and context; and to introduce frequent and simple internal evaluation tools to guide efforts to improve community-based groups’ functioning and activities.     

An application of the Impact Evaluation Process for designing a performance measurement and evaluation framework in K-12 environments

This article illustrates the application of the Impact Evaluation Process for the design of a performance measurement and evaluation framework for an urban high school. One of the key aims of this framework is to enhance decision-making by providing timely feedback about the effectiveness of various performance improvement interventions. The framework design process is guided by the Impact Evaluation Process, and included the participation of key stakeholders including administrative and teaching staff who all contributed to the performance measurement and evaluation framework design process. Key performance indicators at the strategic, tactical, and operational levels were derived from the school vision, and linked to specific interventions to facilitate the continuous evaluation and improvement process.     

Framework for culturally competent decisionmaking in child welfare

This article provides a framework to understand the cultural, social, political, and economic factors that affect decisionmaking when working with ethnically and racially diverse families in the child welfare system. The article describes external factors affecting the decisionmaking process, including community environment, agency structure, and family characteristics. It then reviews the core stages of the casework process, describing key decisions during intake, assessment, service planning, implementation, evaluation, and closure. Although the framework is based on casework process in the child welfare system, it can be adapted to other child-serving systems, including education, mental health, and juvenile justice.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.     

The Professional Schools. By William J. McGlothlin. New York: The Center for Applied Research in Education, 1964. 111 pp. $3.95

This paper presents a conceptual framework for viewing social planning and social administration as one “seamless web” of activity concerned with the identification of goals and the development of appropriate administrative structures for their implementation. The major educational implication of this analysis is that curriculum planning should maximize the integration of social policy, social planning, and administration. Alternative approaches to achieving this integration are discussed using data from a recent Council on Social Work Education survey.     

Understanding the factors that influence health promotion evaluation: The development and validation of the evaluation practice analysis survey

The demand for improved quality of health promotion evaluation and greater capacity to undertake evaluation is growing, yet evidence of the challenges and facilitators to evaluation practice within the health promotion field is lacking. A limited number of evaluation capacity measurement instruments have been validated in government or non-government organisations (NGO), however there is no instrument designed for health promotion organisations. This study aimed to develop and validate an Evaluation Practice Analysis Survey (EPAS) to examine evaluation practices in health promotion organisations. Qualitative interviews, existing frameworks and instruments informed the survey development. Health promotion practitioners from government agencies and NGOs completed the survey (n = 169). Principal components analysis was used to determine scale structure and Cronbach’s α used to estimate internal reliability. Logistic regression was conducted to assess predictive validity of selected EPAS scale. The final survey instrument included 25 scales (125 items). The EPAS demonstrated good internal reliability (α > 0.7) for 23 scales. Dedicated resources and time for evaluation, leadership, organisational culture and internal support for evaluation showed promising predictive validity. The EPAS can be used to describe elements of evaluation capacity at the individual, organisational and system levels and to guide initiatives to improve evaluation practice in health promotion organisations.     

Linking child welfare and mental health using trauma-informed screening and assessment practices

An abundance of research suggests that children in the child welfare system (CWS) have experienced numerous traumatic events and are exhibiting traumatic stress symptoms. Therefore, it is critical that the CWS work closely with the mental health system to ensure that these children receive the appropriate trauma screening, trauma-focused assessment, and referral to the appropriate trauma-focused mental health services. This paper will begin by providing a concrete definition of trauma-focused screening and highlighting how that differs from a more comprehensive trauma-focused assessment process and a psychological evaluation. From there, the authors will highlight existing trauma-focused screening and assessment tools that are used widely within CWSs and the challenges related to integrating trauma-focused screening practices into CWSs. The authors will provide recommendations for ways in which child welfare jurisdictions can integrate trauma-focused screening practices into their daily practice.     

A GENOGRAM WITH AN ATTITUDE

The approach described in this paper is predicated on the fundamental belief that in order to become competent and ethical practitioners, students must understand themselves and how they see others. They must be given tools and skills that facilitate examinations of their own assumptions and beliefs about themselves, other, and how the world works. It is also essential that students examine how these assumptions and beliefs will influence the way they choose to conduct therapy. Once they are aware of their biases, they must learn to choose to consciously influence themselves in a way that permits their clients the largest room for change within the clients' own contexts of belief, understanding, experience, and possibility. In this paper I describe the use of the genogram as a tool to facilitate the process of gaining self knowledge. This tool has been used in a multicultural counseling course, as well as in professional development workshops for psychologists, mental health counselors, and marriage and family therapists.     

Using developmental evaluation as a system of organizational learning: An example from San Francisco

In the last 20 years, developmental evaluation has emerged as a promising approach to support organizational learning in emergent social programs. Through a continuous system of inquiry, reflection, and application of knowledge, developmental evaluation serves as a system of tools, methods, and guiding principles intended to support constructive organizational learning. However, missing from the developmental evaluation literature is a nuanced framework to guide evaluators in how to elevate the organizational practices and concepts most relevant for emergent programs. In this article, we describe and reflect on work we did to develop, pilot, and refine an integrated pilot framework. Drawing on established developmental evaluation inquiry frameworks and incorporating lessons learned from applying the pilot framework, we put forward the Evaluation-led Learning framework to help fill that gap and encourage others to implement and refine it. We posit that without explicitly incorporating the assessments at the foundation of the Evaluation-led Learning framework, developmental evaluation’s ability to affect organizational learning in productive ways will likely be haphazard and limited.     

Tobacco Control and Smoking Cessation Efforts in an Inner-City African American Community

This paper describes the conceptual framework that guided the development and implementation of a large-scale, community-based health initiative to lower the prevalence of smoking in an urban African American community. This project developed culturally-sensitive approaches to reducing smoking in the community and to promoting tobacco control efforts developed and implemented by community members. A randomized clinical-trial methodology was used to test the efficacy of the culturally-sensitivey community-developed smoking cessation interventions in lowering smoking rates as compared with a self-help approach. Two theoretical models guided the intervention strategies: a macro-level model applicable to the community as a whole, and a psychological process model applicable to individuals. The community model was based on community systems theory and incorporated the Readiness for Change Model which was applied in both the individual and organizational models. In addition, culturally-sensitive data collection methods were developed to improve the reliability and validity of project data, especially in determining the smoking prevalence rates and smoking behaviors of hard-to-reach, inner-city African Americans. Since the health of individuals is related to the health of their communities, smoking cessation and tobacco control activities that are integrated into the framework of the community (Le., churches, city-council, housing developments, community organizations), and incorporate culturally-relevant and specific interventions can be effective methods for achieving behavioral and societal change.     

Measuring Walsh’s Family Resilience Framework: Reliability and Validity of the Family Resilience Assessment Among Women with a History of Breast Cancer

Walsh’s family resilience framework presents a comprehensive and inclusive lens of family process that may assist clinicians, family service professionals, and health care providers with understanding the functioning of individuals and families in their care. A mechanism for using this framework is needed, as is empirical evidence of its cogency. This article reports on the development of the Family Resilience Assessment, a 29-item survey intended to assess dimensions of Walsh’s framework. The Family Resilience Assessment is presented as a reliable and valid mechanism for assessing the family resilience framework. Results of the use of the survey with women with a history of breast cancer demonstrate that it is a reliable instrument with content and construct validity.     

Tobacco control and smoking cessation efforts in an inner-city African American community

This paper describes the conceptual framework that guided the development and implementation of a large-scale, community-based health initiative to lower the prevalence of smoking in an urban African American community. This project developed culturally-sensitive approaches to reducing smoking in the community and to promoting tobacco control efforts developed and implemented by community members. A randomized clinical-trial methodology was used to test the efficacy of the culturally-sensitive, community-developed smoking cessation interventions in lowering smoking rates as compared with a self-help approach. Two theoretical models guided the intervention strategies: a macro-level model applicable to the community as a whole, and a psychological process model applicable to individuals. The community model was based on community systems theory and incorporated the Readiness for Change Model, which was applied in both the individual and organizational models. In addition, culturally-sensitive data collection methods were developed to improve the reliability and validity of project data, especially in determining the smoking prevalence rates and smoking behaviors of hard-to-reach, inner-city African Americans. Since the health of individuals is related to the health of their communities, smoking cessation and tobacco control activities that are integrated into the framework of the community (i.e., churches, city-council, housing developments, community organizations), and incorporate culturally-relevant and specific interventions can be effective methods for achieving behavioral and societal change.     

Measuring organizational health in child welfare agencies

In recent years, federal funds have been directed toward programs to improve the organizational health and functioning of public and tribal child welfare agencies. This study introduces a battery of instruments aimed at holistic measurement of organizational health factors in public, private, and tribal child welfare agencies. The Comprehensive Organizational Health Assessment (COHA) was designed as part of a federally funded grant to aid development of organizational interventions in child welfare settings. Assessment results are used diagnostically to identify strengths and challenges and to guide the development of targeted systems-change interventions and to track change over time. This study describes the iterative process of developing and testing the measurement tools, including results from exploratory and confirmatory factor analyses. The final, revised set of measures includes 20 scales and 3 indices measuring aspects of individual, work unit, and organizational functioning.