Perceived family impact of volunteering among reintegrating post-9/11 Veterans

Family issues are common among returned post-9/11 Veterans. Traumatic brain injury (TBI), post-traumatic stress disorder (PTSD), and depression are each independently related to divorce whereas community ties and social support are protective factors for the family during reintegration. Evidence from elders on the benefits of one intervention, community volunteering, may indicate “spillover effects” of these benefits into the family. Few measures exist to assess the impact of military Veteran volunteering on the family. The authors report (1) an adaption of a benefits measure from elders to Veterans, (2) its preliminary reliability and validity, and (3) differences among subgroups. Reintegrating post-9/11 Veterans (N = 346) who completed a 6-month, stipended volunteer program were surveyed. Perceived impact of volunteering on the family was assessed after completion of the program using an 11-item self-report measure. Rank-based nonparametric tests were used to detect significant differences among subgroups. Preliminary findings support the scale’s adaptation to Veterans, internal consistency, and construct validity. At least one perceived family impact indicator differed significantly (p < .05) between subgroups based on demographic and psychological factors. Veterans in this civic service program perceived that their volunteering may have affected their families.     

The opportunities and challenges of multi-site evaluations: Lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.     

Puberty Education in a Global Context: Knowledge Gaps,Opportunities, and Implications for Policy

Youth programs and policies provide opportunities for institutions and societies to support healthy adolescent development. Puberty education programs are universally important, as they provide crucial knowledge and skills to help youth and their caregivers navigate the physical, emotional, and interpersonal changes of puberty with positive outcomes. However, few puberty programs have been rigorously evaluated, resulting in a lack of evidence‐based knowledge and practice in this area. This review examines the status of research on puberty education and related programs and draws on the broader intervention literature and recent research findings on adolescence to identify program features that might improve program effectiveness. Implications for policy are also discussed. The need for rigorous program evaluation is emphasized throughout.     

Strengthening child and youth programs: A look at inter-organizational mentoring strategies

Community-based non-profit organizations rarely have access to research or evaluation evidence to inform their programs and often lack the capacity to gather or use this information independently. In 2016, Wisdom2Action—a network of knowledge mobilization (KMb) experts, policy makers and service providers across Canada—launched an inter-organizational mentorship program to facilitate the implementation and sharing of best and promising practices within community-based programs for young people. This article outlines the findings from a developmental evaluation of eight mentoring relationships. Drawing on the Promoting Action on Research in Health Sciences (PARiHS) model of KMb, we look at mentoring as a type of facilitation that supports the increased use of evidence and evaluation information by non-profit organizations and identify key themes that support effective organizational mentorship in this sector. Findings reinforce the importance of establishing connected relationships and understanding context in mentoring relationships, creating adaptive and responsive work plans, ensuring consistent communication, and maintaining a focus on capacity-building if knowledge mobilization is to occur.     

Analyzing free-text survey responses: An accessible strategy for developing patient-centered programs and program evaluation

Despite widespread availability of yoga in the Veterans Health Administration (VA), it remains unclear how to best evaluate yoga programs. This is particularly problematic for programs aimed at veterans with mental health concerns, as evaluation typically focuses narrowly upon mental health symptom severity, even though program participants may have other health-related priorities. We analyzed responses to free-text questions on 237 surveys completed by veterans with mental health concerns enrolled in a yoga program at six VA clinics in Louisiana to characterize veteran participants’ experiences with yoga. Qualitative analysis resulted in 15 domains reflecting veterans’ individual health-related values and priorities. We use results to illustrate the potential for analysis of free-text responses to reveal valuable insights into patient experiences, demonstrating how these data can inform patient-centered program evaluation. The approach we present is more accessible to those responsible for decision-making about local programs than conventional methods of analyzing qualitive evaluation data.     

VA Staff Perceptions of Barriers and Facilitators to Home-and Community-Based Placement Post–Hospital Discharge

This study identifies factors U.S. Department of Veterans Affairs (VA) staff perceived to promote or impede home- and community-based services (HCBS) placement post–hospital discharge among Veterans cared for within the VA. Data derive from 35 semi-structured interviews with staff from 12 VA medical centers from around the country. VA staff reported that Veteran’s care needs and social and financial resources influence HCBS placement. They also reported prerequisites for successful placement, including housing, unpaid informal care, and non-VA services funded privately and by public programs such as Medicaid and the Older Americans Act. Lack of staffing and failure to offer the specific types of services needed limit referral to and use of HCBS. Budgetary imperatives influence the relative availability of HCBS across VA medical centers. Findings highlight patient-, provider-, and system-level constraints that impede successful placement at home and in the community of Veterans in need of long-term services and supports after hospitalization.     

Why so many “rigorous” evaluations fail to identify unintended consequences of development programs: How mixed methods can contribute

Many widely-used impact evaluation designs, including randomized control trials (RCTs) and quasi-experimental designs (QEDs), frequently fail to detect what are often quite serious unintended consequences of development programs. This seems surprising as experienced planners and evaluators are well aware that unintended consequences frequently occur. Most evaluation designs are intended to determine whether there is credible evidence (statistical, theory-based or narrative) that programs have achieved their intended objectives and the logic of many evaluation designs, even those that are considered the most “rigorous,” does not permit the identification of outcomes that were not specified in the program design. We take the example of RCTs as they are considered by many to be the most rigorous evaluation designs. We present a numbers of cases to illustrate how infusing RCTs with a mixed-methods approach (sometimes called an “RCT+” design) can strengthen the credibility of these designs and can also capture important unintended consequences. We provide a Mixed Methods Evaluation Framework that identifies 9 ways in which UCs can occur, and we apply this framework to two of the case studies.     

Evaluation and Evaluative Rigor in the Nonprofit Sector

Nonprofit organizations are under more pressure than ever to provide “rigorous” evidence of program impact. However, as expectations for rigorous evidence rise, many nonprofits struggle with challenges that inhibit evaluation. Recognizing these trends and tensions, this study identifies catalysts and obstacles to evaluation activity and the correlates of evaluative rigor among US nonprofits based on a national survey of registered public charities (n = 311). Results reveal that the most important catalysts to evaluation include the desire to improve program effectiveness and legitimacy, while the most important obstacles include insufficient time and money. Moreover, regression analysis finds that evaluation appears to be most rigorous when (1) evaluation is a priority, (2) a supportive organizational culture exists, (3) management requires evaluation, (4) evaluation is not primarily motivated by personal interest, and (5) evaluation is likely to reveal success. Overall, intrinsically motivated evaluation appears to be more rigorous than externally mandated evaluation, suggesting that stakeholders should work to help capacitate receptive nonprofits to improve evaluative rigor instead of imposing external requirements.     

Military spouses speak up: A qualitative study of military and Veteran spouses’ perspectives

ABSTRACT

The purpose of this qualitative study was to examine military family life through the lens of military and Veteran spouses. Twenty-two female, active-duty military and Veteran spouses provided extensive written responses to essay questions as part of an online survey that were analyzed with qualitative data analysis methods. Findings revealed seven main themes: how military life has influenced health care access, spouses’ identities as caregivers, marital relationships, health outcomes, social support, spouses’ educational and career opportunities, as well as their personal growth. In summary, their experiences indicate that military service affects all aspects of their lives. Spouses gave numerous suggestions as to how service providers can address their unique experiences to improve the supports they offer military families, including ensuring the spouse is involved in health care, and providing support programs that focus on the entire family.     

Effectiveness of the Coral Tree Family Service family inpatient unit: a longitudinal study exploring change in family functioning

Child mental health disorders represent a major burden to public health in Australia due to high prevalence rates, the widespread impact across domains, and the potential for difficulties to persist into adolescence and adulthood. Extensive evidence exists for the use of parent management training and cognitive behavioural therapy to treat difficulties experienced by children; however, a proportion of children do not benefit as expected from these treatments. The use of complementary therapeutic approaches and variations to the mode and intensity of existing intervention is warranted; family inpatient units represent a unique example of this. The aim of this study is to evaluate the effectiveness of a well-established Australian inpatient unit that admits the whole family, for children aged 12 years and younger referred with mental health, behavioural, or emotional difficulties. This study's longitudinal within-subject study design utilised routinely collected outcome data from the systemic clinical outcome and routine evaluation, 15 item (SCORE-15)—a valid and reliable measure of family functioning—from a sample of 980 participants attending the Family Residential Program. This study reported significant improvement in family functioning across all outcome variables between baseline and post-intervention, with no variables returning to pre-intervention levels at follow-up. Clinically significant changes in overall family functioning showed that 37% of participant scores moved from the clinical range at baseline to the nonclinical range post-intervention. This study represents the first empirical evaluation of the Family Residential Program, and the reported results provide compelling evidence for the program to effect improvement in family functioning for families with long-standing and severe difficulties. Poor rates of questionnaire completion following intervention frequent the literature base of family inpatient units, with low post-intervention (56%) and follow-up (12%) rates evident in this study. This study uniquely contributes to the growing evidence base of family inpatient units using rigorous evaluation methods.     

Challenges and successes to the implementation of a zoo conservation-education program

Zoo conservation-education programs have potential to address biodiversity loss, although evidence of their effectiveness to encourage social change is in its infancy. Moreover, how a program is implemented may influence program efficacy, yet there is little evidence of the factors that shape this process in zoos. Accordingly, through a process evaluation of Zoos Victoria’s ‘When Balloons Fly” (WBF) conservation-education program against marine debris, we identified barriers and enablers to the implementation of the program which can be addressed to improve future initiatives. Between April-May 2018, 24 Zoos Victoria professionals completed an online survey focussed on identifying challenges and successes in implementing WBF. Four participants additionally completed a follow-up telephone interview. Themes were identified and organised according to the model of diffusion in service organisations (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). Our results illuminate that features of the organisation (e.g., structure, culture) rather than features of individuals (e.g., skills, motivation) or characteristics of the conservation-education program itself (e.g., messaging), were most influential to implementation success. Based on these findings, attention to the organisational context is critical for promoting and evaluating the success of conservation-education programs and must be given significant attention alongside program characteristics and staff capability to deliver them.     

A Model for Developing and Managing Distance Education Programs Using Interactive Video Technology

Based on two years of experience with a distance MSW program, the authors present an evolving model for the development, management, and evaluation of distance education graduate programs that use interactive video technology. The model includes five core components: (a) accreditation standards compliance, (b) resource requirements, (c) curriculum adaptation, (d) faculty development, and (e) program evaluation. While not an exhaustive listing, these components are advanced as central to the effective planning and administration of distance education programs in social work.     

Portfolio assessment in a collaborative program evaluation: the reliability and validity of a family literacy portfolio

Evaluations increasingly use open-ended assessments, such as portfolios, to gather information about program impact. Portfolios, however, present challenges in achieving acceptable levels of reliability and often lack evidence to support validity. This article describes the components of a portfolio assessment system developed in a collaborative evaluation of a family literacy program. Staff of a family literacy program and two evaluators collaborated to develop the assessment and the tools that frame the review of the portfolios. Rubrics, exemplar portfolios, portfolio logs, and lists of ‘look-fors’ were developed to guide raters in the scoring and to improve the reliability of scores and validity of decisions about the program. Inter-rater reliability for the portfolio assessment was 0.79 for holistic scores and 0.74 for analytic scores. With respect to validity, holistic and analytic scores correlated with criterion measures from 0.50 to 0.59, providing initial evidence of a common construct.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Ten steps to making evaluation matter

This paper proposes ten steps to make evaluations matter. The ten steps are a combination of the usual recommended practice such as developing program theory and implementing rigorous evaluation designs with a stronger focus on more unconventional steps including developing learning frameworks, exploring pathways of evaluation influence, and assessing spread and sustainability. Consideration of these steps can lead to a focused dialogue between program planners and evaluators and can result in more rigorously planned programs. The ten steps can also help in developing and implementing evaluation designs that have greater potential for policy and programmatic influence. The paper argues that there is a need to go beyond a formulaic approach to program evaluation design that often does not address the complexity of the programs. The complexity of the program will need to inform the design of the evaluation. The ten steps that are described in this paper are heavily informed by a Realist approach to evaluation. The Realist approach attempts to understand what is it about a program that makes it work.     

Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.     

Cultural adaptation and implementation of evidence-based parent-training: A systematic review and critique of guiding evidence

With advances in knowledge regarding efficacious evidence-based interventions, there have been significant attempts to culturally adapt, implement, and disseminate parent training interventions broadly, especially across ethnic and cultural groups. We sought to examine the extent to which researchers and developers of evidence-based parent training programs have used cultural adaptation models, tested implementation strategies, and evaluated implementation outcomes when integrating the interventions into routine care by conducting a systematic review of the literature for four evidence-based parent training interventions: Parent-Child Interaction Therapy (PCIT), The Incredible Years (IY), Parent Management Training-Oregon Model (PMTO^TM), and the Positive Parenting Program (Triple P). A total of 610 articles across the four programs were identified. Of those, only eight documented a rigorous cultural adaptation process, and only two sought to test the effectiveness of implementation strategies by using rigorous research designs. Our findings suggest that there is much work to be done to move parent-training intervention research towards a more rigorous examination of cultural adaptation and implementation practices.     

Designing and implementing an evaluation of a national work support program

Welfare reforms in the 1990s have shifted governments around the world towards financial assistance conditional on work. While large-scale rigorous research on welfare-to-work programs has demonstrated effectiveness towards employment in other countries, no such micro-level evaluation of a policy has ever been conducted in Singapore. This article describes the process of developing a large experimental evaluation of the Work Support Program, which the Ministry of Community Development, Youth and Sports started in 2006. The lessons learned from planning and implementing the research can be helpful to future researchers in negotiating long-term rigorous evaluations in an environment where collaborators lack sufficient research knowledge. Insights include ways to focus on the essentials, find alternative experimental designs, collaborate effectively, and adapt instruments across cultures.     

Policy Legitimacy,Rhetorical Politics,and the Evaluation of City‐Street Video Surveillance Monitoring Programs in Canada

This article examines the evaluation and legitimization of city‐street video surveillance in Ontario, Canada. We demonstrate how monitoring programs are promoted, designed, and justified based on rhetorical claims about their efficacy. Although program evaluation data do not commonly corroborate or support program objectives, we show how the rhetorical politics of video surveillance evaluation research finds legitimacy by adhering to the Ontario privacy commissioner's best practices guidelines—guidelines that are formulated to minimize rhetoric in system design. The findings raise questions about how city‐street video surveillance remains a viable policy option given the lack of evidence pertaining to its usefulness.     

Evaluation plan of the 6for6 research skills program for rural and remote physicians

Overwhelming issues and barriers often prevent rural and remote physicians (RRPs) from pursuing the many socially accountable research questions they encounter on a daily basis. Although research training programs can empower RRPs to rise to these challenges, there is a lack of evidence on how they should be developed and refined. At Memorial University, a faculty development program (FDP) called 6for6 has been helping RRPs surmount their research quagmires and engage in scholarship since 2014. After an initial three-year (2014-17) pilot, we prepared a detailed plan to evaluate the 6for6 research FDP for RRPs and inform future years of delivery. Using a modified Delphi method and participatory action model a group of program team members, stakeholders and evaluation experts developed an evaluation plan including a logic model and an evaluation matrix addressing five key themes. To our knowledge, this is the first evaluation plan for a research-focused FDP targeting RRPs. While this plan was developed specifically for the 6for6 FDP, our approach to its development may be useful to any institution interested in evaluating an FDP with limited resources.