An approach to design long-term monitoring and evaluation frameworks in multi-actor systems--a case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors. This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring. An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult. From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

Process evaluation of an environmental walking and healthy eating pilot in small rural worksites

Small Steps are Easier Together (SS) was a pilot environmental intervention in small rural worksites in Upstate New York in collaboration with Extension educators. Worksite leaders teamed with co-workers to select and implement environmental changes to increase walking steps over individual baseline and to choose healthy eating options over 10 weeks. Participants were 226 primarily white, women employees in 5 sites. A mixed methods process evaluation, conducted to identify determinants of intervention effectiveness and to explain differences in outcomes across worksites, included surveys, self-reports of walking and eating, interviews, focus groups, and an intervention log. The evaluation assessed reach, characteristics of recruited participants, dose delivered, dose received, and context and compared sites on walking and eating outcomes. Emergent elements of participant-reported dose received included: active leadership, visible environmental changes, critical mass of participants, public display of accomplishments, accountability to co-workers, and group decision making. Participants at sites with high reach and dose were significantly more likely than sites with low reach and dose to achieve intervention goals. Although this small pilot needs replication, these findings describe how these evaluation methods can be applied and analyzed in an environmental intervention and provide information on trends in the data.     

Correlates of perceived effectiveness of the Safe Schools/Healthy Students Initiative

A three-level growth-curve model was applied to estimate perceived impact growth trajectories, using multi-year data from project and school surveys on outcome and program implementation collected from 59 sites and approximately 1165 participating schools in the Safe Schools and Healthy Students Initiative. Primary interest is to determine whether and how project-level and school-level correlates affect schools’ perceptions of the Initiative's effectiveness over time when the effects of the pre-grant environmental conditions, grant operations, and near-term outcomes are considered. Coordination and service integration, comprehensive programs and activities for early childhood development, and change in school involvement were found to be significant predictors of school-perceived overall impact when the effect of poverty was considered. Partnership functioning, perceived importance of school resources, and school involvement were found to be significant predictors of school-perceived impact on substance use prevention when the effect of poverty was considered.     

Risk Factors for Homelessness Among Lesbian, Gay, and Bisexual Youths: A Developmental Milestone Approach

Lesbian, gay, and bisexual (LGB) youths are over-represented in the homeless population. To examine why some LGB youths become homeless, this report compares homeless and non-homeless LGB youths. Of the 156 LGB youths, 48% reported ever being homeless (i.e., running away or being evicted from home). Results indicate that sexual orientation awareness and the initiation of sexual behavior occurred earlier in homeless than in non-homeless LGB youths and predated the first homeless episode. Substance use was more frequent and first occurred at an earlier age in homeless as compared to non-homeless LGB youths; however, substance use occurred subsequent to first homelessness. Childhood sexual abuse was associated with homelessness; and, early sexual orientation development was related to homelessness among youths without a history of sexual abuse. Findings suggest that interventions should help youths cope with their unfolding sexual orientation and work to prevent or address the consequences of sexual abuse.     

Benefits and costs associated with mutual-help community-based recovery homes: The Oxford House model

We used data from a randomized controlled study of Oxford House (OH), a self-run, self-supporting recovery home, to conduct a cost-benefit analysis of the program. Following substance abuse treatment, individuals that were assigned to an OH condition (n = 68) were compared to individuals assigned to a usual care condition (n = 61). Economic cost measures were derived from length of stay at an Oxford House residence, and derived from self-reported measures of inpatient and outpatient treatment utilization. Economic benefit measures were derived from self-reported information on monthly income, days participating in illegal activities, binary responses of alcohol and drug use, and incarceration. Results suggest that OH compared quite favorably to usual care: the net benefit of an OH stay was estimated to be roughly $29,000 per person on average. Bootstrapped standard errors suggested that the net benefit was statistically significant. Costs were incrementally higher under OH, but the benefits in terms of reduced illegal activity, incarceration and substance use substantially outweighed the costs. The positive net benefit for Oxford House is primarily driven by a large difference in illegal activity between OH and usual care participants. Using sensitivity analyses, under more conservative assumptions we still arrived at a net benefit favorable to OH of $17,830 per person.     

Predicting engagement in a transition to parenthood program for couples

Fostering participant engagement is a challenging but essential component of effective prevention programs. To better understand which factors influence engagement, this study examines several predictors of couple engagement in Family Foundations (FF), a preventive intervention for first-time parents shown to enhance parent mental health, couple relations, parenting quality, and child adjustment through age 3 years. FF consists of a series of classes delivered through childbirth education departments at local hospitals. Baseline data on socio-demographics, parent mental health, and couple relationship quality were examined as predictors of participants’ level of engagement in FF (n = 89 couples, 178 individuals). Sociodemographic variables such as parent gender, socioeconomic status, and age predicted program engagement to a limited extent. However, findings indicated that marital status was the best predictor of engagement. Discussion focuses on how findings can inform the development of practices that promote engagement, such as the use of targeted outreach efforts for individuals most at risk of disengagement.     

Facilitating Mental Health Service Use for Caregivers: Referral Strategies among Child Welfare Caseworkers

Unmet needs for mental health care are common among caregivers involved in the child welfare system. Although child welfare caseworkers are well positioned to identify service needs and refer caregivers to treatment, little is known about the types of referral strategies used in practice, or their effectiveness for promoting mental health service use. The current study examined child welfare caseworkers' use of different referral strategies and the extent to which these strategies are associated with caregivers' receipt of mental health services within a national sample of child welfare cases. Analyses of the second cohort of families from the National Survey of Child and Adolescent Well-Being suggest that child welfare workers more often use informational strategies for referring caregivers, including suggesting treatment or providing information about treatment options. However, social referral strategies such as providing caregivers with direct assistance in completing applications and making and attending appointments were associated with a greater likelihood of caregivers receiving mental health services. Findings support evidence from other service contexts that service use is facilitated by caseworkers' direct support for arranging services. Implications for research and for child welfare managers and administrators are discussed.     

Levels of interventions for MFTs working with family businesses

Family businesses (FBs) are a significant population in the world and therefore part of most practicing marriage and family therapists (MFTs) clientele; however, little is mentioned about FBs in the training of MFTs. This article offers some guidance to practicing MFTs who service this population, as well as MFTs who wish to expand their practice into a focused consultation practices with FB systems. This article uses Doherty's Levels of Family Involvement Model as a road map for MFTs to organize the vast amount of literature on FB systems as well as the many ways in which MFTs might serve FBs. We also offer suggestions for the necessary skills, experiences, and levels of engagement required at each level of intervention provided by MFTs.     

Changing attitudes about self-injury prevention management: lessons learned

In the behavioral health environment, nurses often use continuous staff monitoring and, at times, physical restraints, to manage the severity of patients' self-injury. Both options put staff in control, are the most restrictive in nature, and can be financially draining on the hospital's budget. This can result in negative reactions by both patients and staff. It is important to develop a program that will empower patients to control their behavior and allow staff to be aware of their perceptions and attitudes toward patients who self-injure. This article describes the leadership initiative that drove the development, training, and implementation of a self-injury prevention project and the lessons learned by staff.