Lessons learned about collaborative evaluation using the Capacity for Applying Project Evaluation (CAPE) framework with school and district leaders

Schools, districts, and state-level educational organizations are experiencing a great shift in the way they do the business of education. This shift focuses on accountability, specifically through the expectation of the effective utilization of evaluative-focused efforts to guide and support decisions about educational program implementation. In as much, education leaders need specific guidance and training on how to plan, implement, and use evaluation to critically examine district and school-level initiatives. One specific effort intended to address this need is through the Capacity for Applying Project Evaluation (CAPE) framework. The CAPE framework is composed of three crucial components: a collection of evaluation resources; a professional development model; and a conceptual framework that guides the work to support evaluation planning and implementation in schools and districts. School and district teams serve as active participants in the professional development and ultimately as formative evaluators of their own school or district-level programs by working collaboratively with evaluation experts. The CAPE framework involves the school and district staff in planning and implementing their evaluation. They are the ones deciding what evaluation questions to ask, which instruments to use, what data to collect, and how and to whom results should be reported. Initially this work is done through careful scaffolding by evaluation experts, where supports are slowly pulled away as the educators gain experience and confidence in their knowledge and skills as evaluators. Since CAPE engages all stakeholders in all stages of the evaluation, the philosophical intentions of these efforts to build capacity for formative evaluation strictly aligns with the collaborative evaluation approach.     

An approach to design long-term monitoring and evaluation frameworks in multi-actor systems--a case in water management

Learning-by-doing and adaptive management require careful monitoring and evaluation of the outcomes of environmental policies and programs under implementation. Selecting relevant indicators is difficult, especially when monitoring over a longer period of time. Further challenges arise when policies are developed as a collaborative effort among multiple actors. This paper discusses an approach to design frameworks for long-term monitoring and evaluation in multi-actor systems. It uses Dynamic Actor Network Analysis (DANA) as an actor-sensitive method to reconstruct program theories. This is combined with elements of assumption-based planning to identify critical assumptions and associated indicators to incorporate the dynamic aspects related to long-term monitoring. An application of this approach is described for a case of water management in the Netherlands. Here, mapping multiple perspectives and identifying critical assumptions helped to broaden the scope of monitoring in important ways. Identifying associated indicators and expectations on their development in response to policy implementation proved more difficult. From this case, it can be concluded that the approach is feasible, useful, but also demanding. However, with continuing trends of networked governance and adaptive management, additional efforts to reflect these trends in monitoring and evaluation, through this and similar approaches, are needed.     

Process evaluation of an environmental walking and healthy eating pilot in small rural worksites

Small Steps are Easier Together (SS) was a pilot environmental intervention in small rural worksites in Upstate New York in collaboration with Extension educators. Worksite leaders teamed with co-workers to select and implement environmental changes to increase walking steps over individual baseline and to choose healthy eating options over 10 weeks. Participants were 226 primarily white, women employees in 5 sites. A mixed methods process evaluation, conducted to identify determinants of intervention effectiveness and to explain differences in outcomes across worksites, included surveys, self-reports of walking and eating, interviews, focus groups, and an intervention log. The evaluation assessed reach, characteristics of recruited participants, dose delivered, dose received, and context and compared sites on walking and eating outcomes. Emergent elements of participant-reported dose received included: active leadership, visible environmental changes, critical mass of participants, public display of accomplishments, accountability to co-workers, and group decision making. Participants at sites with high reach and dose were significantly more likely than sites with low reach and dose to achieve intervention goals. Although this small pilot needs replication, these findings describe how these evaluation methods can be applied and analyzed in an environmental intervention and provide information on trends in the data.     

Correlates of perceived effectiveness of the Safe Schools/Healthy Students Initiative

A three-level growth-curve model was applied to estimate perceived impact growth trajectories, using multi-year data from project and school surveys on outcome and program implementation collected from 59 sites and approximately 1165 participating schools in the Safe Schools and Healthy Students Initiative. Primary interest is to determine whether and how project-level and school-level correlates affect schools’ perceptions of the Initiative's effectiveness over time when the effects of the pre-grant environmental conditions, grant operations, and near-term outcomes are considered. Coordination and service integration, comprehensive programs and activities for early childhood development, and change in school involvement were found to be significant predictors of school-perceived overall impact when the effect of poverty was considered. Partnership functioning, perceived importance of school resources, and school involvement were found to be significant predictors of school-perceived impact on substance use prevention when the effect of poverty was considered.     

Benefits and costs associated with mutual-help community-based recovery homes: The Oxford House model

We used data from a randomized controlled study of Oxford House (OH), a self-run, self-supporting recovery home, to conduct a cost-benefit analysis of the program. Following substance abuse treatment, individuals that were assigned to an OH condition (n = 68) were compared to individuals assigned to a usual care condition (n = 61). Economic cost measures were derived from length of stay at an Oxford House residence, and derived from self-reported measures of inpatient and outpatient treatment utilization. Economic benefit measures were derived from self-reported information on monthly income, days participating in illegal activities, binary responses of alcohol and drug use, and incarceration. Results suggest that OH compared quite favorably to usual care: the net benefit of an OH stay was estimated to be roughly $29,000 per person on average. Bootstrapped standard errors suggested that the net benefit was statistically significant. Costs were incrementally higher under OH, but the benefits in terms of reduced illegal activity, incarceration and substance use substantially outweighed the costs. The positive net benefit for Oxford House is primarily driven by a large difference in illegal activity between OH and usual care participants. Using sensitivity analyses, under more conservative assumptions we still arrived at a net benefit favorable to OH of $17,830 per person.     

Facilitating Mental Health Service Use for Caregivers: Referral Strategies among Child Welfare Caseworkers

Unmet needs for mental health care are common among caregivers involved in the child welfare system. Although child welfare caseworkers are well positioned to identify service needs and refer caregivers to treatment, little is known about the types of referral strategies used in practice, or their effectiveness for promoting mental health service use. The current study examined child welfare caseworkers' use of different referral strategies and the extent to which these strategies are associated with caregivers' receipt of mental health services within a national sample of child welfare cases. Analyses of the second cohort of families from the National Survey of Child and Adolescent Well-Being suggest that child welfare workers more often use informational strategies for referring caregivers, including suggesting treatment or providing information about treatment options. However, social referral strategies such as providing caregivers with direct assistance in completing applications and making and attending appointments were associated with a greater likelihood of caregivers receiving mental health services. Findings support evidence from other service contexts that service use is facilitated by caseworkers' direct support for arranging services. Implications for research and for child welfare managers and administrators are discussed.     

Levels of interventions for MFTs working with family businesses

Family businesses (FBs) are a significant population in the world and therefore part of most practicing marriage and family therapists (MFTs) clientele; however, little is mentioned about FBs in the training of MFTs. This article offers some guidance to practicing MFTs who service this population, as well as MFTs who wish to expand their practice into a focused consultation practices with FB systems. This article uses Doherty's Levels of Family Involvement Model as a road map for MFTs to organize the vast amount of literature on FB systems as well as the many ways in which MFTs might serve FBs. We also offer suggestions for the necessary skills, experiences, and levels of engagement required at each level of intervention provided by MFTs.     

Raising awareness of psychosocial factors in the occurrence of low back symptoms in developing countries

Although there is strong evidence that single physical, psychosocial and organisational risk factors are each independent predictors of low back symptoms (LBS), little is known about their combination/interaction, particularly in those working in developing countries. A total of 1294 Indonesian coal mining industry workers reported information about physical, psychosocial, and organisational factors using self-reported questionnaires and were placed into one of four combination exposure groups: 1) high physical (working with bent trunk; whole body vibration exposure; lifting) and high psychosocial (high effort; low reward; work stress), 2) high physical and low psychosocial, 3) low physical and high psychosocial, and 4) low physical and low psychosocial (as the reference group). The odds of LBS in the high physical and high psychosocial group was 5.15 times (95% confidence interval 2.69-9.86) that of the reference group. Psychosocial factors played a more important role than physical factors in increasing the likelihood of reporting LBS. Permanent and night shift workers were more likely to report LBS. These findings imply that efforts to reduce LBS in this workplace should address both psychosocial and physical factors but psychosocial factors should be the priority, with a special focus on permanent and night shift workers.