Using logic model and visualization to conduct portfolio evaluation

Portfolio evaluation is the evaluation of multiple projects with a common purpose. While logic models have been used in many ways to support evaluation, and data visualization has been used widely to present and communicate evaluation findings, adopting logic models for portfolio evaluation and using data visualization to share findings simultaneously is surprisingly limited in the literature. With the data from a sample portfolio of 209 projects which aims to improve the system of early care and education (ECE), this study illustrated how to use logic model and data visualization techniques to conduct a portfolio evaluation by answering two evaluation questions: “To what extent are the elements of a logic model (strategies, sub-strategies, activities, outcomes, and impacts) reflected in the sample portfolio?” and “Which dominant paths through the logic model were illuminated by the data visualization technique?” For the first question, the visualization technique illuminated several dominant strategies, sub-strategies, activities, and outcomes. For the second question, our visualization techniques made it convenient to identify critical paths through the logic model. Implications for both program evaluation and program planning were discussed.     

Urban community as resource: Evaluation of the mentors in Toledo schools program

We report findings from an evaluation of the Mentors in Toledo Schools program (MITS), which pairs adult community volunteers with elementary school students in need of reading support. Begun in 2012, MITS now operates in five elementary schools in Toledo Public Schools. A quasi-experimental study of 379 students (n = 128 mentored, n = 251 comparison) tested the effectiveness of MITS, using changes (fall to spring) in students’ scores on STAR Reading. Multilevel models demonstrate students who participated in MITS made statistically significant, and educationally meaningful, reading gains compared to students who did not participate. Further, an investigation of the importance of dosage revealed that number of mentoring sessions is related to students’ outcomes; students who participated in reading mentoring at least once per week showed the most educationally meaningful improvements. We also report qualitative feedback from adult program participants, including teachers, mentors, and site coordinators, to help contextualize the STAR Reading results. These findings speak to the opportunity for community volunteers to act as low-cost and effective supports for urban elementary school students who are at-risk for reading failure.     

Towards an assessment for organizational participatory research health partnerships: A systematic mixed studies review with framework synthesis

Within the health sciences, organizational participatory research (OPR) is defined as a blend of research and action, in which academic researchers partner with health organization members. OPR is based on a sound partnership between all stakeholders to improve organizational practices. However, little research on the evaluation of OPR health partnership exists. This systematic mixed studies review sought to produce a new theoretical model that structures the evaluation of the OPR processes and related outcomes of OPR health partnerships. Six bibliographic databases were searched together with grey literature sources for OPR health partnership evaluation questionnaires. Six questionnaires were included, from which a pool of 95 OPR health partnership evaluation items were derived. The included questionnaires were appraised for the quality of their origin, development and measurement properties. A framework synthesis was performed using an existing OPR framework by organizing questionnaire items in a matrix using a hybrid thematic analysis. This led to our proposed Organizational Participatory Research Evaluation Model (OPREM) that includes three axes, Trust, Collective Learning and Sustainability (with specific dimensions) and 95 items. This model provides information to help stakeholders comprehensively structure the evaluation of their partnerships and subsequent improvement; thus, potentially helping to improve health organization practices.     

基于协调发展度的重大工程团队免疫系统健康评价研究

为评价重大工程团队免疫系统的健康和协调发展状况,本文提出了一种基于协调发展度的评价模型。首先通过文献查询和深度访谈,构建能够体现重大工程团队免疫系统特征的健康评价指标体系;其次利用改进的DEMATEL确定指标权重,然后采用多层次模糊综合评价将相关指标定量化,最终得出重大工程团队免疫系统的健康指数和协调发展度;再次确立重大工程团队免疫系统健康等级和协调发展度划分标准,同时提出免疫系统健康的跟踪评价策略;最后,利用本文模型对某跨国输气管道工程团队的免疫系统进行实证研究。结果表明,协调发展度模型既能反映重大工程团队免疫系统的整体健康程度,又能衡量固有性免疫和适应性免疫的协调发展状况,同时对免疫系统健康的跟踪评价也取得了良好效果。     

Wilderness adventure therapy effects on the mental health of youth participants

Adventure therapy offers a prevention, early intervention, and treatment modality for people with behavioural, psychological, and psychosocial issues. It can appeal to youth-at-risk who are often less responsive to traditional psychotherapeutic interventions. This study evaluated Wilderness Adventure Therapy (WAT) outcomes based on participants’ pre-program, post-program, and follow-up responses to self-report questionnaires. The sample consisted of 36 adolescent out-patients with mixed mental health issues who completed a 10-week, manualised WAT intervention. The overall short-term standardised mean effect size was small, positive, and statistically significant (0.26), with moderate, statistically significant improvements in psychological resilience and social self-esteem. Total short-term effects were within age-based adventure therapy meta-analytic benchmark 90% confidence intervals, except for the change in suicidality which was lower than the comparable benchmark. The short-term changes were retained at the three-month follow-up, except for family functioning (significant reduction) and suicidality (significant improvement). For participants in clinical ranges pre-program, there was a large, statistically significant reduction in depressive symptomology, and large to very large, statistically significant improvements in behavioural and emotional functioning. These changes were retained at the three-month follow-up. These findings indicate that WAT is as effective as traditional psychotherapy techniques for clinically symptomatic people. Future research utilising a comparison or wait-list control group, multiple sources of data, and a larger sample, could help to qualify and extend these findings.     

Transforming a municipal school sports programme through a critical communicative methodology: The role of the of advisory committee

This research applies a communicative methodology (CM) to the transformation and improvement of the Municipal Comprehensive School Sports Programme in Segovia, Spain (MCSSP), using egalitarian dialogue, based on validity rather than power claims to achieve intersubjectivity and arrive at consensus between all of the Programme’s stakeholders through the intervention of an advisory committee (AC). The AC is a body comprising representatives of all stakeholder groups involved in the programme. During the 2013–2014 academic year the programme’s AC met four times, operating as a communicative focus group (CFG). The meetings focused on: (1) excluding dimensions (barriers preventing transformation) and transforming dimensions (ways of overcoming barriers), (2) the programme’s strengths, (3) the programme’s weaknesses and specific actions to remedy them, and (4) the resulting conclusions which were then incorporated into the subsequent programme contract signed between the University and the Segovia Local Authority for 2014–2018. The key conclusions were: (1) the recommendations of the AC widen the range of perspectives and help the research team to make key decisions and (2) the use of CM to fully evaluate the programme and to reach a consensus on how to improve it proved very valuable.     

Governmentality within Children's Technological Play: Findings from a Critical Discourse Analysis

In many countries today, digital technology and instant communication are embedded in children's everyday lives to the extent that their play frequently incorporates smartphones, the Internet and other technologies. In this paper, we explore the recent historical conditions within the New Zealand context that have increased the accessibility of these technologies and imbued them with particular meanings. We suggest that from a Foucauldian perspective, these technologies can be seen as a form of subtle disciplinary power using techniques of governmentality through which children's ways of thinking are shaped to benefit societal requirements of the current historical era.