Experiments in evaluation capacity building: Enhancing brain disorders research impact in Ontario

This paper is the introductory paper on a forum on evaluation capacity building for enhancing impacts of research on brain disorders. It describes challenges and opportunities of building evaluation capacity among community-based organizations in Ontario involved in enhancing brain health and supporting people living with a brain disorder. Using an example of a capacity building program called the “Evaluation Support Program”, which is run by the Ontario Brain Institute, this forum discusses multiple themes including evaluation capacity building, evaluation culture and evaluation methodologies appropriate for evaluating complex community interventions. The goal of the Evaluation Support Program is to help community-based organizations build the capacity to demonstrate the value that they offer in order to improve, sustain, and spread their programs and activities. One of the features of this forum is that perspectives on the Evaluation Support Program are provided by multiple stakeholders, including the community-based organizations, evaluation team members involved in capacity building, thought leaders in the fields of evaluation capacity building and evaluation culture, and the funders.     

Process evaluation of the ‘Singapore Physical Activity and Nutrition Study’

IntroductionThe Singapore Physical Activity and Nutrition Study (SPANS) aimed to improve the physical activity (PA) and nutrition behaviours of Singaporean women aged 50 years and over. The SPANS program consisted of PA classes, nutrition workshops, telephone dietary counselling, health booklets, a health calendar and program ambassadors. This study aimed to assess and understand the implementation of the program strategies and gain insight into process evaluation components to inform future programs.MethodsThe evaluation was guided by a process evaluation framework and collected data via questionnaires (n = 209), program ambassador documentation and exit interviews with program completers (n = 13) and non-completers (n = 12).ResultsIn total, 295 participants completed the program (response rate = 84 %). Participants reported high levels of satisfaction with the overall program (99.5 %) and program activities (96.7 %), and also rated program ambassadors highly. Participation rates were highest for telephone dietary counselling sessions. The main reason for not attending program activities was having a ‘busy schedule’ (n = 158). Participants cited a need for improved recreational centre facilities and increased flexibility around program delivery.ConclusionsThe process evaluation showed that the program strategies were implemented as planned and were deemed suitable for supporting behaviour change among Singaporean women aged 50 years and over. The program reached and involved the majority of participants throughout the six months. The combination of practical educational resources and supportive program ambassadors were key strategies that facilitated positive PA and dietary behaviours. However, there needs to be some flexibility in the delivery of programs. The findings of this research may inform other programs in the region.     

Looking Inside the “Black Box”

Abstract

The continued devolution of social welfare systems and services in the U.S. resultsin high stakes program evaluations in the field of family support and early intervention. Programs are expected to utilize evidence-based interventions and to demonstrate effectiveness. A look at implementation helps to differentiate between theories that do not work and programs that are not effective. Methods for identifying program implementation are needed. In a 17-site program evaluation, the author and her colleagues developed a methodology for measuring implementation and demonstrated the effects of differential implementation in understanding program outcomes.     

From Successful Family Planning to the Lowest of Low Fertility Levels: Taiwan's Dilemma

Since the implementation of family planning in the 1960s, Taiwan's fertility rate has rapidly decreased. This was praised as a family planning achievement. However, in the 21st century Taiwan has become one of the lowest of low fertility countries like European countries [ Kohler, H. P., Billari, F. C., & Ortega, J. A. (2002) . Population and Development Review , 28 (4), 641–80]. The government has begun to worry that these extremely low birthrates will result in rapid population aging and bring about other negative socio-economic effects. Thus, in its Mega Warmth Social Welfare Program (MWSWP) of 2006, the Taiwanese government targeted the issue of low birthrates. Based on secondary data analysis, we found that the reasons for rapid decrease in Taiwanese birthrates are: (i) a declining marriage rate; (ii) later marriage; (iii) changing attitudes towards child bearing; (iv) the burdens of child care; and (v) an increase in female labour participation rates. The MWSWP includes maternity leave benefits, parental leave benefits, a childcare subsidy system and early children education and care (ECEC). First, this article presents a chronological understanding of the demography in Taiwan. Second, we attempt to evaluate the reasons contributing to the low birthrates. The policy response to this is discussed next. Finally, the article provides a careful conclusion: that the extent to which these policies can significantly stop the population from declining requires further observation.     

Bridging the Implementation Gap in Poor Areas: A Study of How Municipal Socio‐economic Characteristics Impact Intergovernmental Policy Implementation

This article examines the relationship between local socio‐economic characteristics and implementation performance by analyzing the performance of Brazilian municipalities implementing the Bolsa Família Program, the largest conditional cash transfer programme in the world. Counterintuitively, poor municipalities were found to outperform wealthy ones. Analysis of the interactions between local socio‐economic characteristics and municipal administrative capacity, willingness and intergovernmental relations help to explain why this was the case.     

Class Size and Educational Policy: Who Benefits from Smaller Classes?

ABSTRACT

The impact of class size on student achievement remains an open question despite hundreds of empirical studies and the perception among parents, teachers, and policymakers that larger classes are a significant detriment to student development. This study sheds new light on this ambiguity by utilizing nonparametric tests for stochastic dominance to analyze unconditional and conditional test score distributions across students facing different class sizes. Analyzing the conditional distributions of test scores (purged of observables using class-size specific returns), we find that there is little causal effect of marginal reductions in class size on test scores within the range of 20 or more students. However, reductions in class size from above 20 students to below 20 students, as well as marginal reductions in classes with fewer than 20 students, increase test scores for students below the median, but decrease test scores above the median. This nonuniform impact of class size suggests that compensatory school policies, whereby lower-performing students are placed in smaller classes and higher-performing students are placed in larger classes, improves the academic achievement of not just the lower-performing students but also the higher-performing students.     

当前西亚北非地区局势透视

西亚北非动荡进入第三年后，局势变得更加复杂多变。地区大国埃及的国内局势、温和与极端伊斯兰之间的“跷跷板”效应、美国在中东地区的战略政策，以及棘手的伊朗核问题等都是影响西亚北非地区局势的重要因素。鉴于此，西亚北非在可预见的时期内，依然会是持续动荡的局面。     

Using client exit interviews to illuminate outcomes in program logic models: a case example

This paper describes how client exit interviews can assist human service administrators and workers to better understand the outcomes their programs are designed to accomplish. Specifically, the qualitative component of a demonstration family literacy program evaluation is used to illustrate how client input can be used to fine-tune the outcomes component of a program's logic model. An analysis of semi-structured exit interviews with 35 clients, who were randomly selected from all 89 served in the first year of the program, resulted in revision to the program's original logic model, creating explicit ‘testable’ pathways to achieving intended outcomes.     

Cultivating knowledge of resiliency and reintegration among military youth through a national youth leadership program

Youth in military families are frequently challenged by the adjustment demands associated with the deployment and reintegration of a parent. A positive youth development approach was undertaken by the Boys and Girls Clubs of America to develop and implement a Military Teen Ambassadors (MTA) training for youth in military families that would facilitate knowledge of resiliency and reintegration and foster leadership skills to build assets for themselves as well as their peers within their local communities. To determine if MTA was functioning as intended and to refine future programming, this preliminary formative study assessed perceived participant learning outcomes associated with MTA on variables pertaining to knowledge acquisition, perceived skill acquisition, and community needs awareness. Data were collected prior to the training, immediately following the training, and 6 months after the training. Repeated measures analysis indicated significant mean increases over time in knowledge and awareness of resiliency and reintegration; perceived leadership skills; and community awareness. Qualitative findings provided triangulation in the aforementioned areas. These findings strengthen the body of knowledge on resiliency by demonstrating that the 7 Cs model may be an effective strategy to incorporate into leadership development programs seeking to build knowledge of resiliency among military youth. Study limitations, lessons learned, and recommendations for further research are delineated.     

Implementing condom distribution programs in the United States: Qualitative insights from program planners

With the growing number of sexually transmitted infections (STIs) among young people (15–24) in the US, condom distribution programs (CDP) remain an integral part of prevention strategies. The objective of the study was to gather qualitative insights from CDP planners to inform effective CDP implementation. Ten semi-structured interviews with program planners were conducted via telephone (response rate of 58.8%). Condom distribution channels included site-based distribution (n = 6), web-based distribution (n = 4), and distribution via Uber (n = 1). Site-based distribution programs had distribution networks ranging from 100 to 3500 sites in a given jurisdiction. Planners of site-based CDPs experienced challenges with ensuring quality control at sites and supplying condoms to sites. Web-based CDPs allow individuals to order free condoms online. These CDP planners reported growing demand for this discreet service, particularly among young people. Web-based CDPs leveraged e-mail surveys to collect data on sexual behavior and behavior change, yielding response rates as high as 63%. All CDPs emphasized the importance of supplying a variety of products. Total supplies distributed (male condoms, internal condoms, lubricant sachets) ranged from 16,000 to 45 million per year. Novel channels of distribution should be explored to ensure reach to adolescents and young adults.