The importance of systems thinking,context, and inclusion when studying the feasibility of expanding the FoodShare (SNAP) outreach program to rural communities of Northern Wisconsin

Evaluators often neglect to question whether a program has been appropriately designed for a new context prior to a feasibility study’s commencement. In this paper, we document the results of a case study that closely examined context when determining the feasibility of implementing a FoodShare outreach program in rural northern Wisconsin. Using community-based participatory mixed methods, we examine how stakeholder engagement led to both study refinement and a more comprehensive understanding of community food security based on contextual factors and systems thinking. Lessons learned when conducting feasibility studies from this perspective are provided to improve evaluation practice.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

The opportunities and challenges of multi-site evaluations: Lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.     

Monitoring and evaluating transition and sustainability of donor-funded programs: Reflections on the Avahan experience

PurposeIn low and middle-income countries, programs funded and implemented by international donors frequently transition to local funding and management, yet such processes are rarely evaluated. We reflect upon experience evaluating the transition of a large scale HIV/AIDS prevention program in India, known as Avahan, in order to draw lessons about transition evaluation approaches and implementation challenges.ResultsIn terms of conceptualizing the transition theory, the evaluation team identified tensions between the idea of institutionalizing key features of the Avahan program, and ensuring program flexibility to promote sustainability. The transition was planned in three rounds allowing for adaptations to transition intervention and program design during the transition period. The assessment team found it important to track these changes in order to understand which strategies and contextual features supported transition. A mixed methods evaluation was employed, combining semi-structured surveys of transitioning entities (conducted pre and post transition), with longitudinal case studies. Qualitative data helped explain quantitative findings. Measures of transition readiness appeared robust, but we were uncertain of the robustness of institutionalization measures. Finally, challenges to the implementation of such an evaluation are discussed.ConclusionsGiven the scarceness of transition evaluations, the lessons from this evaluation may have widespread relevance.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

The unique character of involvement in multi-site evaluation settings

As the number of large federal programs increases, so, too, does the need for a more complete understanding of how to conduct evaluations of such complex programs. The research literature has documented the benefits of stakeholder participation in smaller-scale program evaluations. However, given the scope and diversity of projects in multi-site program evaluations, traditional notions of participatory evaluation do not apply. The purpose of this research is to determine the ways in which stakeholders are involved in large-scale, multi-site STEM evaluations. This article describes the findings from a survey of 313 program leaders and evaluators and from follow-up interviews with 12 of these individuals.Findings from this study indicate that attendance at meetings and conferences, planning discussions within the project related to use of the program evaluation, and participation in data collection should be added to the list of activities that foster feelings of evaluation involvement among stakeholders. In addition, perceptions of involvement may vary according to breadth or depth of evaluation activities, but not always both. Overall, this study suggests that despite the contextual challenges of large, multi-site evaluations, it is feasible to build feelings of involvement among stakeholders.     

Implementation and process evaluation of a school-based drug abuse prevention program: Project Towards No Drug Abuse

The present study provides an implementation, process, and immediate outcomes evaluation of the classroom component of Project Towards No Drug Abuse (TND). This project involves development and evaluation of a school-based drug abuse prevention curriculum for continuation high school youth, who are at relatively high risk for drug abuse. Three randomized conditions were evaluated: a standard care, classroom only, and classroom plus school-as-community. The latter condition was an enhanced school-based condition which involved outside-of-classroom meetings and activities. Implementation was high in both program conditions even though this was a higher risk context. Process evaluation data were favorable and did not vary between the two program conditions. Immediate outcomes data (knowledge) was higher in the two program conditions than in the standard care condition. Regarding the classroom program, addition of extra-classroom activities does not appear to alter the quality of delivery of the program.     

Time matters: the potential and pitfalls of using mixed methods approaches in longitudinal program evaluation

While social programs are often assessed using short-term impact studies, longitudinal designs allow evaluators to capture change over time, identify longer-term outcomes, adapt instruments, and better understand participants in transition. A mixed methods design can be critical in understanding these dynamics; yet there is a lack of literature exploring the practical considerations of planning and conducting qualitative and quantitative data collection and analysis within longitudinal studies. This paper examines two different mixed methods frameworks used in a 5-year evaluation of three youth entrepreneurship programs in East Africa. We show how the evaluation team dealt with unique challenges across methods and over time, and how the design ultimately facilitated a richer understanding of program impacts and processes. Considerations for conducting this type of study are explored, related to the impact of longevity on analysis and research staff. Successfully using a longitudinal mixed methods approach requires researchers to be strategic and reflexive, and work in close collaboration.     

Mixed‐Gender Groups,Dating, and Romantic Relationships in Early Adolescence

This study examined dating‐stage and developmental‐contextual models of romantic relationships during early adolescence. Same‐gender friendships, affiliation with mixed‐gender groups, dating, and romantic relationships were investigated in a sample of 1,284 young adolescents of diverse ethnocultural backgrounds. Data were collected cross‐sectionally in Grades 5 through 8, as well as longitudinally in the fall and spring of an academic year. Consistent with a stage model, affiliation with mixed‐gender groups and dating were qualitatively distinct activities that were sequentially organized and facilitated the progression from same‐gender friendships to dyadic romantic relationships. The results also provide insights on how the developmental context may alter stage pathways: Dating activities were incorporated with mixed‐gender affiliations, group‐based romantic stages showed more stability than other stages, and the ethnocultural context influenced romantic timing. Finally, results indicated that participation in romantic activities heightened adolescents' future interest in having a romantic relationship.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Urbanization,Proximity, and the Intra-State Context of Women's Representation

ABSTRACT

The impact of context on women candidates has been vigorously studied using comparisons of state-level data and found to be substantial. Many of the variables used in this research were measured at the state level even though their influence is theorized at the district level and observed to vary within states. This essay tests these intra-state contextual influences using regression analysis of state House of Representatives district-level contextual data in four states from 1982-1990. The findings confirm contradict, and provide mixed results for particular state-level results.     

Community Context,Land Use,and First Birth

This article examines the influence of community context and land use on the monthly odds of first birth in a society in the midst of dramatic fertility transition. The theoretical framework guiding our work predicts that proximity to nonfamily services should delay first births by creating opportunities for competing nonfamily activities and spreading new ideas that change expectations about family life. On the other hand, living in agricultural settings that provide opportunities for higher returns to child labor should speed first births. We use a longitudinal, multilevel, mixed‐method data from the Nepalese Himalayas to test these predictions. The empirical results reveal that nonfamily services during childhood and during early adulthood both have important independent influences on the odds of first birth. Also, as predicted, a high density of agricultural land use affects the odds of first births in the opposite direction, speeding first births. This clear pattern of contrasting effects provides important new evidence of the contextual dynamics that produce watershed changes in postmarital birth timing.     

Fathering Involvement to Engagement: A Phenomenological Qualitative Roadmap

This phenomenological research examines the narratives of fathers and mothers about the lived experience of “fathering” to find themes to form dimensions of measuring father engagement. Qualitative responses from a mixed method cross-sectional sample of 191 father and mother participants were analyzed for phenomenological themes related to the lived experience of the fathering experience. Mothers (n?=?24) and fathers (n?=?34) provided narratives related to fathering roles and expectations, including enjoyment, fulfillment and gratitude, a relational and environmental context, the value of fathering, fathering involvement and activities, and emotional bonds and attachment. Themes from these narratives included both cognitive and affective dimensions of father involvement experience. The essence of this experience presents the intersection between the value of involvement within a contextual environment, attachment and the fulfillment and joy of the experience of parenting. Measures of attachment as well as fulfillment in parenting are necessary for future direction of research evaluation.     

Increasing the effectiveness of time-use survey with qualitative methods: The analysis of time–space interaction

Time-use surveys have been rich data sources in many countries for a long time. Turkey was among the countries that realized the potential of time-use surveys quite late and completed the first national survey in 2006. Despite its importance for a wide range of issues and applications, the first survey has flaws in design, which reduce its effectiveness and reliability. This is mostly due to disregarding cultural factors while tracking the methodology of European examples. This study aims to propose more appropriate methods of gathering time-use data in the Turkish context through a field survey in Ankara, the capital city. A mixed methodology that combines quantitative and qualitative methods effectively was applied and used to enrich data. The influence of space use was stressed and leisure activities were utilized to exemplify the use and benefits of mixed methods.     

Dealing with context in logic model development: Reflections from a realist evaluation of a community health worker programme in Nigeria

Logic models (LMs) have been used in programme evaluation for over four decades. Current debate questions the ability of logic modelling techniques to incorporate contextual factors into logic models. We share experience of developing a logic model within an ongoing realist evaluation which assesses the extent to which, and under what circumstances a community health workers (CHW) programme promotes access to maternity services in Nigeria. The article contributes to logic modelling debate by: i) reflecting on how other scholars captured context during LM development in theory-driven evaluations; and ii) explaining how we explored context during logic model development for realist evaluation of the CHW programme in Nigeria. Data collection methods that informed our logic model development included documents review, email discussions and teleconferences with programme stakeholders and a technical workshop with researchers to clarify programme goals and untangle relationships among programme elements. One of the most important findings is that, rather than being an end in itself, logic model development is an essential step for identifying initial hypotheses for tentative relevant contexts, mechanisms and outcomes (CMOs) and CMO configurations of how programmes produce change. The logic model also informed development of a methodology handbook that is guiding verification and consolidation of underlying programme theories.     

Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program

In 2005, state fish and wildlife agency wanted to examine how one of its conservation education programs was providing science-based understanding and outdoor experiences by evaluating students’ knowledge, skills, attitudes and intended behavioral outcomes related to fish, fishing and aquatic habitats in Montana. A key factor in this study was the acceptance by program stakeholders to conduct the evaluation using a utilization-focused evaluation approach to promote usability and accuracy of evaluation results. Using a quasi-experimental non-equivalent group design, more than 2000 students in participating classrooms throughout Montana received a pre-survey, post-survey and an extended post-survey; 114 teachers participated in an Internet survey and 16 program instructors took part in a structured open-ended telephone interview. The participatory approach and mixed methods enhanced abilities to interpret results of student surveys in particular. The user-focused approach was discovered to be personal and situational, allowed the facilitation of the evaluation process with consideration for increased application of evaluation findings and implementation of recommendations from beginning to end. Further development of evaluation pathways is needed to more effectively evaluate outcomes and implement practical and transferable measures to determine if environmental education activities produce desired participant outcomes.     

An evaluation of substance abuse aftercare program for homeless women with children using confounding variable-control design

An outcome evaluation of a substance abuse aftercare program for homeless women with children was conducted using confounding variable-control evaluation design. The confounding variables are chosen from pre-treatment and other contextual variables of the clients that are known to have significant influence on the program outcome, but those that could not have been influenced a priori by the client involvement in in-treatment program activity at Transition House (TH). The latter is the independent variable of this evaluation design. The pre-treatment variables are measured by severity of alcohol and other drug (AOD) problems of the clients, their mental health status, age, and their job status before enrollment in the program. The contextual confounding variables are composed of family and social support available to the clients before and during recovery. While applying multiple regression analysis, we were able to explain 50.8% of the total variance in program outcome by four pre-treatment variables. By adding two contextual variables of family and social support, the total variance in program outcome explained is increased to 64.1%. Finally, by adding the degree of client involvement in in-treatment program activity, we were able to augment the total variance of the program outcome to 69.7%. By estimating the changed variance of program outcome by the in-treatment program activity during the final step, controlling for all other variables previously entered, we were able to establish that client involvement in in-treatment program had unique and positive impact on the program outcome distinct from those explained by the confounding variables. The additional variance uniquely added by in-treatment program activity is 5.6% (p < .001). It has been determined that the degree of client involvement in in-treatment program had positive and systematic impact on the program outcome.     

Rehabilitation Contexts: A Holistic Approach

Psychosocial rehabilitation involves a gamut of activities that need to be personalized to suit an individual's needs. The context in which rehabilitation occurs greatly influences the range and availability of rehabilitation services and their utilization. This study attempts to highlight the contextual factors, with specific reference to globalization and development, the scope of rehabilitation, and the continuum of rehabilitation services that have a bearing on an individual's recovery. The relevance and effectiveness of the bio-psycho-social-spiritual approach in making services more pertinent and applicable are discussed. Implications for program development and policy interventions are suggested. It is hoped that this study will draw attention to the need for openness to a broad range of services even while using specific interventions.