Interpreting interviews

ABSTRACT

Realist evaluation (RE) is increasingly adopted to assess the effectiveness of social programs. RE goes beyond measurement of outcomes, providing insight into how and why programs work by focusing on how participants experience and respond to programs within certain contexts. A RE was conducted on Healthy Start-Départ Santé (HSDS), an early childhood development program in Canada, created to address the prevalence of overweight and obese children. The first step of the RE was to create an initial program theory (IPT) to identify key contexts and mechanisms relating to the success of the program. Realist methodology can be challenging, time consuming, and resource intensive and there are few practical examples of how to conduct a RE. This article aims to add to the realist methodological literature by describing the construction of the IPT, reflecting on the process used, and providing a starting point for future realist practitioners.     

Monitoring and evaluating transition and sustainability of donor-funded programs: Reflections on the Avahan experience

PurposeIn low and middle-income countries, programs funded and implemented by international donors frequently transition to local funding and management, yet such processes are rarely evaluated. We reflect upon experience evaluating the transition of a large scale HIV/AIDS prevention program in India, known as Avahan, in order to draw lessons about transition evaluation approaches and implementation challenges.ResultsIn terms of conceptualizing the transition theory, the evaluation team identified tensions between the idea of institutionalizing key features of the Avahan program, and ensuring program flexibility to promote sustainability. The transition was planned in three rounds allowing for adaptations to transition intervention and program design during the transition period. The assessment team found it important to track these changes in order to understand which strategies and contextual features supported transition. A mixed methods evaluation was employed, combining semi-structured surveys of transitioning entities (conducted pre and post transition), with longitudinal case studies. Qualitative data helped explain quantitative findings. Measures of transition readiness appeared robust, but we were uncertain of the robustness of institutionalization measures. Finally, challenges to the implementation of such an evaluation are discussed.ConclusionsGiven the scarceness of transition evaluations, the lessons from this evaluation may have widespread relevance.     

Towards Program Evaluation for Practitioner Learning: Human Service Practitioners' Perceptions of Evaluation

Abstract

While human service practice improvement is acknowledged as an important outcome of program evaluation, much evaluation is narrowly focused on accountability requirements. This type of evaluation often has limited use and relevance to human service practitioners. In exploring forms of evaluation that do have practice relevance, this article presents the experiences and perceptions of human service practitioners subject to evaluations that set out to foster learning. Across the cases participants expressed concerns about data quality and credibility, the relevance of evaluation data to the program context, the invasiveness of the evaluation, management of the process (particularly limited consultation with practitioners), and the lack of effective dissemination. These cases illustrate the challenges for evaluation to be useful to practice, and to foster meaningful improvements to services.     

"It Has Made College Possible for Me": Feedback on the Impact of a University-Based Center for Students in Recovery

Objective: The authors aimed to gain information on (1) the challenges for recovering students on a university campus and (2) the most helpful components of a collegiate recovery program.

Participants: The 15 students in the study were all in recovery from substance abuse. They entered the university and also entered the campus recovery program either in fall 2002 or fall 2003.

Methods: Semistructured interviews were conducted with students multiple times during their first academic year.

Results: Participants identified several challenges on the university campus. Many believed they would have dropped out of school or relapsed without the support of the recovery program, and they described aspects of the program that were particularly important to them.

Conclusions: Support from a campus recovery program is essential for many recovering students. There are a variety of recovery program components that can foster the sense of community that was so important to the students in this study.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

The State of Our Art

Abstract

In the second part of this two-part paper, the author examines intergenerational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, which appeared in the previous issue, conceptual foundations for intergenerational programming were considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross-cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Adapting a Family-Based HIV Prevention Program for HIV-Infected Preadolescents and Their Families

Summary

This article describes a family-based HIV prevention and mental health promotion program specifically designed to meet the needs of perinatally-infected preadolescents and their families. This project represents one of the first attempts to involve perinatally HIV-infected youth in HIV prevention efforts while simultaneously addressing their mental health and health care needs. The program, entitled CHAMP+ (Collaborative HIV Prevention and Adolescent Mental Health Project-Plus), focuses on: (1) the impact of HIV on the family; (2) loss and stigma associated with HIV disease; (3) HIV knowledge and understanding of health and medication protocols; (4) family communication about puberty, sexuality and HIV; (5) social support and decision making related to disclosure; and (6) parental supervision and monitoring related to sexual possibility situations, sexual risk taking behavior and management of youth health and medication. Findings from a preliminary evaluation of CHAMP+ with six families are presented along with a discussion of challenges related to feasibility and implementation within a primary health care setting for perinatally infected youth.     

Process evaluation of the ‘Singapore Physical Activity and Nutrition Study’

IntroductionThe Singapore Physical Activity and Nutrition Study (SPANS) aimed to improve the physical activity (PA) and nutrition behaviours of Singaporean women aged 50 years and over. The SPANS program consisted of PA classes, nutrition workshops, telephone dietary counselling, health booklets, a health calendar and program ambassadors. This study aimed to assess and understand the implementation of the program strategies and gain insight into process evaluation components to inform future programs.MethodsThe evaluation was guided by a process evaluation framework and collected data via questionnaires (n = 209), program ambassador documentation and exit interviews with program completers (n = 13) and non-completers (n = 12).ResultsIn total, 295 participants completed the program (response rate = 84 %). Participants reported high levels of satisfaction with the overall program (99.5 %) and program activities (96.7 %), and also rated program ambassadors highly. Participation rates were highest for telephone dietary counselling sessions. The main reason for not attending program activities was having a ‘busy schedule’ (n = 158). Participants cited a need for improved recreational centre facilities and increased flexibility around program delivery.ConclusionsThe process evaluation showed that the program strategies were implemented as planned and were deemed suitable for supporting behaviour change among Singaporean women aged 50 years and over. The program reached and involved the majority of participants throughout the six months. The combination of practical educational resources and supportive program ambassadors were key strategies that facilitated positive PA and dietary behaviours. However, there needs to be some flexibility in the delivery of programs. The findings of this research may inform other programs in the region.     

Hindsight,foresight and insight: The challenges of longitudinal qualitative research

This paper describes a UK based longitudinal qualitative study following approximately 100 young people over 9 years. It provides an overview of the research design, methods of data collection and analysis as well as reflecting on the strengths and weaknesses of the strategies employed. We focus on the value of a devolved research design, complementary cross-section and temporal analytic approaches and the effects of being involved in a longitudinal study on researchers and participants. The paper aims to contribute to a growing understanding of the practical, ethical and epistemological challenges presented by longitudinal qualitative methods, and highlights those challenges that are specific to this method as well as those which are amplified by this kind of approach.     

Portfolio assessment in a collaborative program evaluation: the reliability and validity of a family literacy portfolio

Evaluations increasingly use open-ended assessments, such as portfolios, to gather information about program impact. Portfolios, however, present challenges in achieving acceptable levels of reliability and often lack evidence to support validity. This article describes the components of a portfolio assessment system developed in a collaborative evaluation of a family literacy program. Staff of a family literacy program and two evaluators collaborated to develop the assessment and the tools that frame the review of the portfolios. Rubrics, exemplar portfolios, portfolio logs, and lists of ‘look-fors’ were developed to guide raters in the scoring and to improve the reliability of scores and validity of decisions about the program. Inter-rater reliability for the portfolio assessment was 0.79 for holistic scores and 0.74 for analytic scores. With respect to validity, holistic and analytic scores correlated with criterion measures from 0.50 to 0.59, providing initial evidence of a common construct.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Explaining how unexploded ordnance clearance enhances livelihoods in the Lao PDR

BackgroundFollowing violent conflict, the continued presence of landmines and unexploded ordnance pose a barrier to rebuilding livelihoods. Mine action removes these explosive remnants of conflict to enable communities to safely return contaminated land to productive use. There is limited understanding, however, of how, why, in what context and in what respects mine action contributes to livelihoods. Yet, such information is required for effective resource allocation, checking underlying program assumptions, understanding benefits and potential harms.MethodsThe evaluation was undertaken in the Lao People's Democratic Republic. It used an interpretive case study design and applied the principles of realist evaluation. Program staff and local government authorities were interviewed (N = 37) and program beneficiaries. In total, 38 individual interviews with program beneficiaries were conducted and eighteen focus group interviews (9 with males, 9 with females), each with 6–9 participants.ResultsThe evaluation identified two main mechanisms through which the program ‘worked’: (1) communication pre- and post-clearance and (2) the delivery of the product (cleared land).ConclusionThe realist approach helped to refine the program theory, highlighted the role of self- and task-efficacy and community communication, assisted in identifying contextual factors that influence outcomes and suggested a revision of expected outcomes.     

Distance Education: The Role of the Site Advisory Committee

SUMMARY

This paper describes the role of the Site Advisory Committee (SAC) as a link between the off-campus sites and the host university. The role of the SAC is to provide advice, regional perspectives, and suggestions for distance education program enhancement. SAC activities, structure, and a program evaluation plan will be addressed.     

Evaluation of the RAINN National Sexual Assault Online Hotline

ABSTRACT

This article describes the evaluation of rainn National Sexual Assault Online Hotline, a new model for delivery of rape and sexual assault crisis services through a secure, confidential chat-based online hotline. Outcome data from 623 visitors and 54 volunteers for the first 11 months of the program are presented. The results indicate that the model is viable and useful for approximately 80 percent of visitors, and volunteers are satisfied with the program. Recommendations for program development and further research are presented.     

Teaching Note—A Call for Including Theories of Evaluation in Program Evaluation Courses Taught in Schools of Social Work

ABSTRACT

Many schools of social work offer courses on program evaluation. However, one component of program evaluation—theories of evaluation—may all too often be left out of the curriculum. This Teaching Note defines and describes evaluation theory and the benefits of including it in a program evaluation curriculum in schools of social work. Specific ideas for incorporating this content into MSW and doctoral courses in schools of social work are provided.     

Meeting the needs of adolescents living with HIV through home based care: Lessons learned from Tanzania

PurposeHome based care is central to HIV care and support throughout Africa. Most programs have not addressed the unique needs of adolescents living with HIV, who display lower retention in HIV care and treatment adherence. This study examined the experiences of adolescents living with HIV in Tanzania in order to identify ways to improve home based care to better meet their needs.MethodsWe conducted a qualitative study in Dar es Salaam and Tanga Region consisting of in-depth interviews with 14 adolescents living with HIV, 10 primary caregivers, and 12 providers assigned to their households. Interviews examined adolescents' experiences of growing up with HIV, perceptions of current home based care, and challenges in increasing acceptability of services.ResultsWe found the program did not align well with adolescents' expectations. Adolescents felt home based care to be more relevant to their caregivers and did not form independent relationships with providers. They expressed anxiety that participation might lead to inadvertent disclosure of their status and consequent discrimination. Both adolescents and their caregivers felt disappointment that no material or financial support was available despite widespread poverty, although they appreciated receiving psychosocial support and practical referrals. Providers demonstrated motivation to work with adolescents but acknowledged lacking requisite skills such as ways to communicate with young people and key messages to deliver.ConclusionsDespite challenges, we identified feasible adaptations to make home based care more adolescent-centered. These include actively engaging adolescents in program design, improving provider training in communication with adolescents, and proactively addressing unrealistic expectations. Finally, increasing referral links to a wider range of services could improve program effectiveness by integrating it into a broader development approach.     

The Implementation of a “Work First” Welfare-to-Work Program in a Changing Environment

ABSTRACT

Current welfare-to-work programs grapple with the issue of putting welfare recipients to work immediately or investing in long-term education and training. Because of the 1996 welfare reform time limits, “work first” approaches are predominating. This paper examines the innovative program developed by Goodwill Industries of Southern California, which meets the challenges of implementing a work first model with a low-skill Spanish-speaking population, while also providing services to overcome barriers to employment. This is accomplished in the context of a unique profit-generating business enterprise, which makes program participants both clients and employees.     

Less Pain,More Gain

SUMMARY

Applied social statistics is typically one of the most dreaded courses in a social work program. Technological advances in computer applications offer tools to lessen the pain and increase the gains associated with teaching and learning statistics. This paper reports on a modest research project resulting in a computer-based approach to teaching Applied Social Statistics. This approach makes full use of MicroCase, a statistical software package. Results of evaluative efforts which appraised the computer technology-enhanced statistics course are presented. Course format, classroom procedures, workbook and homework assignments, evaluation tools, and teaching strategies are also suggested.