Can a workbook work? Examining whether a practitioner evaluation toolkit can promote instrumental use

In large-scale, multi-site contexts, developing and disseminating practitioner-oriented evaluation toolkits are an increasingly common strategy for building evaluation capacity. Toolkits explain the evaluation process, present evaluation design choices, and offer step-by-step guidance to practitioners. To date, there has been limited research on whether such resources truly foster the successful design, implementation, and use of evaluation findings. In this paper, we describe a multi-site project in which we developed a practitioner evaluation toolkit and then studied the extent to which the toolkit and accompanying technical assistance was effective in promoting successful completion of local-level evaluations and fostering instrumental use of the findings (i.e., whether programs directly used their findings to improve practice, see Patton, 2008). Forensic nurse practitioners from six geographically dispersed service programs completed methodologically rigorous evaluations; furthermore, all six programs used the findings to create programmatic and community-level changes to improve local practice. Implications for evaluation capacity building are discussed.     

Recruitment and retention strategies for out-of-school-time programs

Many out-of-school-time (OST) practitioners seek ways to maximize enrollment, enhance frequency of participation, and ensure retention in OST programs, so that the multiple potential benefits of these programs are realized. Three critical areas of program quality are important underpinnings for success in recruitment and retention of youth in OST programs: a sense of safety and community; committed program staff; and challenging, age-appropriate, and fun activities. In addition, a review of over sixty OST evaluations reveals five promising recruitment strategies to boost enrollment: (1) matching the program to participants' needs (offering activities not readily available elsewhere or offering flexible schedules), (2) demonstrating the importance of participation to young people and their families, (3) reaching out directly to youth and their families in their homes and communities, (4) recruiting in peer circles, and (5) making a special effort to recruit at-risk youth. Finally, the evaluation literature points to five promising strategies to enhance regular participation and longterm program retention: (1) sending a clear message that regular attendance is important; (2) setting realistic goals to promote regular attendance, especially as youth grow older; (3) finding a balance among academic and other activities; (4) using incentives; and (5) keeping teens involved with opportunities for leadership, community service, and paid employment.     

Evaluating development programs: building joint activity

There has been an increasing plea for a participatory approach to development programs. It is argued that for programs to be sustainable, they need to be conceptualized, negotiated and run by all stakeholders jointly. Concurrent with this trend, there has been an increasing demand for program evaluation strategies, which encompass an evaluation of the nature and level of participation of the different stakeholders in the program. This paper proposes a theoretical framework for such evaluation and subsequently provides a practical illustration. Engeström's notions of ‘activity system’ and ‘learning by expanding’, are proposed as a framework for the evaluation of participation and an action evaluation methodology is suggested as the practical tool within such framework. The paper presents a case study of a conservation education program in Africa, which operationalizes the proposed theoretical framework.     

Program Evaluation Practice in the Nonprofit Sector

Nonprofit organizations are increasingly being asked to demonstrate the effectiveness of their programs and services. Faced with this challenge, agencies that decide to engage in program evaluation must choose among various approaches and methods. This article provides a window into why and how nonprofit organizations are conducting program evaluations, and it reveals the factors nonprofit agencies identify as contributing to a useful and credible evaluation. In addition, the article addresses the role of stakeholder participation in program evaluation.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

The State of Our Art

Abstract

In the second part of this two-part paper, the author examines intergenerational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, which appeared in the previous issue, conceptual foundations for intergenerational programming were considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross-cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Planning an integrated agriculture and health program and designing its evaluation: Experience from Western Kenya

Multi-sectoral programs that involve stakeholders in agriculture, nutrition and health care are essential for responding to nutrition problems such as vitamin A deficiency among pregnant and lactating women and their infants in many poor areas of lower income countries. Yet planning such multi-sectoral programs and designing appropriate evaluations, to respond to different disciplinary cultures of evidence, remain a challenge. We describe the context, program development process, and evaluation design of the Mama SASHA project (Sweetpotato Action for Security and Health in Africa) which promoted production and consumption of a bio-fortified, orange-fleshed sweetpotato (OFSP). In planning the program we drew upon information from needs assessments, stakeholder consultations, and a first round of the implementation evaluation of a pilot project. The multi-disciplinary team worked with partner organizations to develop a program theory of change and an impact pathway which identified aspects of the program that would be monitored and established evaluation methods. Responding to the growing demand for greater rigour in impact evaluations, we carried out quasi-experimental allocation by health facility catchment area, repeat village surveys for assessment of change in intervention and control areas, and longitudinal tracking of individual mother-child pairs. Mid-course corrections in program implementation were informed by program monitoring, regular feedback from implementers and partners’ meetings. To assess economic efficiency and provide evidence for scaling we collected data on resources used and project expenses. Managing the multi-sectoral program and the mixed methods evaluation involved bargaining and trade-offs that were deemed essential to respond to the array of stakeholders, program funders and disciplines involved.     

The opportunities and challenges of multi-site evaluations: Lessons from the jail diversion and trauma recovery national cross-site evaluation

Multi-site evaluations, particularly of federally funded service programs, pose a special set of challenges for program evaluation. Not only are there contextual differences related to project location, there are often relatively few programmatic requirements, which results in variations in program models, target populations and services. The Jail Diversion and Trauma Recovery–Priority to Veterans (JDTR) National Cross-Site Evaluation was tasked with conducting a multi-site evaluation of thirteen grantee programs that varied along multiple domains. This article describes the use of a mixed methods evaluation design to understand the jail diversion programs and client outcomes for veterans with trauma, mental health and/or substance use problems. We discuss the challenges encountered in evaluating diverse programs, the benefits of the evaluation in the face of these challenges, and offer lessons learned for other evaluators undertaking this type of evaluation.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

More than just being there: balancing the participation equation

The research and evaluation evidence is mounting: out-of-school-time (OST) programs can keep young people safe, support working families, and improve academic achievement and social development. Over 6 million children are enrolled in after-school programs nationwide, but an estimated 14.3 million children still care for themselves in the nonschool hours. Because of this discrepancy, OST stakeholders need information about how to maximize participation in OST programs. The Harvard Family Research Project (HRFP) has developed a conceptual model, based on scholarly theory, empirical research, and knowledge gained from providers, that describes the characteristics that predict participation in OST programs as well as the potential benefits of that participation. In the center of the model, participation is conceived as a three-part construct of enrollment, attendance, and engagement. This equation serves as the basis for framing this issue of New Directions for Youth Development. The chapter provides an overview of why participation in OST programs matters for young people, describes some of the barriers and challenges to youth participation, teases out more precise definitions of participation, and presents HFRP's conceptual model of participation. It focuses on the participation equation and concludes by highlighting some overarching themes that recur throughout the issue and that have an impact on future directions for research and evaluation.     

Context matters: Using mixed methods timelines to provide an accessible and integrated visual for complex program evaluation data

The need for conducting evaluations which reflect of the influence of context on complex programs is increasingly recognized in the field of evaluation. Better data visualization techniques for connecting context with program evaluation data are needed. We share our experience developing a mixed methods timeline to visualize complexity and context with evaluation data. Mixed methods timelines provide a meaningful way to show change over time in both a visually stimulating and accessible format for evaluation audiences. This paper provides an innovative example of using mixed methods timelines to integrate evaluation data with key program activities and milestones, while also showing internal and external contextual influences in one cohesive visual. We present methods and best practices for collecting contextual data and for incorporating a variety of data sources into such a visual. We discuss several strategies to collect and organize context related data including: qualitative interviews, program materials, narrative reports, and member checking with stakeholders and staff. Gathering multiple perspectives is essential to better capture the multi-layered elements of program activities and context.     

Evaluation plan of the 6for6 research skills program for rural and remote physicians

Overwhelming issues and barriers often prevent rural and remote physicians (RRPs) from pursuing the many socially accountable research questions they encounter on a daily basis. Although research training programs can empower RRPs to rise to these challenges, there is a lack of evidence on how they should be developed and refined. At Memorial University, a faculty development program (FDP) called 6for6 has been helping RRPs surmount their research quagmires and engage in scholarship since 2014. After an initial three-year (2014-17) pilot, we prepared a detailed plan to evaluate the 6for6 research FDP for RRPs and inform future years of delivery. Using a modified Delphi method and participatory action model a group of program team members, stakeholders and evaluation experts developed an evaluation plan including a logic model and an evaluation matrix addressing five key themes. To our knowledge, this is the first evaluation plan for a research-focused FDP targeting RRPs. While this plan was developed specifically for the 6for6 FDP, our approach to its development may be useful to any institution interested in evaluating an FDP with limited resources.     

Evaluation of a multi-site program designed to strengthen relational bonds for siblings separated by foster care

Sibling relationships in foster care settings have received increased attention in recent years. Despite growing evidence regarding the protective potential of sibling relationships for youth in care, some sibling groups continue to experience foster care related separation, and few programs exist to address the needs of these youth.This study describes and evaluates Camp To Belong, a multi-site program designed to provide short-term reunification to separated sibling groups through a week-long summer camp experience. Using a pre-test post-test survey design, this paper examines changes in youth ratings of sibling conflict and sibling support across camps located in six geographically distinct regions of the United States. The effects of youth age, number of prior camp exposures, and camp location were tested using multilevel modeling procedures.Findings suggest that participation in Camp To Belong may reduce sibling conflict, and improvements in sibling support are noted for youth who have had prior exposure to the camp’s programming. Camp-level variance in the sibling support outcome highlight the complex nature of relationships for siblings separated by foster care, and suggest the need for additional research. Lessons learned from this multi-site evaluation and future directions are discussed.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

The program effect of batterer programs in three cities

Recent meta-analyses and experimental designs of batterer program evaluations suggest little or no program effect. This finding may be compromised by a variety of analytical issues. Instrumental variable analysis addresses some of these issues, especially the relationship of program dropout to batterer reassault. This method of analysis was, therefore, used to test for program effect in a multi-site evaluation. The sites were three well-established batterer programs using a gender-based, cognitive-behavioral approach (n = 640). Completing a batterer program reduced the likelihood of reassault by 44% to 64%, depending on the specification used. Completing a 3-month program appeared to be as effective as completing a 5 1/2- or 9-month program. This moderate effect is an accomplishment considering the problems associated with the program participants and the inconsistency in the criminal justice system at the research sites. The findings remain tentative because of weak instrumental variables for reassault, but do confirm the need for more complex analyses of program effect.     

“Compassion,pleasantry, and hope”: A process evaluation of a volunteer-based nonprofit

As funders continue to emphasize the importance of documented results, nonprofit organizations must work to complete program evaluations that are both valuable and feasible. The purpose of this paper is to document a practical process evaluation of a southeastern nonprofit, a local Meals on Wheels. Using a mixed methods approach, we sought to answer four evaluation questions: (1) What are the essential program components, as identified by key stakeholders; (2) To what extent are volunteers implementing the identified essential components as intended; (3) What is the level of volunteer satisfaction with the program; and (4) What suggestions do stakeholders have for improving the program? Our findings indicate that most aspects of the program were implemented as intended, but inconsistencies occurred when volunteers were unsure of their assigned duties. In addition, volunteers had high levels of satisfaction and specific suggestions for improvement. From these results, we developed a conceptual model of factors contributing to quality of implementation and volunteer satisfaction that may be generalizable to other volunteer-based nonprofits. Specifically, we identified three factors that helped to facilitate satisfaction and performance: leadership, social contact, and fulfillment. Finally, this process evaluation demonstrates the feasibility of developing and implementing evaluation tools in similar organizations.     

Facilitating evaluations of innovative,competence-based assessments: Creating understanding and involving multiple stakeholders

Schools are held more responsible for evaluating, quality assuring and improving their student assessments. Teachers’ lack of understanding of new, competence-based assessments as well as the lack of key stakeholders’ involvement, hamper effective and efficient self-evaluations by teachers of innovative, competence-based assessments (CBAs). While evaluating two CBAs in Agricultural Vocational Education and Training institutions, two interventions in the evaluation process aimed to tackle these problems were examined: (1) starting with explicating the CBA in the teacher team using a concrete explication format and (2) qualitatively involving key stakeholders (i.e., teachers, students and employers) in the evaluation of the CBA through mixed-group interviews. Quantitative and qualitative analysis, as well as stakeholders’ perceptions are used to find indications for the added value of these interventions for evaluation and further improvement of the CBAs. Results show that external facilitation is needed to make both interventions work.However, under this condition, explicating the CBA led to more complete, concrete and shared understandings of the actual CBA among teachers and mixed-groups interviews resulted in more concrete and elaborate evaluations of the CBAs’ quality and more ideas for improvement. Both interventions can facilitate building up elaborate, more valid and concrete arguments for CBA quality in self-evaluations, certainly in the case of evaluating innovative assessments. Lessons learned will provide guidelines for incorporating the interventions into other evaluations of innovative programs.     

Predicting participation and outcomes in out-of-school activities: similarities and differences across social ecologies

The majority of research on out-of-school-time activity participation has focused on its relation to academic and social development, presumed to be consequences of participation, rather than on antecedents or predictors of participation. Understanding who participates in these programs can assist program directors in improving and sustaining youth involvement. This chapter uses data from two research study samples to examine differences in children's activity participation based on family social ecology and child gender and how the relations between participation and outcomes vary based on sample, gender, and activity type. Although children in both samples were of roughly the same age and were assessed for similar outcomes, their family incomes, socioeconomic status, ethnicity, and neighborhoods were very different. Findings suggest that participation in activities varies depending on the young person's social ecology, age, and gender. Furthermore, participation in activities was typically associated with positive youth outcomes, but these relations varied depending on the level of youth participation, type of activity, and social ecology.