A Model for Developing and Managing Distance Education Programs Using Interactive Video Technology

Based on two years of experience with a distance MSW program, the authors present an evolving model for the development, management, and evaluation of distance education graduate programs that use interactive video technology. The model includes five core components: (a) accreditation standards compliance, (b) resource requirements, (c) curriculum adaptation, (d) faculty development, and (e) program evaluation. While not an exhaustive listing, these components are advanced as central to the effective planning and administration of distance education programs in social work.     

Can Marriage and Relationship Education Be an Effective Policy Tool to Help Low-Income Couples Form and Sustain Healthy Marriages and Relationships? A Review of Lessons Learned

This review summarizes and synthesizes what researchers and practitioners have learned about the potential of public policy support for marriage and relationship education (MRE) to help lower income individuals and couples form and sustain healthy marriages and relationships. In short, this review documents modest, early evidence that low-income couples participate in well-designed MRE programs when they are offered, enjoy the educational experience, and report that the program is helpful. Practitioners have been going through a fast and steep learning process to figure out how best to recruit and maintain participation and adapt curricula to meet unique needs and situations. The evidence from the early outcome studies provides some support for the notion that MRE programs can have positive, modest effects on low-income couples' relationships, at least in the short run. However, much more research is needed to answer this question more definitively. Fortunately, more high-quality evaluation research will be coming over the next few years.     

A novel approach for evaluating programs designed to serve military veterans: Using an adapted common components analysis

Evaluators are challenged to keep pace with the vast array of Veteran support programs operating in the United States, resulting in a situation in which many programs lack any evidence of impact. Due to this lack of evidence, there is no efficient way to suggest which programs are most effective in helping Veterans in need of support. One potential solution to this dilemma is to reconceptualize program evaluation, by moving away from evaluating programs individually to evaluating what is common across programs. The Common Components Analysis (CCA) is one such technique that aggregates findings from programs that have undergone rigorous evaluation at the level of program components (e.g., content, process, barrier reduction). Given that many Veteran programs lack outcome evidence from rigorous studies, an adaptation to CCA is needed. This report examines cross-sectional data from a pilot study using an adapted CCA across five domains of well-being (i.e., employment, education, legal/financial/housing, mental/physical health, and social/personal relationships). The purpose of this preliminary study is to determine the feasibility of eliciting program nominations and program components from Veterans via an online survey. When coupled with a longitudinal research design, this adaptation to CCA will allow for stronger causal claims about the expected impact of different program components within and across a variety of domains.     

A theoretically derived model of premarital intervention: The building of a family system

Premarital intervention programs are on the rise in the United States despite the absence of persuasive empirical data that substantiate their effectiveness. In addition to this lack of strong empirical support, many premarital counseling programs fail to articulate any coherent theory of family process or family intervention which serve as conceptual guideposts for formulating program goals and devising program activities. In this article, the authors describe an experimental program that is conceptually grounded in theories of family process and family intervention. The theoretical rationale for each element included in the program is discussed and numerous examples of the program's content and exercises are offered. Finally, the authors discuss a number of issues central to the evaluation of premarital counseling programs.     

Process evaluation of the ‘Singapore Physical Activity and Nutrition Study’

IntroductionThe Singapore Physical Activity and Nutrition Study (SPANS) aimed to improve the physical activity (PA) and nutrition behaviours of Singaporean women aged 50 years and over. The SPANS program consisted of PA classes, nutrition workshops, telephone dietary counselling, health booklets, a health calendar and program ambassadors. This study aimed to assess and understand the implementation of the program strategies and gain insight into process evaluation components to inform future programs.MethodsThe evaluation was guided by a process evaluation framework and collected data via questionnaires (n = 209), program ambassador documentation and exit interviews with program completers (n = 13) and non-completers (n = 12).ResultsIn total, 295 participants completed the program (response rate = 84 %). Participants reported high levels of satisfaction with the overall program (99.5 %) and program activities (96.7 %), and also rated program ambassadors highly. Participation rates were highest for telephone dietary counselling sessions. The main reason for not attending program activities was having a ‘busy schedule’ (n = 158). Participants cited a need for improved recreational centre facilities and increased flexibility around program delivery.ConclusionsThe process evaluation showed that the program strategies were implemented as planned and were deemed suitable for supporting behaviour change among Singaporean women aged 50 years and over. The program reached and involved the majority of participants throughout the six months. The combination of practical educational resources and supportive program ambassadors were key strategies that facilitated positive PA and dietary behaviours. However, there needs to be some flexibility in the delivery of programs. The findings of this research may inform other programs in the region.     

The problem of replicability in program evaluation. The component solution using the example of case management

Specificity and replicability of a program are crucial in order to rigorously evaluate the program's effectiveness. However, the traditional approaches to evaluation have ignored the problem that most social and mental health programs consist of vaguely specified interventions and cannot meet minimal standards of replicability. Consequently, the results of effectiveness evaluations of poorly defined programs have little general generalizability. Nevertheless, effectiveness information concerning su ch programs is much needed. One solution to this problem is to focus on program components, examining interventions at a level where operational definitions can be employed and replicability is possible. In this paper, this approach is expanded using the example of case management programs. A component-oriented approach ensures a systematic evaluation of program effectiveness for which the first question for evaluators is whether a program consists of interventions that reliably produce specific immediate outcomes. Then, and only then, is it possible to address questions concerning whether a program will reliably produce those outcomes that are the overall objectives of the program.     

Faith-based evaluation: accountable to whom, for what?

Findings, issues, and lessons learned about program evaluation are examined from a national qualitative study of 15 faith-based human service programs targeting those in need in urban areas. Using a grounded theory design, five properties emerge as part of the evaluation network: (1) philosophy of accountability, (2) legitimacy, (3) evaluation design, (4) feedback loop, and (5) barriers to evaluation. While funders expect measurable outcomes to evaluate service effectiveness, respondents acknowledge other competing expectations of multiple constituents in religious and secular communities. What emerges is an excellent example of managing multiple program evaluation demands in programs that are particularly facile at process evaluation in the interest of quality service and relationship building. The article concludes with important lessons learned about the process of program evaluation.     

An Empirically Based Framework for Content Selection and Management in Divorce Education Programs

Contemporary research has shown that divorce education programs are becoming mandated or recommended in numerous states throughout the nation. Cooccurring with the popularity of these programs is the variation in content of divorce education programs. The focus of this project was to systematically explore the research literature on factors related to divorce adjustment and identify findings to be used in content selection and development for divorce education programs. The divorce-stress-adjustment perspective was utilized to frame the divorce process affecting both children and adults. A systematic literature review resulted in identification of 103 published studies that fit the project criteria. Selected research findings identified in the literature review process were organized according to a conceptual framework designed to categorize content in a tiered approach useful for selecting and managing content in divorce education programs. Implications of the project findings for content identification in divorce education, program development, and program evaluation are explored.     

Challenges and successes to the implementation of a zoo conservation-education program

Zoo conservation-education programs have potential to address biodiversity loss, although evidence of their effectiveness to encourage social change is in its infancy. Moreover, how a program is implemented may influence program efficacy, yet there is little evidence of the factors that shape this process in zoos. Accordingly, through a process evaluation of Zoos Victoria’s ‘When Balloons Fly” (WBF) conservation-education program against marine debris, we identified barriers and enablers to the implementation of the program which can be addressed to improve future initiatives. Between April-May 2018, 24 Zoos Victoria professionals completed an online survey focussed on identifying challenges and successes in implementing WBF. Four participants additionally completed a follow-up telephone interview. Themes were identified and organised according to the model of diffusion in service organisations (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004). Our results illuminate that features of the organisation (e.g., structure, culture) rather than features of individuals (e.g., skills, motivation) or characteristics of the conservation-education program itself (e.g., messaging), were most influential to implementation success. Based on these findings, attention to the organisational context is critical for promoting and evaluating the success of conservation-education programs and must be given significant attention alongside program characteristics and staff capability to deliver them.     

Conceptual Framework for Marriage Education Programs for Stepfamily Couples with Considerations for Socioeconomic Context

Stepfamily couples face unique challenges that put them at higher risk for dissolution than non-stepfamily couples. Risks for stepfamily couples are magnified in the context of lower economic resources, making low-income stepfamily couples a key target population for intervention and for community education programs. Existing programs and services for couples often do not address the unique situations of stepfamilies. This article builds on previous reviews that have offered information on important research themes and assessment of available curricula for stepfamily couples and presents a more comprehensive conceptual model to guide program design and services for stepfamily couples inclusive of specific objectives, as well as conditions and antecedents affecting content and prioritized needs. The goal is to enhance efforts to strengthen more vulnerable couples in complex families.     

Applicability of standards for evaluations of educational programs, projects and materials in an international setting

This paper starts with a short discussion of the book Standards for Evaluations of Educational Programs, Projects and Materials by the Joint Committee on Standards for Educational Evaluation (McGraw-Hill, 1981). It then goes on to examine the usefulness in countries other than the United States of these standards and argues that evaluation of “programs” is more complicated and more politically biased than evaluation of “projects” and “materials.” In Europe, evaluation of educational programs (school systems) has proved to be highly correlated with the overriding objectives of different political parties, e.g., conservative, liberal, socialist, whereas educational policy in the United States is less bound to political parties, varying rather from case to case. In order to show how generally agreed upon educational goals can be interpreted according to different values, the author has reanalysed some studies of mean achievement in reading comprehension and science for 10- and 14-year-old students in 15 countries (IEA Project, 1973). The rank order of these countries is changed if, instead of national mean achievement, one takes proportion of students under a minimum level or above a prescribed top level as the criterion. Also the national (program) standard is evaluated in terms of between-school variance in results and in proportion of single-school units with exceptionally low or exceptionally high mean results. The conclusion is that the book on standards for evaluations of educational programs provides a good checklist of prerequisites for a reliable and valid evaluation. However, it is also concluded that—due to differences in values of program outcomes—such standards do not guarantee that the result of the evaluation will be indisputable.     

Evaluating utility conservation programs: The BPA plan

This paper summarizes a detailed plan for evaluation of the residential energy conservation programs offered by the Bonneville Power Administration (BPA). The major purpose of the evaluation is to determine the energy savings attributable to the programs and to identify differences between program participants and nonparticipants. The focus of this paper is on the sampling issues associated with reliable estimates of energy savings. The methodology attempts to correct for the self-selection bias likely to occur in conservation program evaluation by using intentional nonrandom sampling. In addition, the paper suggests a variety of analytical approaches to analyzing the data with respect to energy savings.The proposed design, using four different groups to estimate program effects, is a comprehensive endeavor. This is appropriate for BPA because of the substantial investment it is making in conservation. However, the design is flexible and allows for use offewer groups to reduce evaluation costs.     

Internship training in marriage and family therapy: a survey of doctoral program objectives and implementation

This article presents the results of a survey that examined the implementation of the Commission on Accreditation for Marriage and Family Therapy Education (COAMFTE) internship requirement by accredited doctoral Marriage and Family Therapy (MFT) programs. Representatives from each of 10 doctoral MFT programs provided responses to questions regarding program objectives for internship training. Respondents also provided subjective impressions regarding the effectiveness with which established internship goals are accomplished, associated benefits and difficulties, and adequacy of the standards. Results suggest that although most programs perceive the internship requirement as beneficial, many express concern regarding the effects of the internship requirement on other aspects of training and in relationship to the overall goals for doctoral MFT education. Conclusions and recommendations are offered based on the mixed findings.     

Theory-based evaluation in practice. What do we learn?

Theory-based evaluation (TBE) explores the how and why of program success or failure. Advocates of TBE claim that it produces information unavailable in traditional process and outcome studies. This article examines six published papers of TBEs. It finds that the authors of the papers do not always make explicit the relation of their data to the theory of the program. Nevertheless, it was evident in one or more cases that TBE identified unnecessary program components, located intermediary changes, raised new questions, contributed to a paradigm shift, highlighted the difficulties of taking successful pilot programs to scale, and provided clarity and focus for the evaluation. Interestingly, in none of the studies was the original theory completely right. Lessons for the future of TBE are drawn.     

Setting standards of performance for program evaluations: the case of the teaching hospital general medicine group practice program.

Program evaluation is like research in its use of the scientific method. An important difference is that evaluations result in judgments of merit. What are the standards for making the judgments? Little attention has been paid to their selection and use. This article reports on how standards were set in an evaluation of the structure of fifteen of the nation's university hospitals who participated in the Teaching Hospital General Medicine Group Practice Program (sponsored by the Robert Wood Johnson Foundation). Many sources were used to select standards including a review of the literature, expert advice and actual data from two years of the programs's performance. Also, the standard-setting process was a participatory one in which all potentially competing views were provided with a forum for discussion. Finally, standards were set in advance of the collection of information, facilitating the selection of study designs and analysis techniques. Almost all project directors stated that the process helped them in program planning and gave them ideas for research and evaluation. Because standard-setting has mutual benefits, we recommend that it take place during program planning.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

An investigation of a workplace-based return-to-work program for shoulder injuries

The purpose of this study was to investigate and evaluate the current workplace management of rotator cuff injuries in a manufacturing plant. The secondary aims were to examine the impact of the company's return-to-work processes, compare outcomes to current industry standards for work (re)entry and to identify the components that characterized this workplace-based return-to-work (RTW) program. This investigation involved a case study approach comprised of an examination of the program context using interviews, onsite visits, a document review and a retrospective analysis of the RTW experiences of 184 workers with shoulder injuries. Findings revealed that the workplace-based RTW program was consistent with and shaped by the organizational culture of problem solving, knowledge exchange and equitable participation of workers, supervisors and health professionals. These components contributed to the program in achieving the following outcomes for workers with shoulder injuries. One-third of workers were placed on modified duties within three days, 56% of workers who engaged in an early RTW program returned to work within one month. Overall, 87.8% of workers with rotator cuff injuries successfully returned to pre-injury work. The implications of developing capacity for workplace-based programs to manage injuries at work are discussed.     

Evaluating the integration of strategic priorities within a complex research-for-development funding program

This paper examines the application of Complexity Theory constructs to a research-for-development program evaluation and presents an overview of the implications and promising approaches for evaluating complex programs. We discuss lessons learned from an evaluation completed for the International Development Research Centre’s Food, Environment and Health (FEH) program, which investigated the integration and outcomes of five strategic program priorities: partnerships, southern leadership, gender and equity, scale, and environmental sustainability. We present interpretations from a secondary, thematic content analysis that categorized evaluation findings across four complexity constructs: emergence, unpredictability, contradiction and self-organization. Viewing the evaluation through these constructs surfaced some important features of the FEH program to date, specifically its evolving approach, adaptiveness to emergent issues, non-linear outcomes, and self-organizing agents, which had several implications for the evaluative process. We conclude that the most appropriate evaluation designs for complex funding programs are participatory (to explore all stakeholders' influence), adaptive (to capture the unexpected) and assess external contexts. The application of complexity constructs may be useful for evaluators to gain a deeper understanding of how program contexts change in the face of complexity and why some evaluation methods work more effectively than others.     

Using network analysis to conduct a system-wide program evaluation within a university

ObjectiveTo conduct a system-wide assessment using social network analysis (SNA) to examine how 14 important issues (e.g., consent; racism) are addressed through education, training, and programming at a university.MethodsEvaluators conducted interviews with campus departments responsible for educating/training on the 14 issues. Interviews revealed which programs (n = 52) were offered that addressed the 14 issues, and data on audience characteristics, date of delivery, and which issues were covered in each program were collected. SNA was used to calculate degree and create visualization graphs illustrating patterns of content-coverage across all 52 programs.ResultsThe average degree was 19.38 (SD = 9.70), meaning programs overlapped in topic area with nearly 20 other programs, on average. Most programs (n = 36; 69.2 %) were attended by audiences of 500 people or less. “Diversity and inclusion” represented the topic area with the most programs (n = 23), whereas “suicide” and “bullying/hazing” had the least number of programs (n = 3). Degree was negatively correlated with attendance numbers (r=-.310, p < .001), indicating the more a program overlapped in content with other programs, the smaller the audience.ConclusionsThis study supports the use of network analysis in conducting systemic evaluations of programs offered at a university, complementing the work of ongoing, local-level program evaluations.     

How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.