Piloting online WellnessRx learning modules: Demonstration of developmental evaluation

WellnessRx is a health initiative focusing on healthy living through education, knowledge translation, and community engagement. Stakeholders of WellnessRx identified web-based education learning modules on nutrition and physical education as a priority to be integrated into existing health sciences curricula, as well as adapted for use by health professionals. Five learning modules were created with essential knowledge, skills, attitudes and resources or tools for health professional students and practitioners. As part of the ‘developmental evaluation framework’ for WellnessRx, two of these modules were piloted within two health professional student programs. This paper describes the pilot-evaluation experience involving student surveys, focus groups and interviews, and faculty perspectives. For both modules, student pre-post knowledge assessments indicated some improvements in post-module knowledge. Post module evaluations by students indicated benefits with the online delivery being flexible for access, self-health, case-based assessments and useful nutrition and physical activity guides. Challenges for students included their time to do the modules and the activity expectations. Instructors felt each module could be better targeted to different years within an undergraduate program. Through developmental evaluation, the pilot results along with recommendations and lessons learned provided the direction needed to further develop the WellnessRx logic model and proposed learning modules.     

Learning by doing: an experiential approach to program evaluation

Despite the increased importance of producing social workers who are prepared to evaluate practice, there is a paucity of literature addressing the pedagogical challenges of program evaluation courses. The present study evaluates the structure and pedagogical approach of a newly implemented program evaluation course. The course took a team-based, experiential learning approach to designing a program evaluation plan for students’ field placement agencies. Five student-led focus groups (N = 44) were conducted at the conclusion of the course to address two aims: to assess student perceptions of the content, structure, and delivery of the course; and to assess student attitudes toward applied research. Data were collected between November and December 2014 in a public university’s graduate social work program in the northeastern United States. The steps of grounded theory data analysis were implemented to elicit major themes from student feedback. Results suggest that perceptions of applicability play a central role in shaping student attitudes toward program evaluation. Experiential and team-based pedagogical approaches appear to increase accessibility of course content for students. Suggestions for enhancing student learning in program evaluation courses are discussed.     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

Evaluating from the outside: conducting cross-cultural evaluation research on an American Indian reservation

There is limited guidance for conducting competent and responsive cross-cultural evaluation research with American Indian communities. The authors draw on Fisher and Ball's Tribal Participatory Research Model to highlight ways in which this project is attempting to be culturally appropriate and sensitive as they partner with an American Indian community to implement and evaluate a youth-based initiative. Challenges encountered during the evaluation are shared, as well as the authors' collective responses to such challenges. Implications for future cross-cultural evaluation researchers are also discussed in light of these experiences.     

Learning technologies at the service of higher education: global trends and local Israeli opportunities

This paper provides an overview of global trends in e-Learning and opportunities opened by harnessing learning technologies at the service of the Israeli higher education. The introduction of innovative learning technologies is examined in light of an organizational change. A series of substantial issues should be taken into account to successfully manage the change. Among these are: rethinking of academic learning and instruction, redefining the role of the teachers and the students, decoupling the learning materials and the learning process, new approaches to student assessment and program evaluation, and instituting a well thought out organizational plan including a support center. The above-mentioned issues guided the introduction of a recent call for project proposals for the integration of technologies into Israeli higher education. The main principles, criteria and results of this call are described.     

Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program

In 2005, state fish and wildlife agency wanted to examine how one of its conservation education programs was providing science-based understanding and outdoor experiences by evaluating students’ knowledge, skills, attitudes and intended behavioral outcomes related to fish, fishing and aquatic habitats in Montana. A key factor in this study was the acceptance by program stakeholders to conduct the evaluation using a utilization-focused evaluation approach to promote usability and accuracy of evaluation results. Using a quasi-experimental non-equivalent group design, more than 2000 students in participating classrooms throughout Montana received a pre-survey, post-survey and an extended post-survey; 114 teachers participated in an Internet survey and 16 program instructors took part in a structured open-ended telephone interview. The participatory approach and mixed methods enhanced abilities to interpret results of student surveys in particular. The user-focused approach was discovered to be personal and situational, allowed the facilitation of the evaluation process with consideration for increased application of evaluation findings and implementation of recommendations from beginning to end. Further development of evaluation pathways is needed to more effectively evaluate outcomes and implement practical and transferable measures to determine if environmental education activities produce desired participant outcomes.     

Applying utilization-focused evaluation to high school hazing prevention: A pilot intervention

Researchers argue that hazing can contribute to an abusive school climate and interfere with a positive learning environment for students. National efforts exist for establishing policies, protocols, evaluation, and education for students, administrators, and staff to prevent hazing at the college level, but this work has yet to be applied broadly in a high school context. In response to this gap, researchers implemented a pilot project at two high schools in Maine that consisted of hazing prevention training and assessment. This paper discusses the design, methods, and lessons learned through this collaborative, utilization-focused, and mixed-method training and evaluation with school personnel and high school student participants.     

Teaching Successful Community Practice in Social Work

ABSTRACT

This article identifies and addresses a severe problem in the management of community practice projects of students in their second and third year in the Undergraduate Social Work program at the University of Haifa School of Social Work; namely, the current binary evaluation method based on total success or failure of the project.

Using a systems approach, this study creates a framework by which to ensure effective and positive outcomes for community practice projects carried out by undergraduate social work students (Freund, Arkin & Saltman, 1999). This framework is unique in that it draws on knowledge gained from organizational and community theories, as well as instructional theories for teaching social work practice.

The proposed framework includes three components that describe the factors necessary to help transform the student's learning experience: the organization's characteristics, the student's resources, and the evaluative measures.

This inter-conceptual approach enhances the ability of the instructor, the student, and the organization to build an appropriate organizational foundation for students operating communal programs. The underlying assumption is that an integrative approach promotes a better understanding of the components essential for building and evaluating social plans conducted by social work students.     

The context of program implementation and evaluation: A pilot study of interorganizational differences to improve child welfare reform efforts

This article contributes to the growing literature on evaluation and implementation science by examining the interaction between staff perceptions of organizational strength with perceptions and indicators of program fidelity. As part of a pilot project related to the evaluation of the Family to Family initiative, a survey was distributed to employees within two urban child welfare agencies with a total of 410 respondents across both sites, for a combined response rate of 72.2%. Survey results were analyzed both in terms or respondents' perception of their agency as well as in relation to measures of program performance and workload. Multivariate models show that organizational indicators are the most significant and positive predictors of perceived program implementation. Specifically, staff who positively perceived the availability of information within their agency also believed that the programs were well implemented in their agency. These findings suggest that as the value of program changes are articulated within an organization, the implementation of the initiative is perceived to improve.     

TEACHING NOTES: MAKING STATISTICS "REAL" FOR SOCIAL WORK STUDENTS

This article presents results from an evaluation of service learning in statistics courses for master of social work students. The article provides an overview of the application of a community-based statistics project, describes student feedback regarding the project, and illustrates some strengths and limitations of using this pedagogy with social work statistics courses. Student evaluations (N=38) suggest that implementing a service-learning pedagogy allows students to master basic statistical analysis skills and see practical applications for statistics. Such an approach also provides myriad benefits for faculty, although incorporating service learning into a statistics course can present specific challenges.     

Correlates of honor ratings in a clinical clerkship employing a faculty forum evaluation system

5he purpose of this study was to determine whether or not suspected variables affected a surgery clerk's chances of being awarded an honor rating. Findings indicated a significant relationship between a student receiving an honor rating and his or her preceptor's predetermined level of student advocacy, the number of completed patient interview and physical examination write-ups, and final examination scores. There was no significant relationship found between honor ratings and the preceptor's status, the sequence in which the student was discussed or the length of time spent discussing the student at the final evaluation meeting, or the number of clinical faculty present at the meeting. These results lend support to the faculty forum evaluation approach, but suggest a need for further scrutiny of some influencing variables to ensure all students are fairly considered and honor ratings judiciously awarded.     

Design and implementation of a telecollaboration project for primary school students to trigger intercultural understanding

ABSTRACT

Telecollaboration involving children around the world allows them to develop international understanding (IU). This paper reports on the design and implementation of a telecollaboration project between Japanese and Australian primary school students. The main purpose of the project, which integrates the contents of Social Studies and foreign language (FL) education, is to promote student interest in learning their own and a foreign language, as well as a foreign culture, using an educational social network site (SNS), in this case Edmodo. This study focused on Japanese students in particular. The exchange of cultural information online became a meaningful and authentic activity for communication in FL. Additionally, the evaluation showed that the Japanese students became more motivated to learn English and initiate IU. A curriculum-based telecollaboration project involving a long-term plan – instead of a one-off activity – is recommended in order to develop young students’ IU in FL education.     

The research to teaching initiative: infusing faculty research into the MSW curriculum

Social work educators have a responsibility to facilitate student utilization of research‐based knowledge. Tremendous research based knowledge is generated at many schools, but due to multiple responsibilities, faculty often find it difficult to share their findings with fellow faculty outside of their immediate project team, or to inquire into the research efforts of their colleagues within the school. This paper describes the development, implementation and preliminary evaluation of Research to Teaching, an initiative to infuse the curriculum of a school of social work with current research findings from its faculty. The goal of the Research to Teaching initiative is to take current publications from faculty, and distill them into easy‐to‐use lecture materials including slides summarizing the content, teaching notes, discussion questions, and group activities such as skill practice, simulation, and other integrative activities. These lectures are placed on a site within the school's larger website, and are accessible to all faculty, students, alumni, and other interested individuals.     

The results framework--an innovative tool for program planning and evaluation.

This article constitutes a case study of the development and implementation of the "results framework," an innovative planning and evaluation tool that is rapidly becoming a standard requirement for United States Agency for International Development (USAID) projects. The framework is used in a USAID-funded regional initiative for HIV/AIDS prevention in Central America. This new program evaluation and monitoring tool provides many advantages over traditional evaluation approaches that use outside consultants to provide midterm and end-of-project evaluations. The results-framework process, which spans the life of the project, provides an opportunity for program staff, donors, partners, and evaluators to work as a team to collect and use rich, longitudinal data for project planning, implementation, and evaluation purposes.     

Building evaluation capacity in Spain: a case study of rural development and empowerment in the European Union

The development of European Community administrative authority has greatly influenced the development of an evaluation culture among the southern and central member states of the European Union. The present case study from Spain provides an example of this diffusion through the use of an empowerment evaluation approach to build evaluation capacity within the context of rural development. The study focuses on the evaluation process over a 10-year period of three local development programs run by three local partnerships created in the rural areas of the Madrid region through the European LEADER initiative. Critical components of empowerment evaluation are discussed, as well as the impact, limitations, difficulties, and applicability of the approach to rural development in the European Union. Conclusions are presented concerning evaluation capacity building and the development of an evaluation culture.     

Integrating Research and Practice into an Interviewing Skills Project

This paper reports an approach to improving the integration of research and practice instruction by engaging students in a research project. Through the project, the students evaluate an effort to improve their interviewing and relationship-building skills. The skills-development component relies primarily on student role playing and peer feedback, and was implemented as part of a required second-year master of social work research course. Two randomized experimental evaluations of the project’s effectiveness in improving interviewing skills yielded inconsistent results. The authors describe strategies for teaching and assessing interviewing skills and for teaching research.     

The Taping Project

This article describes and evaluates the effectiveness of a “Taping Project,” a teaching method for student self-evaluation in racism courses. The project offers students the opportunity (a) to hear in private the assumptions about themselves and others that they have brought into the course, (b) to reexamine these assumptions in the context of what they learn in the course, and (c) to consider if and how these assumptions have changed. Formal evaluation of the project’s effectiveness in helping students observe and reflect critically on their own development is reported in two ways: content analysis of 53 student essays using grounded theory analysis, and student and faculty evaluations of the assignment.     

School-Based Intergenerational Programs

Abstract

This paper focuses on experiences learned from being involved in the development of a sustainable school-based intergenerational program that simultaneously incorporated lifespan education into a Grade 2-3-4 curriculum. The program operated in an Australian primary school. A model was developed through this initiative that involved a tripartite collaborative partnership between school, university educators, and the primary school community (in particular its older adults). Older adult volunteers from the school's community were invited to teach students. Program monitoring consisted of contextual and observational analysis ensuring the program was sensitive to school culture and took account of participants' experiences. This monitoring process occurred throughout the planning, development, and implementation stages. Although school educators were always pressed for time, all participants actively contributed to the program. All of the educators saw purpose and worth in having the program. Ongoing support from teachers, the principal, and university personnel plus the provision of additional resources contributed to high student involvement and the overall success of the initiative. Having a school-based coordinator was found to be necessary, yet, for school educators, problematic because of constraints on their time. The findings from this project indicate that the older adult volunteers were enriched by their involvement in the program and able to provide meaningful learning experiences for students. As well, the study suggests that educators would be better prepared for implementing this area of learning opportunities if lifespan education were provided in their undergraduate and post-graduate courses as well as in professional development initiatives.