Blazing an evaluation pathway: Lessons learned from applying utilization-focused evaluation to a conservation education program

In 2005, state fish and wildlife agency wanted to examine how one of its conservation education programs was providing science-based understanding and outdoor experiences by evaluating students’ knowledge, skills, attitudes and intended behavioral outcomes related to fish, fishing and aquatic habitats in Montana. A key factor in this study was the acceptance by program stakeholders to conduct the evaluation using a utilization-focused evaluation approach to promote usability and accuracy of evaluation results. Using a quasi-experimental non-equivalent group design, more than 2000 students in participating classrooms throughout Montana received a pre-survey, post-survey and an extended post-survey; 114 teachers participated in an Internet survey and 16 program instructors took part in a structured open-ended telephone interview. The participatory approach and mixed methods enhanced abilities to interpret results of student surveys in particular. The user-focused approach was discovered to be personal and situational, allowed the facilitation of the evaluation process with consideration for increased application of evaluation findings and implementation of recommendations from beginning to end. Further development of evaluation pathways is needed to more effectively evaluate outcomes and implement practical and transferable measures to determine if environmental education activities produce desired participant outcomes.     

Rock Up: An initiative supporting students' wellbeing in their transition to secondary school

This paper reports the background and findings of the community-based pilot program ‘Rock Up’ to school. The Rock Up program was directed at a niche section of young students in Grades 5, 6, and 7 to respond to identified risk factors associated with disengagement from formal education. The Rock Up approach was to energise these students' connection to learning as they approach the difficult stage of transition from primary school to secondary school. Ultimately, Rock Up aims to assist those students in their development of wellbeing, within and outside of the school environment. The operational measures of wellbeing covered; academic, social, emotional, behavioural, and school absence. Thirteen students, who had been identified by their respective teachers, were invited to participate in individual and/or group activities that focused on their wellbeing and developing their readiness for secondary school. Written and verbal feedbacks from students, teachers, parents, and the program facilitator, were in the main very positive. Analysis of the data of the first round of the program using a three-wave questionnaire and a participatory action research approach indicated the need to proceed to a new round using more refined processes. When fully developed, Rock Up is intended to be the vanguard for collaborative practice between the education and community sectors throughout Victoria and to be an example of an efficient and effective model of transition for this age-group.     

Student perceptions of international education and study abroad: a pilot study at York University,Canada

International student mobility has been identified as a key strategy for the internationalization of higher education. Although an institutional priority, Canada has among the lowest levels of international student mobility, with only 2% of full-time university students participating in study-abroad programs. This pilot study, conducted at a large public university in Toronto, examined the value that students place on international education, their awareness of opportunities made available by the university, their attitudes toward, perceptions of, and preferences toward study abroad, and the institutional and individual factors that influence their intent to engage in study abroad. The study found associations between students’ intent to study abroad with their perceived social and institutional support and academic hassles at the host and home institution. It identified three distinct groups within the population, those intending to study abroad, those unsure about their plans, and a third group who does not seek to pursue study abroad. In terms of applied value, the findings will inform program administrators how to customize their support services and programs to both assist interested students and attract new students that otherwise would not be interested in such an experience.     

Prevention of homicidal violence in schools in Germany: the Berlin Leaking Project and the Networks Against School Shootings Project (NETWASS)

Since 1999, Germany has experienced at least twelve serious cases of targeted school violence. This article describes two projects designed to fill the gap between universal prevention and emergency response in preventing severe forms of school violence in Germany. The Berlin Leaking Project examined the viability of preventive efforts based on early identification of leaking behavior that often precedes targeted school attacks. Leaking refers to any behavior or communication that indicates a student is preparing to carry out a violent attack. This would include explicit or implied threats of violence, apparent fascination with prior acts of violence such as Columbine, and any evidence of planning or preparation to carry out an attack. The NETWASS project will test a training program and intervention strategy based on those findings, examining the usefulness of a threat assessment approach to prevent violence by training teachers to recognize leaking behavior by students. This approach is extended by training teachers on a larger scale to identify leaking and then having a school-based team evaluate the student and initiate appropriate interventions, such as mental health services, and in some cases, law enforcement action.     

Research and Teaching of Casework

Older patients discharged from nursing homes were randomly assigned to student teams from medicine, nursing, and social work (C.A.R.E.) or to no specific aftercare. Patients seen by student teams had significantly more knowledge about community resources than did control patients, but showed no difference in rate of readmission or level of adjustment. The 24 students in the program developed significantly better attitudes toward the aged than control students. A student team approach offering services to the elderly may be one way of positively shaping attitudes of young preprofessionals in the health field.     

Learning by doing: an experiential approach to program evaluation

Despite the increased importance of producing social workers who are prepared to evaluate practice, there is a paucity of literature addressing the pedagogical challenges of program evaluation courses. The present study evaluates the structure and pedagogical approach of a newly implemented program evaluation course. The course took a team-based, experiential learning approach to designing a program evaluation plan for students’ field placement agencies. Five student-led focus groups (N = 44) were conducted at the conclusion of the course to address two aims: to assess student perceptions of the content, structure, and delivery of the course; and to assess student attitudes toward applied research. Data were collected between November and December 2014 in a public university’s graduate social work program in the northeastern United States. The steps of grounded theory data analysis were implemented to elicit major themes from student feedback. Results suggest that perceptions of applicability play a central role in shaping student attitudes toward program evaluation. Experiential and team-based pedagogical approaches appear to increase accessibility of course content for students. Suggestions for enhancing student learning in program evaluation courses are discussed.     

Same or different? A comparative study of habitus—ultra-orthodox and secular students of teaching

Although the state ideology assumes that by virtue of their affiliation with the “same” religion, Jewish citizens of Israel are a culturally homogeneous population, the school system makes room for institutions with different orientations to the faith. To assess whether teachers in schools that differ in their approach to religion are likely to educate children toward a unified Israeli–Jewish culture, we investigated aspects of the habitus, the configuration of dispositions into which student teachers had been socialized. The inquiry is based on research literature related to choosing teaching as a vocation and to the significance of habitus in education. A closed questionnaire on motives for choosing to teach disclosed that secular students cited intrinsic motives (creativity, individual interest), and ultra-orthodox students placed more emphasis on extrinsic motives (gaining a livelihood, community esteem). Findings from semi-structured interviews highlight conspicuous differences between the groups in their conceptions of habits, their processing of attitudes, their values, and their orientation to the functions of teaching. We conclude that an intensive program of intercultural education is needed to overcome the structural and curricular divisions institutionalized in the Israeli school system.     

School-Based Intergenerational Programs

Abstract

This paper focuses on experiences learned from being involved in the development of a sustainable school-based intergenerational program that simultaneously incorporated lifespan education into a Grade 2-3-4 curriculum. The program operated in an Australian primary school. A model was developed through this initiative that involved a tripartite collaborative partnership between school, university educators, and the primary school community (in particular its older adults). Older adult volunteers from the school's community were invited to teach students. Program monitoring consisted of contextual and observational analysis ensuring the program was sensitive to school culture and took account of participants' experiences. This monitoring process occurred throughout the planning, development, and implementation stages. Although school educators were always pressed for time, all participants actively contributed to the program. All of the educators saw purpose and worth in having the program. Ongoing support from teachers, the principal, and university personnel plus the provision of additional resources contributed to high student involvement and the overall success of the initiative. Having a school-based coordinator was found to be necessary, yet, for school educators, problematic because of constraints on their time. The findings from this project indicate that the older adult volunteers were enriched by their involvement in the program and able to provide meaningful learning experiences for students. As well, the study suggests that educators would be better prepared for implementing this area of learning opportunities if lifespan education were provided in their undergraduate and post-graduate courses as well as in professional development initiatives.     

Effects of a participatory ergonomics intervention computer workshop for university students: a pilot intervention to prevent disability in tomorrow's workers

A participatory approach was used to create a computer ergonomics workshop for college students, incorporating an instructional systems design process and adult learning inquiry perspectives. The primary objective of this participatory ergonomic pilot intervention was to involve students throughout the training design process in solving computer workstation ergonomic problems and adopting healthy computing behaviors. Students' level of participation included becoming part of the training design team, a co-facilitator, or a student trainee. A second objective was to examine the translation of an industrial office ergonomics training program into a college computer ergonomics training program. The long term goal was to reduce upper extremity symptoms and disability. The program was piloted at one private university. The three student trainees significantly increased their knowledge of computer ergonomics from 69% of test items answered correctly pre-training to 82% post-training. Trainees were also successful in conducting computer ergonomic evaluations of students' computing work areas. They achieved 100% accuracy in identifying ergonomic problems and proposing solutions in five ergonomic workstation assessments in the field. This approach was successful in creating a sense of ownership among the student developers and facilitators as reflected in their self-reports during a post-intervention debriefing. The results of this pilot study justify formal controlled trials of this intervention in university students, who will become tomorrow's workers.     

Piloting online WellnessRx learning modules: Demonstration of developmental evaluation

WellnessRx is a health initiative focusing on healthy living through education, knowledge translation, and community engagement. Stakeholders of WellnessRx identified web-based education learning modules on nutrition and physical education as a priority to be integrated into existing health sciences curricula, as well as adapted for use by health professionals. Five learning modules were created with essential knowledge, skills, attitudes and resources or tools for health professional students and practitioners. As part of the ‘developmental evaluation framework’ for WellnessRx, two of these modules were piloted within two health professional student programs. This paper describes the pilot-evaluation experience involving student surveys, focus groups and interviews, and faculty perspectives. For both modules, student pre-post knowledge assessments indicated some improvements in post-module knowledge. Post module evaluations by students indicated benefits with the online delivery being flexible for access, self-health, case-based assessments and useful nutrition and physical activity guides. Challenges for students included their time to do the modules and the activity expectations. Instructors felt each module could be better targeted to different years within an undergraduate program. Through developmental evaluation, the pilot results along with recommendations and lessons learned provided the direction needed to further develop the WellnessRx logic model and proposed learning modules.     

Government Documents Come of Age

In January 2007, the Oregon State Board of Education mandated a new Oregon diploma that strengthened high school graduation requirements with the aim of improving student readiness for college and career. Among the major changes was a requirement that students demonstrate proficiency in nine “essential skills” that included the expectation that students must “demonstrate global literacy.” This article examines the process through which Oregon developed and plans to implement the essential skills and compares its conceptualization of global literacy with alternate models of global education in other states and nations. The authors identify political and institutional factors influencing the shape and direction of those efforts and offer recommendations regarding a potential framework for defining, implementing, and assessing global literacy in Oregon high schools.     

The Child Labor Amendment in the Social Studies Class

As part of a geography-based minority recruiting program (MRP), teachers were asked to serve as mentors for teams of high school students interested in both postsecondary education and geography as a major discipline or career path. The goal was to determine whether teacher participation in MRP programs was an effective tool to mentor underrepresented populations in geography. One component of the program evaluation examined the impact of teacher participation in a MRP by analyzing participating teachers’ classroom strategies and students’ performance in the school year following the summer program. Teachers were surveyed and participated in follow-up interviews during the school year following their participation in the MRP to determine the effects of the program's professional development components. The research hoped to reveal: (a) the participant-teachers’ function and role as mentors for students and the changes of participant-teachers’ behaviors, and (b) how any changes in their classrooms have affected their students’ understanding of geography. Data collection, reduction and analysis followed a grounded-theory approach to qualitative research. The research confirmed that participants leave professional development with an increased enthusiasm for and a better understanding of geography that they shared with their colleagues and students. Furthermore, teachers reported overcoming campus and district barriers in order to implement strategies and technologies provided during the MRP. More promising findings validate the prevailing thought that the positive energy of the MRP continues to reach into the classrooms in the following years. By understanding the impacts of professional development on participating teachers, the discipline may better recruit and retain high-achieving minority students.     

(Re)Conceptualising inclusive education in New Zealand: listening to the hidden voices

This article explores the experiences of 14 students with severe and complex behaviour, their caregivers, and teachers/principals as the students transition from a disestablished residential school back to mainstream schools, with support from the Intensive Wraparound Service (IWS). Interviews were the primary source of data collection. Data were collated into 14 cases, and analysed using a general inductive approach. Two broad themes are addressed in the article: education and relationships/communication. The research found that the IWS is not living up to its rhetoric. Suggestions made for improvement are based on approaches that participants found worked for them.     

An Evaluation of a Parent-Developed,Parent-Run Parent Education Program

This article reports on a new approach to parent education in which a group of parents developed a parent education curriculum and facilitated the program. The approach is based upon a model of empowerment that asserts that parents, working collectively, can affect meaningful change in their lives. After describing the HOPE (Helping Our Parents to be Educators) Project, the article explores whether the program successfully influenced parents' abilities to parent well. Using a repeated-measures model, the study found that participation resulted in improved communication within the family, enhanced social support, better attitudes toward the child, and increased parental empowerment.     

“Teachers think the kids around here,don't really want to learn”: Street-identified black men and women's attitudes toward teachers and schooling

This street participatory action research project explored the reflective schooling experiences of street identified Black men and women (ages 18–35) in two small low-income neighborhoods. Secondary analysis of survey (N = 520) and interview (N = 46) data examined: (1) How are attitudes toward schooling and teachers affected by race, gender and age?; and (2) How do students utilize a street-identity as a site of resilience inside schools? Overall, street-identified study participants held positive attitudes toward schooling, but generally performed poorly in schools and had negative experiences with educators. No significance was found as a function of gender and age regarding attitudes toward schooling and attitudes toward teachers. Also, interview results, across gender and age, suggest school-related structural challenges and poor teacher-student relationships contributed to severe conflict between students and teachers; and between students. Interviewees argued some Black students internalized a street identity or became disruptive and even engaged in school violence as a protective mechanism to endure hostile schooling environments. Moreover, Street PAR is discussed as a method and intervention to improve student performance and resolve concerns between students and educators.     

A re-evaluation of project pride, a redesigned school-based drug abuse prevention program

The present study examined the effectiveness of Project PRIDE, a school-based, counselor-administered, drug and alcohol prevention program. The study is presented in the context of Project PRIDE'S efforts to keep itself current and effective via continual evaluation-based development. In this outcome evaluation, Project PRIDE participants demonstrated greater pretest to posttest gains on five of six outcome measures compared to control students, although many of the changes were small or moderate and only two were statistically significant. Further, as predicted, gains in more factual or objective areas of the program were greater than in more subjective areas dealing with student perceptions and feelings. The limitations and implications of the findings are discussed. Also discussed are the advantages of the Binomial Effect Size Display as a data presentation mode that promotes both client and general audience understanding of results.     

Project HOPE: Evaluation of Health Science Career Education Programming for Rural Latino and European American Youth

Project HOPE comprises a health science career pipeline program based in social cognitive career theory. The program aims to promote health science career exploration for rural and Latino middle school students while simultaneously emphasizing the importance of achieving in math and science subjects. The authors conducted 2 studies to evaluate Project HOPE using baseline and follow‐up evaluation data. Results of Study 1 indicated that Project HOPE was associated with increases in math/science self‐efficacy for both Latino (n = 42) and European American (n = 31) students. Results of Study 2 indicated that Project HOPE was associated with higher health science career self‐efficacy beliefs for European American students (n = 22) and higher health science career interests for Latino students (n = 28). These results suggest that domain‐specific career interventions for middle school students that focus on jobs tied to the local economy may be a useful way to engage students in career exploration.     

Assessing student staff motivation and satisfaction to strengthen health education services

A student staff inquiry project in the department of health education at a large state university assessed the motivation and satisfaction of student staff and peer educators. The students who managed this Student Inquiry Project created the interview tool, collected and analyzed the data for themes, and developed follow-up recommendations. Through this student-driven qualitative process, several themes emerged, including the need for more department visibility, student opportunities, student identity, skill building, and appreciation. The department is using the recommendations to strengthen student connection, satisfaction, and training.     

Project Northland in Croatia: results and lessons learned

As part of an international public health partnership formed in 2001 we assessed the impact of Project Northland in Croatia--an adolescent alcohol school-based prevention intervention implemented in Split--on students. The curriculum was implemented in 13 randomly selected schools with another 13 schools serving as controls. The evaluation included quantitative and qualitative methods, using pre/post-tests to compare intervention and controls and follow-up focus groups with parents and teachers. Quantitative data showed the intervention was more successful in changing attitudes in the first two years, but had less impact on older students. Qualitative data suggest benefits from the program overall. Given the intervention was successful in the earlier grades it should be initiated earlier to have a greater impact. Challenges and lessons learned are offered.     

Facilitating Shared Online Group Learning between Carers,Service Users and Social Work Students

We report on our progress in developing a pilot online group teaching innovation with 14 participants, including five service users and carers, and nine social work students. Enhancing student learning by providing shared educational opportunities between students, service users, and carers can be a challenge to organise but the project demonstrates some success in doing so. The evaluation suggests that even the smallest level of service user and carer involvement can enhance student learning and personal development during social work education and training.