Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Evaluation capacity building (ECB) interventions and the development of sustainable evaluation practice: An exploratory study

Evaluation capacity building (ECB) is a practice that can help organizations conduct and use evaluations; however, there is little research on the sustainable impact of ECB interventions. This study provides an empirical inquiry into how ECB develops sustained evaluation practice. Interviews were conducted with 15 organizational leaders from non-profits, higher education institutions, and foundations that “bought in” to ECB and were at least six months removed from an ECB contract. The result of this work highlights how sustained evaluation practice developed over time and what these practices looked like in real-world settings. A developmental, iterative cycle for how ECB led organizations to sustain evaluation practice emerged around key components to sustainability. First, leadership supported ECB work and resources were dedicated to evaluation. Staff began to conduct and use evaluation, which led to understanding the benefits of evaluation, and promoted value and buy-in to evaluation among staff. Common barriers and emerging sustainability supports not previously identified by ECB literature—the “personal” factor and ongoing ECB practitioner contact—are described. Practical tips for ECB practitioners to promote sustainability are also detailed.     

Training in Depth Interviewing,William H. Banaka,New York: Harper and Row, 1971. 196 Pages, $3.95, Paperback

This paper reports on a communication assessment/development and evaluation program for directors, managers, and supervisors of a large metropolitan welfare department, with the involvement of the School of Social Work faculty at the University of Minnesota and a communications consultant, through the mechanism of a staff development project. In addition, it points out three significant future challenges oriented toward: (1) school of social work curriculum development, (2) responsibility of welfare agencies to provide training opportunities in management and communication, and (3) the support needs of governmental agencies.     

Schools must include faculty and staff in sexual violence prevention efforts

Creating a normative campus environment intolerant to sexual violence is important for prevention. While prevention initiatives focusing on students are vital, faculty and staff have a central role in supporting and sustaining a comprehensive strategy for preventing campus sexual violence. Nationwide, colleges and universities recently implemented campus climate surveys. At Emory, we decided to survey faculty and staff as well as students, motivated by our use of an ecological framing of campus sexual violence. Faculty and staff are long-term members of the community, and can provide stability and continuity that reinforces prevention efforts prioritized for students. We recommend that schools use a trauma-informed approach to guide the involvement of faculty and staff in prevention. We encourage colleges and universities to consider the experiences and needs of their faculty and staff, as professionals who serve as leaders on campus and as those who guide students through their academic experiences.     

Awareness of Gambling-Related Problems, Policies and Educational Programs Among High School and College Administrators

This brief report summarizes a survey of high school and college representatives and their awareness toward gambling-related problems. The Massachusetts Council on Compulsive Gambling developed a survey instrument to review the policies and training programs of 20 high schools and 10 colleges that were located within the catchment areas of Massachusetts Department of Public Health state-sponsored gambling treatment programs. The results revealed that there is an important discrepancy between the prevalence of gambling-related problems among young people and the awareness of these problems among educators. High school and colleges evidence a paucity of existing gambling-related regulations or policies. There is little opportunity for students and educators to learn within the school setting about gambling and its potential hazards. Without sufficient in-service education and training for faculty and staff, there is little likelihood that this group of educators can engage in the early identification or prevention efforts that are so vital to advancing the health and welfare of young people.     

Contextual influence on evaluation capacity building in a rapidly changing environment under new governmental policies

Evaluation capacity building (ECB) is a context-dependent process. Contextual factors affecting ECB implementation have been explored theoretically and practically, but their influence within a changing environment has seldom been discussed. This study examined essential context-sensitive parameters, particularly those involved in implementing new governmental policies regarding higher education. Taiwan was used as a case study for exploring the effect of contextual change on ECB attributes from the perspectives of training receivers and providers. Surveys and interviews were used for data collection and importance–performance analysis was applied for data analysis. Four prominent features were identified. First, the ECB attributes perceived as important by receivers were performed adequately, whereas those perceived as less important were performed less well. Second, under new policies, training provider designed training covering a wide range of ECB, whereas receivers focused on those can be directly applied in evaluation process. Third, in a small education system such as Taiwan’s, the complexity of peer review is high and ethical issues become important. Fourth, because the evaluation structure has been changed from single- to dual-track, receivers expect more training for institution staff, whereas providers insist on hierarchical training. Aligning ECB supply and needs is paramount for adaptation to new policies.     

Self-evaluation of assessment programs: a cross-case analysis

The goal of this article is to contribute to the validation of a self-evaluation method, which can be used by schools to evaluate the quality of their Competence Assessment Program (CAP). The outcomes of the self-evaluations of two schools are systematically compared: a novice school with little experience in competence-based education and assessment, and an innovative school with extensive experience. The self-evaluation was based on 12 quality criteria for CAPs, including both validity and reliability, and criteria stressing the importance of the formative function of assessment, such as meaningfulness and educational consequences. In each school, teachers, management and examination board participated. Results show that the two schools use different approaches to assure assessment quality. The innovative school seems to be more aware of its own strengths and weaknesses, to have a more positive attitude towards teachers, students, and educational innovations, and to explicitly involve stakeholders (i.e., teachers, students, and the work field) in their assessments. This school also had a more explicit vision of the goal of competence-based education and could design its assessments in accordance with these goals.     

The research to teaching initiative: infusing faculty research into the MSW curriculum

Social work educators have a responsibility to facilitate student utilization of research‐based knowledge. Tremendous research based knowledge is generated at many schools, but due to multiple responsibilities, faculty often find it difficult to share their findings with fellow faculty outside of their immediate project team, or to inquire into the research efforts of their colleagues within the school. This paper describes the development, implementation and preliminary evaluation of Research to Teaching, an initiative to infuse the curriculum of a school of social work with current research findings from its faculty. The goal of the Research to Teaching initiative is to take current publications from faculty, and distill them into easy‐to‐use lecture materials including slides summarizing the content, teaching notes, discussion questions, and group activities such as skill practice, simulation, and other integrative activities. These lectures are placed on a site within the school's larger website, and are accessible to all faculty, students, alumni, and other interested individuals.     

A mixed-method exploratory study of interprofessional education in social work at historically Black colleges and universities: A faculty perspective

ABSTRACT

A mixed-method approach was used to capture social work faculty experiences in integrating interprofessional education (IPE) in the social work curriculum at historically Black colleges and universities (HBCUs). Questions measured attitudes, needs, preparation, and readiness of HBCU faculty to participate in preprofessional and graduate IPE courses. Both quantitative and qualitative analyses were used. The 23 participants indicated that there was strong interest and endorsement for participation in IPE as an educational tool to improve interdisciplinary team work and social care outcomes. However, faculty had limited previous involvement with IPE courses. Many partners were identified for IPE courses with almost all endorsing alcohol and substance abuse counseling, mental health counseling, public health, early childhood education, nursing, rehabilitation counseling, school of divinity, and school of psychology. The following IPE teaching methods were endorsed by almost all of the faculty: seminars, IPE common tools, case analysis, collaborative assessment, role playing, and experiential activities. Qualitative analysis of the open-ended questions yielded five themes: designing/evaluating IPE programs, embedding IPE courses in the social work curriculum, facilitating trust among faculty, removing negative IPE stereotypes, and IPE courses sustainability. The study contributes vital information about an important group of stakeholders whose participation in IPE, heretofore not mentioned in the IPE literature, needs to be integrated. We recommend that the IPE higher education community work with HBCUs to implement IPE.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

University-assisted community school program of West Philadelphia: democratic partnerships that make a difference

The university-assisted community school model is showing results for children and youth in West Philadelphia. The University of Pennsylvania's (Penn's) Center for Community Partnerships has coordinated universitywide efforts, in partnership with the community, in order to create and develop community school programs. The Sayre program aims to become a university-assisted community school, with a comprehensive community problem-solving curriculum and communitywide program that is fully integrated across both the Sayre curriculum and the curriculum of a number of Penn's schools. The Penn-Sayre project demonstrates that higher education can be a permanent anchor for revitalizing schools and communities if the vast resources it possesses, particularly its faculty, students, and staff, are brought to bear in a coordinated fashion.     

Town–gown relationships: Exploring university–community engagement from the perspective of community members

An increasing number of colleges and universities have developed strategies for community engagement. The engagement, however, has been one-sided in that colleges and universities send students, faculty, and staff members out into the community. The current investigation sought to further broaden the definition of community engagement by exploring the benefits that can be accrued by a university when community members are encouraged to explore the cultural, intellectual, athletic, and artistic benefits that are provided in college and university campuses.     

An introduction to the Safe Schools/Healthy Students Initiative

The Safe Schools/Healthy Students (SS/HS) Initiative offers a unique opportunity to conduct large-scale, multisite, multilevel program evaluation in the context of a federal environment that places many requirements and constraints on how the grants are conducted and managed. Federal programs stress performance-based outcomes, valid and reliable data, addressing important problems, ensuring efficiency and fiscal responsibility, reducing burden on federal staff and grantees, and developing and disseminating useful solutions and recommendations. MANILA Consulting Group, Inc., (MANILA), in partnership with Battelle Centers for Public Health Research and Evaluation (Battelle) and RMC Research Corporation (RMC), has been conducting the SS/HS national cross-site evaluation, which involves the coordinated efforts of federal Project Officers, local education agencies, technical assistance providers, communication specialists, and national and local evaluators across a diverse set of socioeconomic and cultural contexts. To date, the national cross-site evaluation has provided data indicating that the SS/HS Initiative is, in fact, meeting these goals. Findings revealed that fewer students reported they had experienced violence and fewer students reported they had witnessed violence. Fully 96 percent of school staff said SS/HS had improved school safety. There was a 263 percent increase in the number of students who received school-based mental health services and a 519 percent increase in those receiving community-based mental health services. In addition, more than 80 percent of school staff reported that they saw reductions in alcohol and other drug use among their students. These encouraging results stress the need for ongoing coordination at all levels of the Initiative to continue to ensure safer schools and healthier students. This article provides an overview of the initiative and introduces four articles in this special issue.     

A Survey of 100 Community Colleges on Student Substance Use,Programming, and Collaborations

Abstract

Objective: The objective was to survey community college personnel about student substance use, and infrastructure (staff and funding), programs, and collaborations dedicated to substance use prevention. Participants: The sample included 100 administrators, faculty, and health services staff at 100 community colleges. Methods: Participants completed a Web-based survey. Results: Participants reported a number of alcohol and other drug (AOD) related concerns. Despite limited staff and funding dedicated to AOD, institutions are implementing a number of programs, although many are not implementing some of the programs popular at traditional 4-year colleges. They are also collaborating with a number of on- and off-campus groups. The availability of staff and funding dedicated to AOD, and the presence of residence halls, is associated with health programming and substance abuse collaborations. Conclusions: Results suggest that there is a need for increased research to understand the most effective AOD prevention strategies for community colleges.     

基于全面发展的高职学生评价的技术框架探讨

从高职院校学生评价现状入手,在浙江广厦建设职业技术学院实施的“驾证式”考评模式的基础上,从知识、技能、素质方面分析学生评价实施框架,以及学院实施“驾证式”考评模式后所取得的教学改革成果。     

Web presence of universities: Is higher education sending the right message online?

Web sites of the 3738 U.S.-based colleges and universities were sampled and analyzed to determine how higher education employs the internet to communicate to key publics overall and students in particular. Analyses revealed prospective donors are the most often-targeted public followed by faculty/staff. Prospective and current students followed. In better news to students, the five items students most often seek on the Web were the five most common items linked from university home pages.     

Applying utilization-focused evaluation to high school hazing prevention: A pilot intervention

Researchers argue that hazing can contribute to an abusive school climate and interfere with a positive learning environment for students. National efforts exist for establishing policies, protocols, evaluation, and education for students, administrators, and staff to prevent hazing at the college level, but this work has yet to be applied broadly in a high school context. In response to this gap, researchers implemented a pilot project at two high schools in Maine that consisted of hazing prevention training and assessment. This paper discusses the design, methods, and lessons learned through this collaborative, utilization-focused, and mixed-method training and evaluation with school personnel and high school student participants.     

Increasing college access for youth aging out of foster care: Evaluation of a summer camp program for foster youth transitioning from high school to college

Young people who transition from the foster care system face many challenges including lack of support and other educational barriers. They are less likely to graduate from high school than their counterparts and go on to college yet despite challenges, many succeed and take advantage of higher education programs.In Michigan, a state with one of the highest percentage of youth in foster care, Michigan State University developed a small scale, targeted intervention to help transitioning foster youth achieve their goals of pursuing higher education. Led by the School of Social Work in collaboration with other colleges and disciplines, it was demonstrated that a campus based learning program for transitioning foster youth can help contribute toward a perceived increase in knowledge and information about college life, funding and admissions procedures. The educational process involved peer support, role modeling, mentoring and active learning sessions led by the faculty and students who were often foster care alumni themselves. Leaders and speakers came from a range of disciplines, institutions and organizations. This approach and curriculum contributed to perceptions of the camp as enhancing life skills, self-concept, empowerment and sense of purpose. Consequently, this program contributed to the resilience of those who attended and potentially helped build steps from care to higher education.     

Correlates of drinking and drug use by higher education faculty and staff: implications for prevention

The prevalence and frequency of drug and alcohol abuse among higher education faculty and staff is not known. Higher education is a cultural environment which, on the one hand, is stressful and, on the other, permits a high degree of autonomy. This relationship of stress and autonomy is predicted to produce drug and alcohol abuse. Information is gathered by anonymous, self-report questionnaires, on the frequency and prevalence of faculty and staff drug and alcohol use at a regional university. Responses show alcohol and drug use within the last year and month among four higher education occupational groups: faculty, administrators, clerical staff, and physical plant/custodial staff. While stress was found to be weakly correlated with some drug use in the last month, depression was consistently correlated with drug use and moderately correlated with suicidal thoughts and tendencies, as well as job dissatisfaction. Drug and alcohol abuse is a factor in behavioral problems that affect faculty/staff and can be addressed through Employee Assistance Programs.