Case study of an evaluation coaching model: Exploring the role of the evaluator

This study examined the role of the external evaluator as a coach. More specifically, using an evaluative inquiry framework (Preskill and Torres, 1999a, Preskill and Torres, 1999b), it explored the types of coaching that an evaluator employed to promote individual, team and organizational learning. The study demonstrated that evaluation coaching provided a viable means for an organization with a limited budget to conduct evaluations through support of a coach. It also demonstrated how the coaching processes supported the development of evaluation capacity within the organization. By examining coaching models outside of the field of evaluation, this study identified two forms of coaching — results coaching and developmental coaching — that promoted evaluation capacity building and have not been previously discussed in the evaluation literature.     

Chaos and crisis: propositions for a general theory of crisis communication

This paper presents key concepts of chaos theory (CT) as a general framework for describing organizational crisis and crisis communication. Principles of predictability, sensitive dependence on initial conditions, bifurcation as system breakdown, emergent self-organization, and fractals and strange attractors as principles of organization are discussed. The implications for crisis communication as an area of practice and inquiry are explored.     

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.     

Factors behind Organizational Learning from Benchmarking: Experiences from Norwegian Municipal Benchmarking Networks

Benchmarking rests on the assumption that it supports organizationallearning and innovation, but the empirical knowledge that underpinsthis perceived means-end relationship is limited. This articledraws on existing research to develop a framework for analyzingorganizational learning outcomes from municipal benchmarking.The framework incorporates explanatory factors at differentlevels (network and municipality), and with different time perspectives(past and present). Empirical results from a nationwide Norwegianbenchmarking project indicate that municipalities do obtainorganizational learning from benchmarking but that care mustbe taken when organizational learning is conceptualized andassessed. Learning should incorporate aiding agenda settingand decision making, as well as changes. Factors such as networkand administrative characteristics and management and politicalparticipation are found to influence learning outcomes. Thereare also indications that learning from benchmarking is subjectto politics. Notably, nonsocialist political regimes are lessreceptive to organizational learning than other regimes andpolitical competition enhances organizational learning frombenchmarking.     

Strengthening child and youth programs: A look at inter-organizational mentoring strategies

Community-based non-profit organizations rarely have access to research or evaluation evidence to inform their programs and often lack the capacity to gather or use this information independently. In 2016, Wisdom2Action—a network of knowledge mobilization (KMb) experts, policy makers and service providers across Canada—launched an inter-organizational mentorship program to facilitate the implementation and sharing of best and promising practices within community-based programs for young people. This article outlines the findings from a developmental evaluation of eight mentoring relationships. Drawing on the Promoting Action on Research in Health Sciences (PARiHS) model of KMb, we look at mentoring as a type of facilitation that supports the increased use of evidence and evaluation information by non-profit organizations and identify key themes that support effective organizational mentorship in this sector. Findings reinforce the importance of establishing connected relationships and understanding context in mentoring relationships, creating adaptive and responsive work plans, ensuring consistent communication, and maintaining a focus on capacity-building if knowledge mobilization is to occur.     

Assumptions,conjectures, and other miracles: The application of evaluative thinking to theory of change models in community development

Unexamined and unjustified assumptions are the Achilles’ heel of development programs. In this paper, we describe an evaluation capacity building (ECB) approach designed to help community development practitioners work more effectively with assumptions through the intentional infusion of evaluative thinking (ET) into the program planning, monitoring, and evaluation process. We focus specifically on one component of our ET promotion approach involving the creation and analysis of theory of change (ToC) models. We describe our recent efforts to pilot this ET ECB approach with Catholic Relief Services (CRS) in Ethiopia and Zambia. The use of ToC models, plus the addition of ET, is a way to encourage individual and organizational learning and adaptive management that supports more reflective and responsive programming.     

Piloting online WellnessRx learning modules: Demonstration of developmental evaluation

WellnessRx is a health initiative focusing on healthy living through education, knowledge translation, and community engagement. Stakeholders of WellnessRx identified web-based education learning modules on nutrition and physical education as a priority to be integrated into existing health sciences curricula, as well as adapted for use by health professionals. Five learning modules were created with essential knowledge, skills, attitudes and resources or tools for health professional students and practitioners. As part of the ‘developmental evaluation framework’ for WellnessRx, two of these modules were piloted within two health professional student programs. This paper describes the pilot-evaluation experience involving student surveys, focus groups and interviews, and faculty perspectives. For both modules, student pre-post knowledge assessments indicated some improvements in post-module knowledge. Post module evaluations by students indicated benefits with the online delivery being flexible for access, self-health, case-based assessments and useful nutrition and physical activity guides. Challenges for students included their time to do the modules and the activity expectations. Instructors felt each module could be better targeted to different years within an undergraduate program. Through developmental evaluation, the pilot results along with recommendations and lessons learned provided the direction needed to further develop the WellnessRx logic model and proposed learning modules.     

Creating positive change in community organizations: A case for rediscovering Lewin

As nonprofits, especially those with a community mission, pursue developmental goals, they may benefit from various planning and analytical tools. This article explores an application of Kurt Lewin's planned change model to an assessment and revisioning process undertaken by a community‐based organization, indicating the ongoing usefulness of Lewin's work in the nonprofit sector. Using a case study approach, we examined the actions embarked on by the organization to address decreased funding and client losses. The Lewin model is overlaid to illustrate its value in structuring organizational decision making and development. The model is a three‐step process of unfreezing from old practices, moving and changing the organization into new directions based on formally identified needs, and refreezing into more effective organizational modes. When combined with other standard practices such as strategic planning, Lewin's model provides a practical and theoretically insightful tool for nonprofits wishing to refine their missions and programs and engage in successful organizational change.     

The successes and challenges of implementing individualised funding and supports for disabled people: an Irish perspective

‘Individualised funding’, which is rooted in the Independent Living Movement, has formed part of a global paradigm shift in support services for disabled people. Against the backdrop of international experience, a political system aligned with the United Kingdom and emergent critics of individualised funding, this article presents findings from an evaluation of four pilot programmes in Ireland. Exemplified by independent-skills development and community integration, these initiatives have been welcomed as a progressive development beyond traditional service provision, with perceived improvements across a range of organisational, personal, health and social care domains. The article explores the importance of ‘natural supports’ and how overly protective behaviour may unintentionally act as a barrier to full implementation. The findings also indicate that unnecessarily complex systems can lead to individual burn-out. Furthermore, a national resource allocation system working in partnership with existing social care professionals and the wider community is recommended, as is learning from overly simplified, group-based ideologies.     

Characteristics of effective summer learning programs in practice

The Center for Summer Learning examined various summer program models and found that there are nine characteristics that provide a framework for effective summer programs. In this chapter, the authors demonstrate how effective practices lead to positive results for young people. The nine characteristics of effective summer learning programs are (1) accelerating learning, (2) youth development, (3) proactive approach to summer learning, (4) leadership, (5) advanced planning, (6) staff development, (7) strategic partnerships, (8) evaluation and commitment to program improvement, and (9) sustainability and cost-effectiveness. These characteristics are divided into two sections. The first three characteristics address a program's approach to learning. Summer instructional techniques are most effective when academic learning is woven into enrichment activities and youth development. The second section covers program infrastructure to ensure the organization achieves and maintains quality programming. The nine characteristics complement each other to ensure a strong program that works to prevent summer learning loss and narrow the achievement gap. To demonstrate the variety of high-quality programs that include the nine characteristics, thirteen program profiles at the conclusion of the chapter each highlight one of the characteristics. These profiles show the various approaches that different summer programs have developed to accelerate academic achievement and promote positive development for young people in their communities.     

Evaluating the iterative development of VR/AR human factors tools for manual work

This paper outlines the approach taken to iteratively evaluate a set of VR/AR (virtual reality / augmented reality) applications for five different manual-work applications - terrestrial spacecraft assembly, assembly-line design, remote maintenance of trains, maintenance of nuclear reactors, and large-machine assembly process design - and examines the evaluation data for evidence of the effectiveness of the evaluation framework as well as the benefits to the development process of feedback from iterative evaluation. ManuVAR is an EU-funded research project that is working to develop an innovative technology platform and a framework to support high-value, high-knowledge manual work throughout the product lifecycle. The results of this study demonstrate the iterative improvements reached throughout the design cycles, observable through the trending of the quantitative results from three successive trials of the applications and the investigation of the qualitative interview findings. The paper discusses the limitations of evaluation in complex, multi-disciplinary development projects and finds evidence of the effectiveness of the use of the particular set of complementary evaluation methods incorporating a common inquiry structure used for the evaluation - particularly in facilitating triangulation of the data.     

Evaluation considerations for community-based gender-informed health interventions

Evaluations of gender-based interventions have been consistently criticized for their lack of methodological rigor. This is largely due to the complex design of many of the interventions, coupled with difficulties in measuring the outcome and impact of these interventions. This article proposes a number of ways to improve these evaluations both at the community and individual level. We recommend use of organizational theory and narrative inquiry methods, such as the appreciative inquiry technique, to examine how communities design gender-based interventions. In addition, we suggest a variety of methods to measure the effects of these interventions on gender norms in the community for example, policy analysis, multilevel modeling, and social conversations. With respect to measuring outcomes at the individual level, we argue for more rigorous evaluation designs in order to improve internal and external validity claims. Additionally, we suggest that evaluations should incorporate different methodologies, for example autobiographical narratives, which allows one to give saliency to the subjective voices of participants. Finally, we emphasize that evaluation designs need to document the long term effects of intervention programs and define the expected outcomes with greater specificity.     

Innovating Practices to Prepare Students for Graduate School: Lessons From a Social Work MOOC

This article describes the implementation of a massive open online course (MOOC) developed by a school of social work and piloted in the incoming 2016 MSW class (N=397). Using implementation science as a framework, we describe the MOOC as an intervention for preparing students for graduate school and share findings from the pilot evaluation. We argue that MOOCs can represent an innovative approach for other schools of social work, but our lessons learned suggest that they also require resources, time investment, organizational support, and understanding of the implications for students and the school. Future questions for research include examining the long-term benefits of MOOCs to students, their broader application, and the cost-benefit analysis for schools interested in developing MOOCs.     

上海社会组织发展机制和体制改革研究

上海社会组织的发展不仅需要创新模式,更需要突破传统体制机制的制约。上海需探索成立市级社会组织管理服务中心,建立社会组织管理服务的第三方评价制度;出台政府购买服务的指导目录,制定政府购买服务的办法,建立政府购买服务资金的增长机制;加大社会组织人才扶持力度,出台公益人才扶持政策,打造公益人才培训基地;制订免税扩围政策,如细化免税对象、扩大免税内容、简化免税审批流程;建立扶持社会组织发展的资金体系,如建立支持草根社会组织的扶持资金、支持企业设立公益基金、设立社会组织发展的公共资金专户;开展社会组织改革创新试点,建立国际社会组织集聚园区,开展服务于外来人口的社会组织综合创新试点,建立与国际对接的志愿者服务体系。     

Extended structures of mediation: Re-examining brokerage in dynamic networks

In this paper we revisit the concept of brokerage in social networks. We elaborate on the concept of brokerage as a process, identifying three distinct classes of brokerage behavior. Based on this process model, we develop a framework for measuring brokerage opportunities in dynamic relational data. Using data on emergent inter-organizational collaborations, we employ the dynamic brokerage framework to examine the relationship between organizational attributes and coordination in the evolving network. Comparing the findings of our process-based definition with traditional, static approaches, we identify important dimensions of organizational action that would be missed by the latter approach.     

Evidence-Based Practice in Social Work: Challenges and Opportunities for Clinicians and Organizations

Evidence-based practice (EBP) is increasingly emphasized in social work, yet effective approaches for translating research evidence into social work practice remain elusive. Despite a growing body of evidence describing effective interventions with a variety of populations, social workers continue to encounter substantial challenges with incorporating knowledge gained from these intervention studies into their routine practice with clients. This paper presents the current research outlining the known barriers and promoters to using EBP in social work clinical practice. Because social workers practice within the context of organizations, we consider the barriers that exist at both the individual and organizational levels that affect clinical social work practice. In addition to addressing the various challenges to incorporating research evidence into practice, we will also discuss a variety of emergent opportunities accompanying the move toward EBP that can be leveraged by clinicians in their social work practice with clients.     

Associating, mobilizing, politicizing: local developmental agency from without

Decades of increase in external aid programs sparked a wide range of criticisms pointing to misaligned interests, lack of accountability, and the reproduction of developmental traps. The success of development from without is more likely if it generates domestic developmental agency. In this article, we contribute by conceptualizing and measuring dimensions of developmental agency. Our research analyzes the strategic case of European Union regional development programs in Eastern Europe, where this external organization spent nearly a decade on establishing local developmental agency. We collected survey data of 1200 local organizations from two regions each in Czech Republic, Hungary, and Poland. We examine the post-accession position of organizations that participated in pre-accession assistance programs. We test a hypothesis of marginalization in the framework of recentralized developmental governance, and we examine links between patterns of pre-accession involvement and post-accession developmental agency. We identify factors that might make external developmental programs more likely to foster local developmental agency.     

Does it make a difference? Evaluation of a Canadian poverty reduction initiative

Program evaluation is an important aspect of any organization. The ability to reflect on past performance and plan for the future is essential to an organization’s health and future growth. This exploratory study is part of a larger program evaluation initiative that examined the efficacy of a regionally based organization that provided funding to community groups to alleviate poverty. The purpose of the study was to explore testimonials provided by participants of the organization’s funded programs, to determine if the participants were satisfied with the programs and the extent to which participants’ perceptions are congruent with the goals of the organization. Content analysis was used to examine 3494 testimonials from 77 different agencies. Three overall themes were determined from the analysis: 1 Developing Social Networks, 2 Learning Comes in Various Forms, and 3 Developing Self-Efficacy. Findings from the testimonials suggest that the benefits of the programs are highly social and that connections with others are important, but also provide rich opportunities for learning new skills and knowledge, as well as gaining confidence and a sense of control. The findings from this analysis support one goal of the organization which is to engage people living in poverty in meaningful ways.     

Towards an assessment for organizational participatory research health partnerships: A systematic mixed studies review with framework synthesis

Within the health sciences, organizational participatory research (OPR) is defined as a blend of research and action, in which academic researchers partner with health organization members. OPR is based on a sound partnership between all stakeholders to improve organizational practices. However, little research on the evaluation of OPR health partnership exists. This systematic mixed studies review sought to produce a new theoretical model that structures the evaluation of the OPR processes and related outcomes of OPR health partnerships. Six bibliographic databases were searched together with grey literature sources for OPR health partnership evaluation questionnaires. Six questionnaires were included, from which a pool of 95 OPR health partnership evaluation items were derived. The included questionnaires were appraised for the quality of their origin, development and measurement properties. A framework synthesis was performed using an existing OPR framework by organizing questionnaire items in a matrix using a hybrid thematic analysis. This led to our proposed Organizational Participatory Research Evaluation Model (OPREM) that includes three axes, Trust, Collective Learning and Sustainability (with specific dimensions) and 95 items. This model provides information to help stakeholders comprehensively structure the evaluation of their partnerships and subsequent improvement; thus, potentially helping to improve health organization practices.     

Teaching Successful Community Practice in Social Work

ABSTRACT

This article identifies and addresses a severe problem in the management of community practice projects of students in their second and third year in the Undergraduate Social Work program at the University of Haifa School of Social Work; namely, the current binary evaluation method based on total success or failure of the project.

Using a systems approach, this study creates a framework by which to ensure effective and positive outcomes for community practice projects carried out by undergraduate social work students (Freund, Arkin & Saltman, 1999). This framework is unique in that it draws on knowledge gained from organizational and community theories, as well as instructional theories for teaching social work practice.

The proposed framework includes three components that describe the factors necessary to help transform the student's learning experience: the organization's characteristics, the student's resources, and the evaluative measures.

This inter-conceptual approach enhances the ability of the instructor, the student, and the organization to build an appropriate organizational foundation for students operating communal programs. The underlying assumption is that an integrative approach promotes a better understanding of the components essential for building and evaluating social plans conducted by social work students.