Being practical, being safe: doing evaluations in contested spaces

This final article summarizes and synthesizes the full special edition. This volume questioned evaluation as philosophy, enterprise, and practice from the point of view of doing evaluation under conditions of social violence, disruption, and division. In this final article, we clarify the issues and problems which challenge the professional practice of evaluation and propose guiding questions for evaluators designing and doing evaluation in worlds like these. We attend to the consequences for evaluation and evaluator of choosing to inhabit these violent worlds as an evaluator.     

Can a workbook work? Examining whether a practitioner evaluation toolkit can promote instrumental use

In large-scale, multi-site contexts, developing and disseminating practitioner-oriented evaluation toolkits are an increasingly common strategy for building evaluation capacity. Toolkits explain the evaluation process, present evaluation design choices, and offer step-by-step guidance to practitioners. To date, there has been limited research on whether such resources truly foster the successful design, implementation, and use of evaluation findings. In this paper, we describe a multi-site project in which we developed a practitioner evaluation toolkit and then studied the extent to which the toolkit and accompanying technical assistance was effective in promoting successful completion of local-level evaluations and fostering instrumental use of the findings (i.e., whether programs directly used their findings to improve practice, see Patton, 2008). Forensic nurse practitioners from six geographically dispersed service programs completed methodologically rigorous evaluations; furthermore, all six programs used the findings to create programmatic and community-level changes to improve local practice. Implications for evaluation capacity building are discussed.     

Agent-based modeling as a tool for program design and evaluation

Recently, systems thinking and systems science approaches have gained popularity in the field of evaluation; however, there has been relatively little exploration of how evaluators could use quantitative tools to assist in the implementation of systems approaches therein. The purpose of this paper is to explore potential uses of one such quantitative tool, agent-based modeling, in evaluation practice. To this end, we define agent-based modeling and offer potential uses for it in typical evaluation activities, including: engaging stakeholders, selecting an intervention, modeling program theory, setting performance targets, and interpreting evaluation results. We provide demonstrative examples from published agent-based modeling efforts both inside and outside the field of evaluation for each of the evaluative activities discussed. We further describe potential pitfalls of this tool and offer cautions for evaluators who may chose to implement it in their practice. Finally, the article concludes with a discussion of the future of agent-based modeling in evaluation practice and a call for more formal exploration of this tool as well as other approaches to simulation modeling in the field.     

Using WWC sanctioned rigorous methods to develop comparison groups for evaluation

Evaluation of program impact in the field of education has been a controversial topic over the years. Although randomized control trials have great advantages in causal inference, they often raise ethical and economic concerns in practice. As an alternative, quasi-experimental designs may provide valid evidence of influence if they are well-designed. In this article, we presented an evaluation case of a district-wide early learning improvement program. To strike a balance between practicability and academic rigor, we developed comparison groups from multiple perspectives, and used a series of tests consistent with WWC 3.0 standards to reach the most valid comparisons. Implications for evaluation practice were discussed.     

高职院校思想政治理论课实践育人体系的构建

高职院校思想政治理论课教学要真正发挥育人的作用,必须坚持理论联系实际,加强实践教学环节,探索并构建一整套实践育人的体系。文章认为构建高职院校思想政治理论课实践育人体系必须确立以素质和能力为导向的实践教学目标、完善保障机制、丰富社会实践的内容和形式、建立多维评价机制。     

Challenges and opportunities for evaluating environmental education programs

Environmental education organizations can do more to either institute evaluation or improve the quality of their evaluation. In an effort to help evaluators bridge the gap between the potential for high quality evaluation systems to improve environmental education, and the low level of evaluation in actual practice, we reviewed recent environmental education literature to reveal the challenges and opportunities for evaluating environmental education programs. The literature review identified strategies for confronting the challenges in environmental education evaluation, as well as notable opportunities for increasing the quality of evaluation in environmental education.     

高职院校顶岗实习质量评价体系的构建研究

顶岗实习是高职教育重要的实践教学环节。针对当前高职院校顶岗实习质量评价中存在的主要问题，构建了一种包含实习前期计划准备、实习中期过程控制和实习后期评价反馈的质量评价体系．该评价体系具有评价主体多元、评价对象多样、评价内容系统、注重过程控制和结果反馈的特点。     

大学生社会公益实践探讨

大学生社会公益实践是由大学生发起或参与的以利他为内容、以公共利益为目标的社会实践活动。大学生社会公益实践活动具有自愿性、社会性、利他性、学习性以及多方联动性等多种属性。应从建构教育启蒙机制、资源整合机制、项目运作机制、风险规避机制和追踪评价机制等方面着手,推动大学生社会公益实践的可持续发展。     

Teaching programme evaluation: A problem of knowledge

This article conceptualises the problem of selecting teaching content that supports the practice of programme evaluation. Knowledge for evaluation practice falls within one of three categories of knowledge that are defined by the different roles they play in supporting practice. First, core knowledge relates to the defining activity of evaluation practice, i.e., that it informs the intellectual task of the determination of a programme’s value. Second, accessory knowledge informs activities that support and facilitate the concretisation of the previous activity in a delivery context (e.g., stakeholder participation, evaluation use, project management, etc.). Third and finally, supplementary knowledge informs activities that may, on occasion, occur during evaluation practice, but without relating to the determination of value, either inherently or in a support role. The selection of knowledge for the teaching of evaluation must match the knowledge needed for the pursuit of effective evaluation practice: core, accessory, and supplementary knowledge. The specifics of these three needs ultimately depend on the characteristics of a given practice. The selection of content for the teaching of evaluation should ideally address these specific needs with the best knowledge available, regardless of its disciplinary origins.     

Reflections on facilitating a participatory community self-evaluation

This paper is a reflective essay about our experience in conducting a participatory, community self-evaluation of a neighborhood revitalization effort in Milwaukee, Wisconsin. We took a controversial approach to evaluation by focusing on learning community dialogues as a vehicle for program evaluation. As we implemented our strategy, two sets of contradictions surfaced. The first points to the fundamental differences between conventional and constructivist research paradigms. The second centers around our focus on dialogue as a strategy for evaluation. In response to these contradictions we performed three complementary roles: program evaluator, process consultant, and learning facilitator. We are not implying that we performed an outstanding job in these roles. Rather, this article is our way of taking stock of what we learned about our practice as evaluators and how we may expand our repertoire of skills in the future.     

An evaluation paradox in social work? An empirical study of evaluation use in connection with temporary programmes in Swedish social work

This article contributes to the debate on evaluation use by analysing temporary national programmes in Swedish social work. Previous empirical research shows a gloomy picture of evaluation use, thus supporting an evaluation paradox: evaluations are not used for learning and development, but are continually prioritised. The aim of this article is to study, through interviews and document analysis, how evaluations of temporary programmes in social work are designed and used. The results support the paradox and we found that it has two dimensions. First, evaluations are prioritised, but they function as a ritual, characterised by routine and unclear intentions, with limited relevance for professional learning and improvement. The second dimension is about limited use by those commissioning the evaluations, but the evaluators are using data for new purposes and in new contexts. This is, at best, a form of conceptual use, hopefully contributing indirectly to social work practice. The article ends in a discussion on how evaluations could be designed for learning and improvement by focusing on three normative models: utilisation-focused evaluation, responsive evaluation and evidence-based practice approach (EBP).     

How performance management regulations shape evaluation practice in South African municipalities

South African municipalities are taking hesitant steps towards institutionalising evaluation practice. This paper explores how evaluation is being institutionalised at a municipal level, looking at both the legislative and regulatory requirements in terms of planning, budgeting, and reporting, as well as opportunities for evaluative practice. A well-established performance management system, coupled with certain constraints to innovation and building an evaluative culture are contributing to the creation of an evaluation function that serves a monitoring purpose. A combination of desktop research and participatory action research was used to understand the municipal practice of evaluation. The results have implications for how decentralised public sector evaluation systems are understood.     

Some characteristics of moral problems in evaluation practice

In order to understand better the nature of moral problems in evaluation, an actual incident from evaluation practice is examined. Based on the incident, six characteristics of moral problems are identified and briefly discussed. The types of research needed on moral aspects of evaluation practice are presented, and ways in which the evaluation profession can aid practitioners in dealing with moral problems are outlined.     

Perspectives on Intergenerational Aspects of Aging and Diversity

Abstract

This paper considers a number of interrelated dimensions that are crucial to the developing intergenerational field: research, policy, practice and theory. The paper argues first, that research (and indeed evaluation) underlies, and is fundamental to, facilitating and understanding the linkages between practice, policy and theory. It further argues that breaking down the barriers between research, policy, practice and theoryparticularly in terms of the language we use- is key if we are to fully realize the potential of this developing field. An early intergenerational study undertaken by the author is used as a tool to illustrate some of the ways in which the field has progressed over the last twenty years. The paper concludes with thoughts and suggestions about the dilemmas and challenges remaining for those in the field.     

Evaluation capacity building (ECB) interventions and the development of sustainable evaluation practice: An exploratory study

Evaluation capacity building (ECB) is a practice that can help organizations conduct and use evaluations; however, there is little research on the sustainable impact of ECB interventions. This study provides an empirical inquiry into how ECB develops sustained evaluation practice. Interviews were conducted with 15 organizational leaders from non-profits, higher education institutions, and foundations that “bought in” to ECB and were at least six months removed from an ECB contract. The result of this work highlights how sustained evaluation practice developed over time and what these practices looked like in real-world settings. A developmental, iterative cycle for how ECB led organizations to sustain evaluation practice emerged around key components to sustainability. First, leadership supported ECB work and resources were dedicated to evaluation. Staff began to conduct and use evaluation, which led to understanding the benefits of evaluation, and promoted value and buy-in to evaluation among staff. Common barriers and emerging sustainability supports not previously identified by ECB literature—the “personal” factor and ongoing ECB practitioner contact—are described. Practical tips for ECB practitioners to promote sustainability are also detailed.     

Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

Publishing systematic, pragmatic case studies in program evaluation: rationale and introduction to the special issue

The present Special Issue proposes that the creation of a database of systematic, pragmatically focused case studies in program planning and evaluation will be useful to both theorist and practitioner. This knowledge-base can (a) facilitate cross-case comparisons for inductively deriving generalizations of best practice; (b) allow practitioners to use the experience of past cases to guide their practice in present cases; and (c) further explore the foundational nature of evaluation as a discipline. This introduction first explores why there are so few published evaluation case studies. It then describes the development of a pragmatic paradigm, including a common analytic framework for facilitating the conduct, documentation, and discussion of individual case studies and for organizing them into a database. The introduction concludes with a cross-case summary comparison of the three cases in this first part of the Special Issue. In the next issue of this journal, a second part of the Special Issue will be presented. This will include three additional case studies and an analysis of all six of the case studies in terms of what we have learned about the theoretical and practical values—as well as the challenges—of creating a database of systematic, pragmatic case studies in evaluation.     

Evaluating Dancing With Parkinson's: Reflections from the perspective of a community organization

In 2015, Dancing With Parkinson's (DWP), a Toronto-based community organization, participated in the Ontario Brain Institute's (OBI) newly launched Evaluation Support Program. This paper reflects on that experience. In particular, we identify the key lessons derived from the OBI initiative, discuss how these lessons have informed DWP practice going forward, and highlight what we consider to be the most valuable aspects of the Evaluation Support Program. While we now recognize the need to establish an evaluation culture within DWP, we find that there are significant challenges associated with both building and sustaining evaluation capacity in the context of a small community-based organization. Whereas DWP has built considerable strengths in terms of informal evaluation capacity, on its own, such capacity is insufficient to, for example, demonstrate DWP's impact to outside audiences or successfully scale up the program.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.