Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Demands of the joint committee's standards for educational evaluation

The study examines demands placed on evaluators by the the Joint Committee on Educational Evaluation's standards. Twenty-two experts in evaluation rated the compatibility of implementing pairs of standards or rated individual standards on a series of scales. Multidimensional scaling of the compatibility judgments revealed five demands of ethical practice: evaluators must attend to the contextual and political circumstances of the evaluation; they must exhibit professionalism in their work; they must conduct the evaluation openly and consult widely; evaluators must meet canons of social science practice; and finally, evaluators' research activities must attend to the specific evaluation setting. The demand to attend to the contextual and political circumstances of the evaluation presents the greatest potential for conflict with other demands and is in greatest need of development.     

Strategies to Address Clinical Bias in the Child Custody Evaluation Process

ABSTRACT

Researchers have explored the existence and effects of bias on decision making for decades. Studies indicate that mental health professionals are not immune from unintentional bias in judgments, including those in forensic situations. Custody evaluators should be aware of the pitfalls that exist in clinical decision-making as well as strategies to address them. This article connects existing insights from therapeutic and forensic literature to the field of custody evaluations. An overview of clinical bias in custody evaluation is offered to familiarize practitioners with the range of possible problems. Examples, such as biases inherent in the scientific process and the clinical relationship, are followed with specific recommendations regarding how to address issues of clinical bias during evaluation. Insights are offered for judges and attorneys to improve the quality of reports they receive.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts

Like artisans in a professional guild, we evaluators create tools to suit our ever evolving practice. The tools we use as evaluators are the primary artifacts of our profession, reflect our practice and embody an amalgamation of paradigms and assumptions. With the increasing shifts in evaluation purposes from judging program worth to understanding how programs work, the evaluator’s role is changing to that of facilitating stakeholders in a learning process. This involves clarifying purposes and choices, as well as unearthing critical assumptions. In such a role, evaluators become major tool-users and begin to innovate with small refinements or produce completely new tools to fit a specific challenge or context.We interrogate the form and function of 12 tools used by evaluators when working with complex evaluands and complex contexts. The form is described in terms of traditional qualitative techniques and particular characteristics of the elements, use and presentation of each tool. Then the function of each tool is analyzed with respect to articulating assumptions and affecting the agency of evaluators and stakeholders in complex contexts.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

The development of education indicators for measuring quality in the English-speaking Caribbean: How far have we come?

Education evaluation has become increasingly important in the English-speaking Caribbean. This has been in response to assessing the progress of four regional initiatives aimed at improving the equity, efficiency, and quality of education. Both special interest groups and local evaluators have been responsible for assessing the progress of education and providing an overall synthesis and summary of what is taking place in the English-speaking Caribbean. This study employed content analysis to examine the indicators used in these education evaluation studies since the declaration of the Caribbean Plan of Action 2000–2015 to determine these indicators’ appropriateness to the Caribbean context in measuring education progress. Findings demonstrate that the English-speaking Caribbean has made strides in operationalizing quality input, process, and output indicators; however quality outcome indicators beyond test scores are yet to be realized in a systematic manner. This study also compared the types of collaborative partnerships in conducting evaluation studies used by special interest groups and local evaluators and pinpointed the one that appears most suitable for special interest groups in this region.     

Politics in evaluation: Politically responsive evaluation in high stakes environments

The role of politics has often been discussed in evaluation theory and practice. The political influence of the situation can have major effects on the evaluation design, approach and methods. Politics also has the potential to influence the decisions made from the evaluation findings. The current study focuses on the influence of the political context on stakeholder decision making. Utilizing a simulation scenario, this study compares stakeholder decision making in high and low stakes evaluation contexts. Findings suggest that high stakes political environments are more likely than low stakes environments to lead to reduced reliance on technically appropriate measures and increased dependence on measures better reflect the broader political environment.     

An evaluation paradox in social work? An empirical study of evaluation use in connection with temporary programmes in Swedish social work

This article contributes to the debate on evaluation use by analysing temporary national programmes in Swedish social work. Previous empirical research shows a gloomy picture of evaluation use, thus supporting an evaluation paradox: evaluations are not used for learning and development, but are continually prioritised. The aim of this article is to study, through interviews and document analysis, how evaluations of temporary programmes in social work are designed and used. The results support the paradox and we found that it has two dimensions. First, evaluations are prioritised, but they function as a ritual, characterised by routine and unclear intentions, with limited relevance for professional learning and improvement. The second dimension is about limited use by those commissioning the evaluations, but the evaluators are using data for new purposes and in new contexts. This is, at best, a form of conceptual use, hopefully contributing indirectly to social work practice. The article ends in a discussion on how evaluations could be designed for learning and improvement by focusing on three normative models: utilisation-focused evaluation, responsive evaluation and evidence-based practice approach (EBP).     

The evaluation of the PACA regional public health plan: reconciling the managerial, scientific and democratic finalities

For more than a decade now, evaluation has developed considerably in France, thanks in particular to the Société Fran?aise de l'évaluation, whose charter sets out a number of principles designed to guide the work of evaluators. This article examines how the evaluation process surrounding a regional public health plan (referred to as PRSP)--itself being a new instrument for regional planning in France--accords with one of these principles, which specifies that evaluation must be framed according to "a three-fold logic involving public management, democracy and scientific debate." Our analysis shows that while this evaluation was driven primarily by managerial concerns (i.e., assessing the capacity of the plan to structure health policy in a region), it also provided an Opportunity for restoring dialogue with a range of actors by opening up a space of cooperation and consultation. In addition, in order to ensure the legitimacy of the evaluation's conclusions, the knowledge produced by the evaluators had to rest on an irreproachable methodology. This example illustrates how evaluation, in the French tradition, is a process that strives to reconcile the viewpoints and expectations of managers, scientists and the general public; it is also a process that brings out lines of tension and areas of complementariness between these three logics.     

Building relationships and facilitating immigrant community integration: An evaluation of a Cultural Navigator Program

Despite the long history of immigration in the United States, communities around the country struggle to integrate newcomers into the economic, cultural, and political spheres of society. Utilizing results from the program evaluation of one public library's Cultural Navigator Program, the authors illustrate how communities and public institutions can promote integration and relationship-building between newly arrived immigrants and long-time residents. Existing social networks within receiving communities, conceptualized in this article as social capital, were leveraged to build capacity among newly arrived immigrants and foster inclusivity and integration at the community level. As a place of intervention, public libraries are suggested as a safe and shared space where community integration can be fostered. Insights derived from the evaluation inform a discussion on engaging approaches to immigrant integration. Lessons learned and recommendations for program evaluators and administrators are provided.     

Logic and evaluation theory

With this series of papers, evaluators are being called to substantiate the rationale and warrant for their own evaluative actions in ways parallel to how evaluators question the logic of program interventions, both as designed and as implemented. This endeavor is timely, appropriate, and important. In these comments, I raise modest questions about the logical constitution of an evaluation theory and about what is missing from a textual reading alone of such theory.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

Beyond resistance: exploring health managers' propensity for participatory evaluation in a developing country

The evaluation of interventions is becoming increasing common and now often seeks to involve managers in the process. Such practical participatory evaluation (PPE) aims to increase the use of evaluation results through the participation of stakeholders. This study focuses on the propensity of health managers for PPE, as measured through the components of learning, working in groups, use of judgment and use of systematic methods. We interviewed 16 health managers to determine the meaning they ascribe to these four components in their practice in a developing country, Haïti. We found that learning was often informal and that all managers attached a negative meaning to the use of judgment. Working in groups was favored by all managers, while the health managers viewed the use of systematic methods differently than do evaluators. The administrative health managers generally ranked lower in propensity for PPE than did their clinical colleagues. Implications for the practice of evaluation are discussed in relation to the work styles exhibited by managers in everyday practice, the proactive repetition of actions, the control exercised by formal procedures, and the collective versus “solitary” image of one's environment of action.