Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts

Like artisans in a professional guild, we evaluators create tools to suit our ever evolving practice. The tools we use as evaluators are the primary artifacts of our profession, reflect our practice and embody an amalgamation of paradigms and assumptions. With the increasing shifts in evaluation purposes from judging program worth to understanding how programs work, the evaluator’s role is changing to that of facilitating stakeholders in a learning process. This involves clarifying purposes and choices, as well as unearthing critical assumptions. In such a role, evaluators become major tool-users and begin to innovate with small refinements or produce completely new tools to fit a specific challenge or context.We interrogate the form and function of 12 tools used by evaluators when working with complex evaluands and complex contexts. The form is described in terms of traditional qualitative techniques and particular characteristics of the elements, use and presentation of each tool. Then the function of each tool is analyzed with respect to articulating assumptions and affecting the agency of evaluators and stakeholders in complex contexts.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Reflections on facilitating a participatory community self-evaluation

This paper is a reflective essay about our experience in conducting a participatory, community self-evaluation of a neighborhood revitalization effort in Milwaukee, Wisconsin. We took a controversial approach to evaluation by focusing on learning community dialogues as a vehicle for program evaluation. As we implemented our strategy, two sets of contradictions surfaced. The first points to the fundamental differences between conventional and constructivist research paradigms. The second centers around our focus on dialogue as a strategy for evaluation. In response to these contradictions we performed three complementary roles: program evaluator, process consultant, and learning facilitator. We are not implying that we performed an outstanding job in these roles. Rather, this article is our way of taking stock of what we learned about our practice as evaluators and how we may expand our repertoire of skills in the future.     

Pathways to becoming an internal evaluator: Perspectives from the Australian non-government sector

There is a lack of clarity around intra-organisational evaluation roles and pathways into these roles in non-government organisations (NGOs). This article presents three auto-narratives from the authors who are working as internal evaluators in the NGO sector. We examine this phenomenon of role ambiguity by exploring our evaluation journeys and struggles to find identities in the formal evaluation community. Findings from the auto-narratives identify implications for the evaluation field regarding professionalisation. This article explores how aspects of professionalisation, such as clarification of roles and tasks of internal evaluators, could facilitate their recruitment, assess credibility and guide career trajectory. Elucidating internal evaluation career pathways contributes to the evaluation discipline by providing information relevant for evaluation capacity building, evaluator training, and the professionalisation movement.     

Aligning collaborative and culturally responsive evaluation approaches

The authors, three African-American women trained as collaborative evaluators, offer a comparative analysis of collaborative evaluation (O'Sullivan, 2004) and culturally responsive evaluation approaches (Frierson, Hood, & Hughes, 2002; Kirkhart & Hopson, 2010). Collaborative evaluation techniques immerse evaluators in the cultural milieu of the program, systematically engage stakeholders and integrate their program expertise throughout the evaluation, build evaluation capacity, and facilitate the co-creation of a more complex understanding of programs. However, the authors note that without explicit attention to considerations raised in culturally responsive evaluation approaches (for example, issues of race, power, and privilege), the voices and concerns of marginalized and underserved populations may be acknowledged, but not explicitly or adequately addressed. The intentional application of collaborative evaluation techniques coupled with a culturally responsive stance enhances the responsiveness, validity and utility of evaluations, as well as the cultural competence of evaluators.     

Revisiting a theory of negotiation: the utility of Markiewicz (2005) proposed six principles

People invited to participate in an evaluation process will inevitably come from a variety of personal backgrounds and hold different views based on their own lived experience. However, evaluators are in a privileged position because they have access to information from a wide range of sources and can play an important role in helping stakeholders to hear and appreciate one another's opinions and ideas. Indeed, in some cases a difference in perspective can be utilised by an evaluator to engage key stakeholders in fruitful discussion that can add value to the evaluation outcome. In other instances the evaluator finds that the task of facilitating positive interaction between multiple stakeholders is just ‘an uphill battle’ and so conflict, rather than consensus, occurs as the evaluation findings emerge and are debated.As noted by Owen [(2006) Program evaluation: Forms and approaches (3rd ed.). St. Leonards, NSW: Allen & Unwin] and other eminent evaluators before him [Fetterman, D. M. (1996). Empowerment evaluation: An introduction to theory and practice. In D. M. Fetterman, S. J. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 3–46). Thousand Oaks, CA: Sage Publications; Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage Publications; Stake, R. A. (1983). Stakeholder influence in the evaluation of cities-in-schools. New Directions for Program Evaluation, 17, 15–30], conflict in an evaluation process is not unexpected. The challenge is for evaluators to facilitate dialogue between people who hold strongly opposing views, with the aim of helping them to achieve a common understanding of the best way forward. However, this does not imply that consensus will be reached [Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage]. What is essential is that the evaluator assists the various stakeholders to recognise and accept their differences and be willing to move on.But the problem is that evaluators are not necessarily equipped with the technical or personal skills required for effective negotiation. In addition, the time and effort that are required to undertake this mediating role are often not sufficiently understood by those who commission a review. With such issues in mind Markiewicz, A. [(2005). A balancing act: Resolving multiple stakeholder interests in program evaluation. Evaluation Journal of Australasia, 4(1–2), 13–21] has proposed six principles upon which to build a case for negotiation to be integrated into the evaluation process. This paper critiques each of these principles in the context of an evaluation undertaken of a youth program. In doing so it challenges the view that stakeholder consensus is always possible if program improvement is to be achieved. This has led to some refinement and further extension of the proposed theory of negotiation that is seen to be instrumental to the role of an evaluator.     

Trustworthiness in evaluation practice: an emphasis on the relational

As canons for trustworthiness developed explicitly in the discourse of qualitative inquiry, the emphasis was on procedural matters rather than fundamentally relational ones. A nod was made to the relational in such strategies as “member checks” but the issues of how the evaluator actually relates to participants and to the larger communities of practice and discourse—matters subsumed under moral principles and ethical standards—were often marginalized. This chapter posits that the first consideration in designing and conducting rigorous evaluation inquiry, and in critiquing the results of any research, should be the study's trustworthiness. Judging a study's trustworthiness, however, should focus on much more than procedural matters; it should also rely on moral principles and ethical standards which specifically address how we relate to an evaluation's participants. Central to these considerations are cross-cultural sensibilities about the nuanced meanings associated with any principles and standards. The chapter provides a theoretical framework for this position and offers two illustrative examples in the form of dialogues.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

Developing a conceptual evaluation framework for gender equality interventions in research and innovation

In this article, we discuss the development of a conceptual evaluation framework to design and assess gender equality interventions and their effects in research and innovation. The conceptual framework presented herewith embraces the complexity, gender-sensitive and theory-based evaluation approaches ensuring that design and evaluation of gender equality interventions consider the complex systems that constitute the context in which the interventions operate. The evaluation framework offers a non-linear concept, where the notion of contribution - not attribution - to achieve impact is central to the integration of team, organizational and system factors in policy design and evaluation. The paper opens the “black box” to address the question of how and why a policy intervention works and in which context and discusses a systematic process on how to approach the interwoven linkages between input, implementation and effects in gender equality interventions in research and innovation, accounting for context sensitivity and methodological pluralism. The evaluation framework may serve as reference for researchers, evaluators, policymakers and other stakeholders in designing and assessing gender equality interventions, and in further developing their evidence, and theoretical and methodological base.     

Aligning collaborative and culturally responsive evaluation approaches

The authors, three African-American women trained as collaborative evaluators, offer a comparative analysis of collaborative evaluation (O'Sullivan, 2004) and culturally responsive evaluation approaches (Frierson et al., 2002, Kirkhart and Hopson, 2010). Collaborative evaluation techniques immerse evaluators in the cultural milieu of the program, systematically engage stakeholders and integrate their program expertise throughout the evaluation, build evaluation capacity, and facilitate the co-creation of a more complex understanding of programs. However, the authors note that without explicit attention to considerations raised in culturally responsive evaluation approaches (for example, issues of race, power, and privilege), the voices and concerns of marginalized and underserved populations may be acknowledged, but not explicitly or adequately addressed. The intentional application of collaborative evaluation techniques coupled with a culturally responsive stance enhances the responsiveness, validity and utility of evaluations, as well as the cultural competence of evaluators.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.     

The manifestations of politics in evaluation: An exploratory study across the evaluation process

Evaluation has been described as a political act. Programs and policies are generated from a political process, and the decision to evaluate and how to use the evaluation are manifestations of the political dynamic. This exploratory study was conducted with practicing evaluators to understand what they view as political situations in the evaluation process and how they responded to these situations. Findings suggest that, in relation to the potential evaluation phases in which each respondent has been involved, evaluations are susceptible to politics when initially attempting to identify stakeholders and when it’s time to report the evaluation findings. Evaluators have also developed multiple strategies for dealing with these situations, including finding allies for the evaluation and working to explain the evaluation process and its implications. We hope that this study will help to inform novice and expert evaluators about the various political situations they may encounter in their practice.     

Organic food and the plural moralities of food provisioning

The aim of this paper is twofold. The first aim is to unfold the moral complexity of organic food consumption as part of household food provisioning. By acknowledging this complexity, and the difficulty of determining what is ‘good’ and ‘right’ in food provisioning, the idea is to allow for a better understanding of how organic food may, or may not, fit in with the various concerns of food provisioning. The second aim is to analyse how food provisioners handle this complexity so that food provisioning can proceed as an ordinary everyday activity.The paper analyses empirical material from a study of household food provisioning in Denmark. Theoretically, it draws on French pragmatic sociology as represented by the work of Boltanski and Thévenot on moral conventions and regimes of engagement. The analysis illustrates that food provisioning involves several competing sets of moral conventions and that the status of organic food in relation to these is often uncertain and contested. However, it also identifies among provisioners different strategies for handling this moral complexity in ordinary everyday life. The paper calls for some modesty in trying to change consumer behaviour in favour of organic products. Providing consumers with more information about organic food may not make it easier to determine what is ‘good’ and ‘right’ when buying food. It may only add to the complexity of food provisioning and thus to the need for compromise and pragmatism.     

Everyday morality in families and a critique of social capital: an investigation into moral judgements,responsibilities, and sentiments in Kyrgyzstani households

This article examines individuals’ lay understandings of moral responsibilities between adult kin members. Moral sentiments and practical judgments are important in shaping kinship responsibilities. The article discusses how judgments on requests of support can be reflexive and critical, taking into account many factors, including merit, social proximity, a history of personal encounters, overlapping commitments, and moral identity in the family. In so doing, we argue that moral responsibilities are contextual and relational. We also analyze how class, gender, and capabilities affect how individuals imagine, expect and discuss care responsibilities. We also offer a critique of social capital theory of families, suggesting that their versions of morality are instrumental, alienated, and restrictive. Although Bourdieu’s concept of habitus overlaps with our proposed moral sentiments approach, the former does not adequately address moral concerns, commitments, and evaluations. The article aims to contribute to a better understanding of everyday morality by drawing upon different literatures in sociology, moral philosophy, postcommunism, and development studies.     

The priority of justice: a framework approach to ethics in program evaluation

The situations of evaluation are inevitably complex and various, often involving conflicts between ethical principles as well as among aims or stakeholders. To meet this challenge, evaluators and stakeholders need an interpretive framework to clarify the issues at hand and open the way to workable solutions. This paper provides a three-part framework of justice to guide practitioners in this interpretive task.First, evaluation is instituted to serve the public with fidelity to the values, standards, and ideals that characterize it as a profession and as an office deserving the public trust, here called public justice. Second, these requirements evoke and entail the more general demands of fair process, mutual respect, and right action known as procedural justice. Third, the operations of office and just process must serve the public good, meeting the demands of distributive justice that govern the allocation of goods and benefits.This framework takes as a starting point the priority of justice as articulated by John Rawls: “Justice is the first virtue of social institutions, as truth is to thought. A theory however elegant and economical must be rejected or revised if it is untrue; likewise laws and institutions no matter how efficient and well-arranged must be reformed or abolished if they are unjust” (1971, p. 3).     

Being practical, being safe: doing evaluations in contested spaces

This final article summarizes and synthesizes the full special edition. This volume questioned evaluation as philosophy, enterprise, and practice from the point of view of doing evaluation under conditions of social violence, disruption, and division. In this final article, we clarify the issues and problems which challenge the professional practice of evaluation and propose guiding questions for evaluators designing and doing evaluation in worlds like these. We attend to the consequences for evaluation and evaluator of choosing to inhabit these violent worlds as an evaluator.     

Charity Transgressions, Trust and Accountability

This research demonstrates how sustained charity fraud is supported when organisations do not develop strong accountability links to salient stakeholders. Whilst increased regulation is one response to reduce charity fraud and to increase organisational accountability, regulators seldom recognise the myriad heterogeneous needs of stakeholders. This research explores the tactics employed by beneficiaries and the donating public to escalate their accountability demands on such charities. By preferring the most powerful stakeholders, charities miss the opportunity to design effective processes to discharge accountability to meet their moral obligations to legitimate stakeholders. This article calls for increased ??stakeholder understanding?? by charity governors as a policy to recognise the moral rights of these stakeholders and to reduce charity transgressions.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

Evaluating Social Service Agencies: Fuzzy Pictures of Organizational Effectiveness

This article presents a case study of how members of three funding organizations evaluated the same two agencies in Canada. The research on which the article is based sheds light on the organizational effectiveness construct, on the ways in which the evaluators use it to reach conclusions on agency effectiveness, and the relation between these conclusions and funders' decisions on agency funding. The authors describe a framework for understanding evaluation processes, describe three funders in terms of this framework, set out predominant patterns in evaluation processes the funders used, and show the effects of these patterns from the agency's perspective. They then discuss the implications of the findings for agency managers and how the findings relate to theories of organization.