Empowerment evaluation: An approach that has literally altered the landscape of evaluation

The quest for credible and actionable evidence to improve decision making, foster improvement, enhance self-determination, and promote social betterment is now a global phenomenon. Evaluation theorists and practitioners alike have responded to and overcome the challenges that limited the effectiveness and usefulness of traditional evaluation approaches primarily focused on seeking rigorous scientific knowledge about social programs and policies. No modern evaluation approach has received a more robust welcome from stakeholders across the globe than empowerment evaluation.Empowerment evaluation has been a leader in the development of stakeholder involvement approaches to evaluation, setting a high bar. In addition, empowerment evaluation’s respect for community knowledge and commitment to the people’s right to build their own evaluation capacity has influenced the evaluation mainstream, particularly concerning evaluation capacity building. Empowerment evaluation’s most significant contributions to the field have been to improving evaluation use and knowledge utilization.     

The manifestations of politics in evaluation: An exploratory study across the evaluation process

Evaluation has been described as a political act. Programs and policies are generated from a political process, and the decision to evaluate and how to use the evaluation are manifestations of the political dynamic. This exploratory study was conducted with practicing evaluators to understand what they view as political situations in the evaluation process and how they responded to these situations. Findings suggest that, in relation to the potential evaluation phases in which each respondent has been involved, evaluations are susceptible to politics when initially attempting to identify stakeholders and when it’s time to report the evaluation findings. Evaluators have also developed multiple strategies for dealing with these situations, including finding allies for the evaluation and working to explain the evaluation process and its implications. We hope that this study will help to inform novice and expert evaluators about the various political situations they may encounter in their practice.     

Human resource management,employee engagement,and nonprofit hospital performance

Human resource management (HRM) has been shown to impact organizational performance, but more research is needed on particular human resource (HR) practices in nonprofits and their effect on performance. In this article, we explore one HRM practice argued to influence performance, employee engagement in decision‐making, examining whether involving staff at different levels of a nonprofit affects nonprofit organizational performance. Drawing on data from a 2011 study of nonprofit hospitals, we find employee engagement has a positive influence on managerial and stakeholder perceptions of organizational performance. Employee engagement in decision‐making is also related to objective nonprofit performance measures, although the effects are less consistent. We conclude by discussing the implications of these findings for nonprofit research and practice.     

Conducting and utilizing evaluation for multiple accountabilities: A study of nonprofit evaluation capacities

This article combines the evaluation capacity building and nonprofit accountability literature to examine how nonprofits use evaluation data to address stakeholder expectations. Our study investigates how staff competency, technological resources, learning climate, and strategic planning influence a nonprofit's ability to demonstrate upward, lateral, and downward accountabilities. Results indicate different combinations of evaluation capacities matter more for particular stakeholder groups. We argue a more integrative nonprofit accountability requires that managers and staff know how to utilize evaluation results for internal and external audiences. Nuances between specific evaluative capacities and their influences on multiple accountabilities suggest several implications for practice.     

Participation in stakeholder-based evaluation: A case study

This paper reports on the evaluation of some of the elements of a multi-component program aimed at improving the accessibility of health and social services for the homeless. The program involved provincial, regional and municipal authorities and more than 20 non-profit and public agencies. In accordance with the overall approach of the program, the principle of stakeholder participation in the evaluation was emphasized by the funding agencies. However, even if the evaluation started under promising conditions, including administrative support, a strong commitment from the involved agencies and experience in formative evaluation within the research team, participation of stakeholders from charitable non-profit agencies met with many difficulties. Using a case study approach, this paper attempts to analyze and discuss how the organizational context of the evaluation, certain stakeholder characteristics and methodological features may have impacted on the evaluation process.     

The use of evaluation models for decision making: Application to the community anti-crime program

Evaluations are typically designed to provide useful information, including measures of program effectiveness. Although this information is intended to be useful, the general literature on evaluations suggests that instances in which evaluations are explicitly used for decision making are rare. This article discusses the use of decision-theoretic evaluation models, and their applicability in practice as well as in principle. A case study is presented from the evaluation of LEAA's Community Anti-Crime Program, in which a decision analysis based on a decision-theoretic evaluation model influenced a programmatic decision which arose during the conduct of the evaluation.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Corporate lobbying: Role perceptions and perceived influence on political decisions of public affairs professionals

This study aims to provide insight into public affairs professionals’ role conceptions and perceived influence on political decision making. We conducted a quantitative online survey with 238 corporate public affairs professionals in Germany who worked either “in-house” at a corporation, for an industry association, or in a public affairs consultancy firm. Based on their main stakeholder orientation (clients, political actors, or society) and primary objectives (mediator, expert, or advocate), a cluster analysis categorized the professionals into four main roles: persuaders, advisors, coordinators, and mediators. Although acting in line with ethical norms and being transparent about their practices were deemed important for all roles, there were differences regarding the balancing of positions and loyalty toward the client. Finally, a regression analysis showed that financial resources, information sovereignty, and ethical considerations were the strongest predictors of public affairs professionals’ perceived influence on political decisions. Implications and avenues for future research are discussed.     

The effects of conflict,quality, and time on small group information use and behavior in evaluative decision making situations

Graduate students simulated school board members making decisions about educational programs. Situation contexts within the program evaluation reports were varied by conflict (high, low), quality (high, low), and time (long, short). Analyses of Variance (ANOVA) of the qualitative measures m the 2 × 2 × 2 factorial design indicated significant results at the .01 level. The quality of decisions affect support for program continuation. Interactions of conflict and quality affected need for more information and more time in decision making. A conflict and time interaction affected preference to talk to other board members. Changes in situational contexts affected information need and use in small, decision making groups.     

Comparative influence of political, cost/benefit, and statistical information on administrative decision making.

The influence of three types of evaluation information was studied in simulated decisionmaking situations within a community mental health center setting. Twelve administrators reviewed one of two decision problems (direct services or indirect services funding) and were then presented political, cost/benefit and statistical information. After each presentation, the administrators rated the importance of each type of information for decision making. Only the cost/benefit data were rated as significantly influencing their decisions. The influence of each type was not dependent upon the decision problem, nor to length of time in administrative positions. The results were interpreted in terms of “who says what, how, and to whom. ” If evaluators desire a greater impact on the decision making behavior of community health administrators, they should consider employing a pay off-based evaluation strategy and providing cost/benefit information.     

Communication of results and utilization in participatory program evaluation

This paper explores the linkages between utilization and stakeholder participation in the evaluation process, specifically in the communication of results phase of evaluation. It is argued that key elements of a participatory evaluation process can be linked to meaningful and multiple forms of results utilization through cognitive and political uses of the evaluation process. Data from two small-scale participatory evaluations are used to support this argument.     

Parenting is risky business: parental risk attitudes in small stakes decisions on behalf of their children

Parents often face risk when making decisions on behalf of their children, since outcomes may affect child development. We perform an incentive-compatible field experiment using the Holt and Laury (Am Econ Rev 92(5):1644–1655, 2002) design to elicit parental risk preferences in a stewardship decision framework. Multivariate analysis using different estimation techniques suggests that parents are significantly more risk-averse when deciding for their child than for themselves. Higher risk aversion is linked to characteristics of parents, not of children. Mood and gender of the deciding parent play a key role. If these results also hold for larger stakes, insights from this study could help to improve decision environments for parents to limit inequality between children due to diverging parental risk preferences.     

Process evaluation of an environmental walking and healthy eating pilot in small rural worksites

Small Steps are Easier Together (SS) was a pilot environmental intervention in small rural worksites in Upstate New York in collaboration with Extension educators. Worksite leaders teamed with co-workers to select and implement environmental changes to increase walking steps over individual baseline and to choose healthy eating options over 10 weeks. Participants were 226 primarily white, women employees in 5 sites. A mixed methods process evaluation, conducted to identify determinants of intervention effectiveness and to explain differences in outcomes across worksites, included surveys, self-reports of walking and eating, interviews, focus groups, and an intervention log. The evaluation assessed reach, characteristics of recruited participants, dose delivered, dose received, and context and compared sites on walking and eating outcomes. Emergent elements of participant-reported dose received included: active leadership, visible environmental changes, critical mass of participants, public display of accomplishments, accountability to co-workers, and group decision making. Participants at sites with high reach and dose were significantly more likely than sites with low reach and dose to achieve intervention goals. Although this small pilot needs replication, these findings describe how these evaluation methods can be applied and analyzed in an environmental intervention and provide information on trends in the data.     

Reframing child protection: A response to a constant crisis of confidence in child protection

Child protection systems appear to be in a continual crisis of confidence. They get criticised for not doing enough to protect some children, whilst at the same time being criticised for being too intrusive or not managing demand. This constant balancing act drives almost continual reforms, none of which appear to reduce further crises of confidence. The central issue facing tertiary child protection systems stems from their function as makers of sometimes highly uncertain risk screening decisions. Uncertainty leads to errors; false positives and false negatives. Two recurring issues challenging child protection agencies are concerns about these errors. Fears about doing too much are concerns about false positives and fears of doing too little are concerns about false negatives. The need to address both issues within the context of uncertain high stakes decision making, in a highly risk intolerant environment leads to poorly formed sentinel event driven policy that in turn leads to organisational fragility. A decision outcome-based performance model based on Signal Detection Theory provides indicators that explicitly outline the link between these two strategic issues facing child protection systems. This has improved dialogue, understanding and support from sponsors. It has led an informed focus on improving decision making and stabilisation of decision thresholds. It demonstrates that Child protection systems are in fact very responsive and do perform well in their decision making (risk screening) function. Social work decision makers provide value in their decision making in spite of highly uncertain decisions to make. Child protection systems do not need reform, they need to be “reframed” to better understand true performance and so avoid poorly informed reactive policy responses to the genuine challenges that they face.     

SCOPE

This article presents an integrated framework for multi‐attribute decision making named SCOPE (System for Combined Planning and Evaluation) that was developed to assess infrastructure policy initiatives—in complex decision environments. The framework comprises scanning as well as assessment issues that are supported by a methodology of both a systemic and a systematic type. Specific use is made of operational research methods such as critical systems heuristics, scenario technique, stakeholder analysis and multi‐attribute decision making (MADM). To deal with issues of complexity and ambiguity, planning is redefined as being a systemic endeavour embedded in multi‐methodology and reflection‐in‐practice. The article addresses the purpose, background and principles of systemic planning, and exemplifies the SCOPE process with reference to the Øresund Fixed Link.     

Measuring societal effects of transdisciplinary research projects: design and application of an evaluation method

Most Transdisciplinary Research (TdR) projects combine scientific research with the building of decision making capacity for the involved stakeholders. These projects usually deal with complex, societally relevant, real-world problems. This paper focuses on TdR projects, which integrate the knowledge of researchers and stakeholders in a collaborative transdisciplinary process through structured methods of mutual learning. Previous research on the evaluation of TdR has insufficiently explored the intended effects of transdisciplinary processes on the real world (societal effects). We developed an evaluation framework for assessing the societal effects of transdisciplinary processes. Outputs (measured as procedural and product-related involvement of the stakeholders), impacts (intermediate effects connecting outputs and outcomes) and outcomes (enhanced decision making capacity) are distinguished as three types of societal effects. Our model links outputs and outcomes of transdisciplinary processes via the impacts using a mediating variables approach. We applied this model in an ex post evaluation of a transdisciplinary process. 84 out of 188 agents participated in a survey. The results show significant mediation effects of the two impacts "network building" and "transformation knowledge". These results indicate an influence of a transdisciplinary process on the decision making capacity of stakeholders, especially through social network building and the generation of knowledge relevant for action.     

The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.     

The Undeserving Rich: “Moral Values” and the White Working Class1

White working‐class citizens who vote for the Republican Party have been fodder for much political discussion and speculation recently, and a debate has arisen about the role that “moral values” played in the political decision making of this segment of voters. In this article, we defend a version of the moral values claim. We show that although the Republicans’ policies are unpopular, they are bundled with an overarching moral framework that is extremely resonant to this set of voters, and we use in‐depth interviews to uncover this framework. A key feature of this framework, on which in the 2004 presidential election George W. Bush scored high and John Kerry scored low, is the appropriate attitude to wealth, which serves as an indicator for a candidate’s general moral philosophy and as a heuristic about whether the candidate will govern with working‐class voters’ interests in mind. National Election Studies data support the argument that this was a key influence on the voting decision in 2004, even controlling for voters’ partisan identification.     

Resilience and Decision‐Making Strategies as Predictors of Career Decision Difficulties

The purpose of this study was to examine resilience and decision‐making strategies as predictors of difficulties experienced during the career decision‐making process. College students (N = 364) responded to measures of resilience, career decision‐making strategies, and career decision difficulties. Results indicated that resilience and decision‐making strategies accounted for 46% of the variance in career decision difficulties. Resilience had a greater influence on problems encountered during decision making than on problems encountered at the outset of the process. Different decision‐making strategies appeared to be related to difficulties encountered at different stages of the decision‐making process. For example, aspiration for an ideal occupation was positively associated only with lack of readiness. Procrastination was the only strategy related to all three decision difficulties: lack of readiness, lack of information, and inconsistent information. The results indicated the importance of decreasing procrastination at all stages of decision making and the need to promote resilience to deal with decision difficulties.