Using hybrid models to support the development of organizational evaluation capacity: a case narrative

The ongoing need for public sector organizations to enhance their internal evaluation capacity is increasingly resulting in the use of hybrid evaluation project models, where internal evaluators work with external contracted evaluators to complete evaluative work. This paper first seeks to identify what is currently known about internal evaluation through a synthesis of the literature in this area. It then presents a case narrative illustrating how internal and external evaluation approaches may be used together to strengthen an evaluation project and to develop the evaluation capacity of the organization. Lessons learned include the need to integrate internal and external resources throughout the evaluation and to clarify expectations at the outset of the project.     

Reflections on facilitating a participatory community self-evaluation

This paper is a reflective essay about our experience in conducting a participatory, community self-evaluation of a neighborhood revitalization effort in Milwaukee, Wisconsin. We took a controversial approach to evaluation by focusing on learning community dialogues as a vehicle for program evaluation. As we implemented our strategy, two sets of contradictions surfaced. The first points to the fundamental differences between conventional and constructivist research paradigms. The second centers around our focus on dialogue as a strategy for evaluation. In response to these contradictions we performed three complementary roles: program evaluator, process consultant, and learning facilitator. We are not implying that we performed an outstanding job in these roles. Rather, this article is our way of taking stock of what we learned about our practice as evaluators and how we may expand our repertoire of skills in the future.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

Being practical, being safe: doing evaluations in contested spaces

This final article summarizes and synthesizes the full special edition. This volume questioned evaluation as philosophy, enterprise, and practice from the point of view of doing evaluation under conditions of social violence, disruption, and division. In this final article, we clarify the issues and problems which challenge the professional practice of evaluation and propose guiding questions for evaluators designing and doing evaluation in worlds like these. We attend to the consequences for evaluation and evaluator of choosing to inhabit these violent worlds as an evaluator.     

Where local and national evaluators meet: unintended threats to ethical evaluation practice

The ethical work of program evaluators is based on a covenant of honesty and transparency among stakeholders. Yet even under the most favorable evaluation conditions, threats to ethical standards exist and muddle that covenant. Unfortunately, ethical issues associated with different evaluation structures and contracting arrangements have received little attention in the evaluation research literature. This article focuses on the unintended ethical threats associated with multitiered evaluation contexts. After briefly reviewing the various frames through which evaluation theory and ethics are commonly viewed, we discuss ethical challenges associated with multitiered evaluation designs including examples drawn from our evaluation projects. The article concludes with specific findings and recommendations for evaluators, grantors, grantees, and researchers.     

Looking Forward Not Looking Back: The Evaluation of Community Living

This article outlines the development of evaluative methods within the field of mental handicap services. It suggests that evaluators have too often taken for granted the tacit assumptions underlying service models, using these to determine the criteria for evaluating success. It argues that evaluators should be more explicit about the values they adopt, and what patterns of life services should be enabling people to achieve. By more closely linking evaluation criteria to what people with a mental handicap should achieve, evaluation can become more effective in promoting positive change in service models. The article describes an approach developed by the authors for the evaluation of the 'All-Wales Strategy for the Development of Services for Mentally Handicapped People'.     

Gender and professional identity: A qualitative study of social workers practising as counsellors and psychotherapists

This article presents an analysis of the interactions between gender and class in the career pathways of social workers practising as counsellors and psychotherapists. Gender is one of the strong patterns found in the empirical data generated by a qualitative study of the professional identity of practitioners in social work. Gender was found to have a strong influence on the career choices made by men and women in social work. Women in the sample have pursued career directions that continue to have a strong clinical focus, combined with other roles such as management, supervision and training. They are more likely to express the need to balance their working lives with the needs of their families. Men are more likely to single-mindedly pursue careers in management and to express feelings of responsibility to provide for their families once children are born. However, the analysis of data also found that men were more likely to identify their family origins as working class, while women identify their family backgrounds as middle class. This patterning shows the complex interactions between gender and class in determining life outcomes. These differential pathways and work preferences need to be recognised and addressed to work towards more equitable outcomes for practitioners within social work, so that structural disadvantages on the basis of gender and class are challenged rather than replicated.     

Involving communities in the evaluation of programmes with ‘at risk’ children and young people

This article explores the relationships between evaluators and those who take part in community‐based programmes. It draws specifically on experience gained from the continuing evaluation of the Communities that Care (CTC) programme in Britain. Different community perspectives towards evaluation are discussed, highlighting the challenges these create for evaluators. In the final sections examples are used to identify different strategies used on the CTC evaluation, showing how ‘trust’, ‘boundary setting’, ‘independence’ and ‘identification of benefits’ are central to including community participants in evaluation. Copyright © 2001 John Wiley & Sons, Ltd.     

Relational responsibilities in responsive evaluation

This article explores how we can enhance our understanding of the moral responsibilities in daily, plural practices of responsive evaluation. It introduces an interpretive framework for understanding the moral aspects of evaluation practice. The framework supports responsive evaluators to better understand and handle their moral responsibilities. A case is introduced to illustrate our argument.Responsive evaluation contributes to the design and implementation of policy by working with stakeholders and coordinating the evaluation process as a relationally responsible practice. Responsive evaluation entails a democratic process in which the evaluator fosters and enters a partnership with stakeholders. The responsibilities of an evaluator generally involve issues such as ‘confidentiality’, ‘accountability’ and ‘privacy’. The responsive evaluator has specific responsibilities, for example to include stakeholders and vulnerable groups and to foster an ongoing dialogue. In addition, responsive evaluation involves a relational responsibility, which becomes present in daily situations in which stakeholders express expectations and voice demands. In our everyday work as evaluators, it is difficult to respond to all these demands at the same time. In addition, this article demonstrates that novice evaluators experience challenges concerning over- and underidenfitication with stakeholders. Guidelines and quality criteria on how to act are helpful, but need interpretation and application to the unique situation at hand.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

Privatisation as professionalisation? Attitudes,motives and achievements among Swedish social workers

The purpose of this article is to discuss the relationship between privatisation and professionalisation in social work. In social services, privatisation is often seen as a politically planned (policy-driven) process as part of liberalisation strategies for market orientation of the welfare state. However, there are several reasons to believe that there is also a process of spontaneous privatisation (sometimes driven by demand), where professionals go private and provide various types of services to local welfare authorities. In this article, our aim is to explore the extent, impact and consequences of such profession-driven privatisation and to discuss whether privatisation is a strategy for professionalisation.

This article investigates: (1) attitudes among social workers and social work students toward privatisation in general and private practice in particular; (2) the extent and types of activities being performed privately; (3) the motives whether or not to choose self-employment; (4) the differences between public and self-employed social workers in terms of professionalisation.

Data are based on surveys of 1,000 Swedish social workers and 801 social work students. The results show ambivalence among professionals toward privatisation. The authors discuss the reasons for this at an individual and collective level. Although the share of social workers in private practice in Sweden is low, between 6 and 8%, more than one third of Swedish social workers expect to be working in private practice within 10 years. The circumstances faced by self-employed social workers, who rank higher on almost every professionalisation indicator (i.e. formal education, attitude toward research, internal status, wage level, autonomy), support the hypothesis about profession-driven privatisation.     

Visual evaluation using microcomputer-based imaging technology

Microcomputer-based imaging technology reestablishes the direct link between evaluation and environmental design by allowing evaluators and urban designers to visually explore and prospectively evaluate alternative simulations of design proposals. This article describes the microcomputer hardware and software used to create simulations and how these simulations are being used in planning education at the author's university. Imaging technology is presented as a useful and promising addition to the existing repertory of evaluation techniques.     

Variables Affecting Readiness to Benefit From Career Interventions

This article identifies and briefly describes the broad range of variables that may influence clients’ readiness to benefit from career interventions. The article also discusses consequences of low readiness for effective use of career interventions and addresses implications for practice as well as for future research. Variables contributing to low readiness for effective use of career interventions include personal characteristics and circumstances; knowledge of self, options, and decision making; and prior experience with career interventions. Consequences of low readiness for using career interventions include premature disengagement, negative perception of skills and interests, selective acquisition of incomplete information, premature choice foreclosure, protracted exploration, dependent decision‐making style, and poor evaluation of options.     

Clarifying the role of belief-motive explanations in multi-stakeholder realist evaluation

In realist evaluation, where researchers aim to make program theories explicit, they can encounter competing explanations as to how programs work. Managing explanatory tensions from different sources of evidence in multi-stakeholder projects can challenge external evaluators, especially when access to pertinent data, like client records, is mediated by program stakeholders. In this article, we consider two central questions: how can program stakeholder motives shape a realist evaluation project; and how might realist evaluators respond to stakeholders’ belief-motive explanations, including those about program effectiveness, based on factors such as supererogatory commitment or trying together in good faith? Drawing on our realist evaluation of a service reform initiative involving multiple agencies, we describe stakeholder motives at key phases, highlighting a need for tactics and skills that help to manage explanatory tensions. In conclusion, the relevance of stakeholders’ belief-motive explanations (‘we believe the program works’) in realist evaluation is clarified and discussed.     

Mixed-methods library evaluation integrating the patron,library, and external perspectives: The case of Namibia regional libraries

The article presents a mixed-methods evaluation of regional libraries in Namibia, which incorporates three perspectives: the patron perspective (library users), the library perspective (library staff, management, and related officials), and the external perspective (including evaluators and monitoring data). Seven data collection methods were used: patron surveys, patron panel studies, focus group discussions, key informant and staff interviews, secondary data analysis, media analysis, and observations. The goal of the evaluation was to assess library performance for both formative and summative purposes by addressing evaluation questions on areas such as library services, use, and operations. Building upon the literature review of how mixed-methods approaches can contribute to library evaluation, the aim of this article is to show how a mixed-methods evaluation can be designed to examine multi-faceted library performance and to illustrate how the evaluation design allows information complementarity and can be utilized to present diverse viewpoints of the above three perspectives. The evaluation design, analysis process, and lessons learned from this study may be useful to evaluators engaged in evaluation of public services or programs (including public libraries) that examine multiple aspects of service performance and involve a variety of stakeholders.     

Agent-based modeling as a tool for program design and evaluation

Recently, systems thinking and systems science approaches have gained popularity in the field of evaluation; however, there has been relatively little exploration of how evaluators could use quantitative tools to assist in the implementation of systems approaches therein. The purpose of this paper is to explore potential uses of one such quantitative tool, agent-based modeling, in evaluation practice. To this end, we define agent-based modeling and offer potential uses for it in typical evaluation activities, including: engaging stakeholders, selecting an intervention, modeling program theory, setting performance targets, and interpreting evaluation results. We provide demonstrative examples from published agent-based modeling efforts both inside and outside the field of evaluation for each of the evaluative activities discussed. We further describe potential pitfalls of this tool and offer cautions for evaluators who may chose to implement it in their practice. Finally, the article concludes with a discussion of the future of agent-based modeling in evaluation practice and a call for more formal exploration of this tool as well as other approaches to simulation modeling in the field.     

Major factors influencing HIV/AIDS project evaluation

This article aimed at finding out if participatory processes (group discussions, enactments, and others) do make a valuable contribution in communication-based project implementation/evaluation and the fight against HIV/AIDS. A case study backed by documentary analysis of evaluation reports and occasional insights from interviews stood as the main methods. To identify values, the state of beneficiaries prior to and after project implementation/evaluation was compared. Participatory processes were noted to have created an enabling environment for project beneficiaries to become activists for social change, leading to the limiting of the spread of HIV/AIDS through sexual behavior and a change of attitude--the essence required for successful intervention. Group participatory processes were also noted to have contributed in overriding, to a great extent, limitations arising from sociodemographic differences in the attainment of project objectives and limitations arising from differences in forms of evaluation (internal versus external evaluators).