Complexity and the social world

While advocating the adoption of qualitative approaches to evaluation, it is important to reflect on the practice. This paper draws on the work of a diversionary project to involve young people at risk in leisure and sporting activities using one to one mentors. The approach adopted was one that was designed to address potentially sensitive issues and place the experience of the key players central to the enquiry. Hence, the intention was that the young people and their mentors should be treated as knowers and actors rather than the objects of the research. The approach discussed here stops short of treating them as 'partners' in the research as discussed previously in this journal, arguing instead for an approach in which skilled researchers actively engage all parties. Such an approach can offer insights inaccessible to more conventional evaluation techniques.     

Using multiple lines of evidence and independent teams to enhance the credibility of evaluations

The utilization of evaluation findings is an ongoing issue among evaluators. This article presents an approach to conducting evaluations that can be used to enhance the credibility of findings and to facilitate the production of adequate documentation in support of evaluation findings. The concepts of multiple lines of evidence and the use of independent teams are discussed. Use of this approach is expected to improve the utilization of evaluation findings.Two program evaluations that were undertaken at Canada's federal department of Consumer and Corporate Affairs are presented. To undertake the data collection and analysis, these studies were broken into separate “evaluation modules, ” each one undertaken by an independent team. The teams reported to an in-house study director, responsible for weighing the evidence and findings of each of the teams and writing the final evaluation report.     

A strategy for monitoring and evaluating massive open online courses

We argue that the complex, innovative and adaptive nature of Massive Open Online Course (MOOC) initiatives poses particular challenges to monitoring and evaluation, in that any evaluation strategy will need to follow a systems approach. This article aims to guide organizations implementing MOOCs through a series of steps to assist them in developing a strategy to monitor, improve, and judge the merit of their initiatives. We describe how we operationalise our strategy by first defining the different layers of interacting agents in a given MOOC system. We then tailor our approach to these different layers. Specifically, a two-pronged approach was developed, where we suggest that individual projects be assessed through performance monitoring; assessment criteria for which would be defined at the outset to include coverage, participation, quality and student achievement. In contrast, the success of an overall initiative should be considered within a more adaptive, emergent evaluation inquiry framework. We present the inquiry framework we developed for MOOC initiatives, and show how this framework might be used to develop evaluation questions and an assessment methodology. We also define the more fixed indicators and measures for project performance monitoring. Our strategy is described as it was developed to inform the evaluation of a MOOC initiative at the University of Cape Town (UCT), South Africa.     

Building a community-based culture of evaluation

In this article we argue for a community-based approach as a means of promoting a culture of evaluation. We do this by linking two bodies of knowledge – the 70-year theoretical tradition of community-based research and the trans-discipline of program evaluation – that are seldom intersected within the evaluation capacity building literature. We use the three hallmarks of a community-based research approach (community-determined; equitable participation; action and change) as a conceptual lens to reflect on a case example of an evaluation capacity building program led by the Ontario Brian Institute. This program involved two community-based groups (Epilepsy Southwestern Ontarioand the South West Alzheimer Society Alliance) who were supported by evaluators from the Centre for Community Based Research to conduct their own internal evaluation. The article provides an overview of a community-based research approach and its link to evaluation. It then describes the featured evaluation capacity building initiative, including reflections by the participating organizations themselves. We end by discussing lessons learned and their implications for future evaluation capacity building. Our main argument is that organizations that strive towards a community-based approach to evaluation are well placed to build and sustain a culture of evaluation.     

The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.     

Ten steps to making evaluation matter

This paper proposes ten steps to make evaluations matter. The ten steps are a combination of the usual recommended practice such as developing program theory and implementing rigorous evaluation designs with a stronger focus on more unconventional steps including developing learning frameworks, exploring pathways of evaluation influence, and assessing spread and sustainability. Consideration of these steps can lead to a focused dialogue between program planners and evaluators and can result in more rigorously planned programs. The ten steps can also help in developing and implementing evaluation designs that have greater potential for policy and programmatic influence. The paper argues that there is a need to go beyond a formulaic approach to program evaluation design that often does not address the complexity of the programs. The complexity of the program will need to inform the design of the evaluation. The ten steps that are described in this paper are heavily informed by a Realist approach to evaluation. The Realist approach attempts to understand what is it about a program that makes it work.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Matching AIDS Service Organizations' Philosophy of Service Provision with a Compatible Style of Program Evaluation

Abstract

The purpose of this paper is to outline the unique features of AIDS Service Organizations (ASOs) that should be considered when developing an evaluation strategy. It describes current evaluation approaches used by ASOs which are incongruent with the service philosophy of these organizations and the drawbacks these current evaluation efforts present. It then describes an alternative evaluation approach that better matches ASOs' philosophy of service delivery with the evaluation process. An illustrative case example is used to show how a more participatory approach can be used in an ASO and why such an approach is more likely to produce meaningful data that will be utilized for program development and refinement.     

Visionary evaluation: Approaching Aboriginal ontological equity in water management evaluation

The 2017 Traditional Owner evaluation of the implementation of the Murray-Darling Basin Plan developed an approach to evaluation that tested the use of Standpoint Theory in the field of natural resource management. This methodological choice was intended to enable First Nation approaches to data generation and use in equal measure to non-indigenous approaches. The method is implemented as a nested, up-hierarchy of scale, enabling a pan-optican dimension of vision from "below" and "above". The paper does not present the evaluative results regarding the implementation of the Plan because that information is co-owned by the participating Nations for their uses. Instead, and in respect of that arrangement, the paper presents the evaluation practices funded by the Murray-Darling Basin Authority. The methodology was negotiated and implemented with the Nations in the pilot study as a co-production across cultural boundaries. The approach was then evaluated by the participants, and these results are reported. All those reviewing the methodology were directly involved in some aspect of the evaluation, 64 % of whom identified as Traditional Owners, 67 % of whom were involved in high level decision-making about the evaluation approach. Traditional Owners rated cultural competence of the tested approach at 68 %, the benefits of the approach at 75 %, satisfaction with the standard of the evaluation at 72 %, and satisfaction with complying with the Basin Plan’s requirements for evaluation at 78 %. Recommendations for broader engagement and better science communication are made.     

Assumptions at the philosophical and programmatic levels in evaluation

Stakeholders and evaluators hold a variety of levels of assumptions at the philosophical, methodological, and programmatic levels. The use of a transformative philosophical framework is presented as a way for evaluators to become more aware of the implications of various assumptions made by themselves and program stakeholders. The argument is examined and demonstrated that evaluators who are aware of the assumptions that underlie their evaluation choices are able to provide useful support for stakeholders in the examination of the assumptions they hold with regard to the nature of the problem being addressed, the program designed to solve the problem, and the approach to evaluation that is appropriate in that context. Such an informed approach has the potential for development of more appropriate and culturally responsive programs being implemented in ways that lead to the desired impacts, as well as to lead to evaluation approaches that support effective solutions to intransigent social problems. These arguments are illustrated through examples of evaluations from multiple sectors; additional challenges are also identified.     

How and why are Theory of Change and Realist Evaluation used in food security contexts? A scoping review

The complex ways in which food security actions lead to nutrition and other health outcomes make it important to clarify what programs work and how, with theory-driven evaluation emerging as a promising approach to evaluate complex programs. However, it is unclear how and why theory-driven evaluation is applied in food security contexts. Our objective is to examine the development and use of Theory of Change and Realist Evaluation to support food security programs globally. Using a systematic search and screening process, we included studies that described a food security program, used a Theory of Change or Realist Evaluation, and presented original research or evaluations. We found a total of 59 relevant Theory of Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change arose in response to three main problems: 1) the need to evaluate under complexity; 2) challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist Evaluation was reported to be developed primarily to understand a program’s outcomes. Reflecting on the problem to be addressed in the evaluation would help improve understandings of the evaluation context, which would then inform the choice and design of an evaluation approach.     

Building evaluation capacity in Spain: a case study of rural development and empowerment in the European Union

The development of European Community administrative authority has greatly influenced the development of an evaluation culture among the southern and central member states of the European Union. The present case study from Spain provides an example of this diffusion through the use of an empowerment evaluation approach to build evaluation capacity within the context of rural development. The study focuses on the evaluation process over a 10-year period of three local development programs run by three local partnerships created in the rural areas of the Madrid region through the European LEADER initiative. Critical components of empowerment evaluation are discussed, as well as the impact, limitations, difficulties, and applicability of the approach to rural development in the European Union. Conclusions are presented concerning evaluation capacity building and the development of an evaluation culture.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

Evaluation capacity building in the nonformal education context: Challenges and strategies

Policymakers’ demand for increased accountability has compelled organizations to pay more attention to internal evaluation capacity building (ECB). The existing literature about ECB has focused on capacity building experiences and organizational research, with limited attention on challenges that internal evaluation specialists face in building organizational evaluative capacity. To address this knowledge gap, we conducted a Delphi study with evaluation specialists in the United States’ Cooperative Extension Service and developed a consensus on the most pervasive ECB challenges as well as the most useful strategies for overcoming ECB challenges. Challenges identified in this study include limited time and resources, limited understanding of the value of evaluation, evaluation considered as an afterthought, and limited support and buy-in from administrators. Alternatively, strategies found in the study include a shift in an organizational culture where evaluation is appreciated, buy-in and support from administration, clarifying the importance of quality than quantity of evaluations, and a strategic approach to ECB. The challenges identified in this study have persisted for decades, meaning administrators must understand the persistence of these issues and make an earnest investment (financial and human resource) to make noticeable progress. The Delphi approach can be used more often to prioritize ECB efforts.     

Updating the FORECAST formative evaluation approach and some implications for ameliorating theory failure,implementation failure,and evaluation failure

Historically, there has been considerable variability in how formative evaluation has been conceptualized and practiced. FORmative Evaluation Consultation And Systems Technique (FORECAST) is a formative evaluation approach that develops a set of models and processes that can be used across settings and times, while allowing for local adaptations and innovations. FORECAST integrates specific models and tools to improve limitations in program theory, implementation, and evaluation. In the period since its initial use in a federally funded community prevention project in the early 1990s, evaluators have incorporated important formative evaluation innovations into FORECAST, including the integration of feedback loops and proximal outcome evaluation. In addition, FORECAST has been applied in a randomized community research trial. In this article, we describe updates to FORECAST and the implications of FORECAST for ameliorating failures in program theory, implementation, and evaluation.     

Comparing the intended consequences of three theories of evaluation

This paper compares selected evaluation theories (Practical Participatory Evaluation, Values-Engaged Evaluation, Emergent Realist Evaluation) on the basis of their stated consequences. The discussion follows Mark's (2008) framework for research on evaluation and uses Miller's (2010) criteria of discernible impact and reproducibility in order to delineate the theories. The research on evaluation outcomes shows that some of the claims made by each evaluation theory are supported with evidence, but many substantial claims remain untested. The short term and long term goals espoused by each theory show noticeable differences in the consequences of each evaluation approach, with different emphases placed on organizational capacity, use of findings, or perceptions of the evaluation as outcomes of the prescribed evaluation approach.     

Planning and implementation of a participatory evaluation strategy: A viable approach in the evaluation of community-based participatory programs addressing cancer disparities

Community-based participatory research (CBPR) has been posited as a promising methodology to address health concerns at the community level, including cancer disparities. However, the major criticism to this approach is the lack of scientific grounded evaluation methods to assess development and implementation of this type of research. This paper describes the process of development and implementation of a participatory evaluation framework within a CBPR program to reduce breast, cervical, and colorectal cancer disparities between African Americans and whites in Alabama and Mississippi as well as lessons learned. The participatory process involved community partners and academicians in a fluid process to identify common ground activities and outcomes. The logic model, a lay friendly approach, was used as the template and clearly outlined the steps to be taken in the evaluation process without sacrificing the rigorousness of the evaluation process. We have learned three major lessons in this process: (1) the importance of constant and open dialogue among partners; (2) flexibility to make changes in the evaluation plan and implementation; and (3) importance of evaluators playing the role of facilitators between the community and academicians. Despite the challenges, we offer a viable approach to evaluation of CBPR programs focusing on cancer disparities.     

Theory-based evaluation of a comprehensive Latino education initiative: an interactive evaluation approach

Latino student access to higher education has received significant national attention in recent years. This article describes a theory-based evaluation approach used with ENLACE of Hillsborough, a 5-year project funded by the W.K. Kellogg Foundation for the purpose of increasing Latino student graduation from high school and college. Theory-based evaluation guided planning, implementation as well as evaluation through the process of developing consensus on the Latino population of focus, adoption of culturally appropriate principles and values to guide the project, and identification of strategies to reach, engage, and impact outcomes for Latino students and their families. The approach included interactive development of logic models that focused the scope of interventions and guided evaluation designs for addressing three stages of the initiative. Challenges and opportunities created by the approach are discussed, as well as ways in which the initiative impacted Latino students and collaborating educational institutions.     

A Computer-Integrated Approach to Program Evaluation

Abstract

Outcome evaluation of group work intervention in residential agencies. Development of a management/clinical information system and related computer applications.

Program evaluation as discussed in this article describes a service delivery-focused approach to program evaluation which is actually designed and implemented in a residential treatment agency for delinquent youth, the evaluation approach represents a practical strategy which applies much of the current thinking and research in the area of user focused evaluation. The approach has been refined by incorporating the product of the authors' experiential learning and pragmatic adaptations which evolved out of field testing the model over several years.

The authors present the overall approach to evaluation and the development of the evaluation model within an agency context. The author's then describe the development of a computer-based system which is designed to serve as both a management information system and a clinical information system. Basic aspects of the computer-based evaluation system are presented. Throughout the article, the authors discuss process-related issues pertaining to the introduction of evaluation to both management and staff and movement toward the implementation of an automated evaluation system.