The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.     

Value for money: A utilization-focused approach to extending the foundation and contribution of economic evaluation

Value for Money (VfM) is an evaluative question about the merit, worth, and significance of resource use in social programs. Although VfM is a critical component of evidence-based programming, it is often overlooked or avoided by evaluators and decision-makers. A framework for evaluating VfM across the dimensions of economy, effectiveness, efficiency, and equity has emerged in response to limitations of traditional economic evaluation. This framework for assessing VfM integrates methods for engaging stakeholders in evaluative thinking to increase acceptance and utilization of evaluations that address questions of resource use. In this review, we synthesize literature on the VfM framework and position it within a broader theory of Utilization-Focused Evaluation (UFE). We then examine mechanisms through which the VfM framework may contribute to increased evaluation use. Finally, we outline avenues for future research on VfM evaluation.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.     

An evaluation paradox in social work? An empirical study of evaluation use in connection with temporary programmes in Swedish social work

This article contributes to the debate on evaluation use by analysing temporary national programmes in Swedish social work. Previous empirical research shows a gloomy picture of evaluation use, thus supporting an evaluation paradox: evaluations are not used for learning and development, but are continually prioritised. The aim of this article is to study, through interviews and document analysis, how evaluations of temporary programmes in social work are designed and used. The results support the paradox and we found that it has two dimensions. First, evaluations are prioritised, but they function as a ritual, characterised by routine and unclear intentions, with limited relevance for professional learning and improvement. The second dimension is about limited use by those commissioning the evaluations, but the evaluators are using data for new purposes and in new contexts. This is, at best, a form of conceptual use, hopefully contributing indirectly to social work practice. The article ends in a discussion on how evaluations could be designed for learning and improvement by focusing on three normative models: utilisation-focused evaluation, responsive evaluation and evidence-based practice approach (EBP).     

Theory-based evaluation in practice. What do we learn?

Theory-based evaluation (TBE) explores the how and why of program success or failure. Advocates of TBE claim that it produces information unavailable in traditional process and outcome studies. This article examines six published papers of TBEs. It finds that the authors of the papers do not always make explicit the relation of their data to the theory of the program. Nevertheless, it was evident in one or more cases that TBE identified unnecessary program components, located intermediary changes, raised new questions, contributed to a paradigm shift, highlighted the difficulties of taking successful pilot programs to scale, and provided clarity and focus for the evaluation. Interestingly, in none of the studies was the original theory completely right. Lessons for the future of TBE are drawn.     

Evaluations of Social Work Education: A Critical Review

Since the introduction in England of the social work degree in 2002, there have been a number of significant changes to social work education culminating with the implementation of the Social Work Reform Board's innovations in 2013. This article critically explores the role of evaluations of changes in social work education and their implications for social work pedagogy. Evaluations can be linked to wider trends in society such as modernity, reflexive modernity, and the audit society. These wider influences affect the use of evaluations. A way forward for social work is proposed. Social work education needs to develop a critical pedagogy of hope to transform the profession's relationship not only to evaluations but to practise as well. The key message of this article is that evaluation studies of social work education mirror the underlying tensions of late modernity. Therefore, social work education needs to adopt a critical pedagogic approach to the use of evaluation research.     

An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.     

An organizational model to distinguish between and integrate research and evaluation activities in a theory based evaluation

Theory-based evaluation (TBE) is an evaluation method that shows how a program will work under certain conditions and has been supported as a viable, evidence-based option in cases where randomized trials or high-quality quasi-experiments are not feasible. Despite the model's widely accepted theoretical appeal there are few examples of its well-implemented use, probably due to time and money limitations necessary for planning and a confusion over the definitions between research and evaluation functions and roles. In this paper, we describe the development of a theory-based evaluation design in a Math and Science Partnership (MSP) research project funded by the U.S. National Science Foundation (NSF). Through this work we developed an organizational model distinguishing between and integrating evaluation and research functions, explicating personnel roles and responsibilities, and highlighting connections between research and evaluation work. Although the research and evaluation components operated on independent budgeting, staffing, and implementation activities, we were able to combine datasets across activities to allow us to assess the integrity of the program theory, not just the hypothesized connections within it. This model has since been used for proposal development and has been invaluable as it creates a research and evaluation plan that is seamless from the beginning.     

Methodological challenges in the implementation and evaluation of social welfare policies

As social reality is quite elusive, even regarding seemingly well-recognized everyday concepts and objects, there are always methodological challenges underlying assessments and evaluations of implementation policies. The present article addresses this area of concern by presenting the results of a rereading of an empirical study of elderly home care services. Our results reveal the emergence of a dissolution of common and professional key concepts and objects in these welfare services to a degree that challenges both the implementation policy and the evaluation of policy. We claim that this has methodological implications for evaluation of implementation policies in general.     

Causal inferences on the effectiveness of complex social programs: Navigating assumptions,sources of complexity and evaluation design challenges

This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention’s effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Mentoring Policy Research Organizations: Project Evaluation Results

The international community has identified a positive link between good governance and economic development. There is an increasing appreciation of the effective role that local policy research organizations (PROs) in transition and developing countries can play in providing evidence-based policy recommendations as the basis for sound legislation and in assessing the efficacy of existing programs. Capacity building programs for PROs are seen as a key component of growth strategies. The donor community has responded with support for such efforts, which take many forms. On the other hand, the evaluation of the effectiveness of these capacity building and mentoring programs has been inadequate. This article presents the evaluation of one program, focusing on two important capacity building program outcomes: improved research capacity and greater effectiveness in the policy arena. A reflexive design is used. The results suggest the program??s approach is promising.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

The use of evaluation models for decision making: Application to the community anti-crime program

Evaluations are typically designed to provide useful information, including measures of program effectiveness. Although this information is intended to be useful, the general literature on evaluations suggests that instances in which evaluations are explicitly used for decision making are rare. This article discusses the use of decision-theoretic evaluation models, and their applicability in practice as well as in principle. A case study is presented from the evaluation of LEAA's Community Anti-Crime Program, in which a decision analysis based on a decision-theoretic evaluation model influenced a programmatic decision which arose during the conduct of the evaluation.     

Designing and implementing an evaluation of a national work support program

Welfare reforms in the 1990s have shifted governments around the world towards financial assistance conditional on work. While large-scale rigorous research on welfare-to-work programs has demonstrated effectiveness towards employment in other countries, no such micro-level evaluation of a policy has ever been conducted in Singapore. This article describes the process of developing a large experimental evaluation of the Work Support Program, which the Ministry of Community Development, Youth and Sports started in 2006. The lessons learned from planning and implementing the research can be helpful to future researchers in negotiating long-term rigorous evaluations in an environment where collaborators lack sufficient research knowledge. Insights include ways to focus on the essentials, find alternative experimental designs, collaborate effectively, and adapt instruments across cultures.     

Interviewer as instrument: accounting for human factors in evaluation research

This methodological study examines an original data collection model designed to incorporate human factors and enhance data richness in qualitative and evaluation research. Evidence supporting this model is drawn from in-depth youth and adult interviews in one of the largest policy/program evaluations undertaken in the United States, the Drug, Alcohol, and Tobacco Education evaluation (77 districts, 118 schools). When applying the explicit observation technique (EOT)--the strategic and nonjudgmental disclosure of nonverbal human factor cues by the interviewer to the respondent during interview--data revealed the observation disclosure pattern. Here, respondents linked perceptions with policy or program implementation or effectiveness evidence. Although more research is needed, it is concluded that the EOT yields richer data when compared with traditional semistructured interviews and, thus, holds promise to enhance qualitative and evaluation research methods. Validity and reliability as well as qualitative and evaluation research considerations are discussed.     

Ten steps to making evaluation matter

This paper proposes ten steps to make evaluations matter. The ten steps are a combination of the usual recommended practice such as developing program theory and implementing rigorous evaluation designs with a stronger focus on more unconventional steps including developing learning frameworks, exploring pathways of evaluation influence, and assessing spread and sustainability. Consideration of these steps can lead to a focused dialogue between program planners and evaluators and can result in more rigorously planned programs. The ten steps can also help in developing and implementing evaluation designs that have greater potential for policy and programmatic influence. The paper argues that there is a need to go beyond a formulaic approach to program evaluation design that often does not address the complexity of the programs. The complexity of the program will need to inform the design of the evaluation. The ten steps that are described in this paper are heavily informed by a Realist approach to evaluation. The Realist approach attempts to understand what is it about a program that makes it work.     

Valuation of road safety effects in cost–benefit analysis

Cost–benefit analysis is a common method for evaluating the social economic impact of transport projects, and in many of these projects the saving of human lives is an issue. This implies, within the framework of cost–benefit analysis, that a monetary value should be attached to saving human lives. This paper discusses the ‘Value of a Statistical Life’ (VoSL), a concept that is often used for monetising safety effects, in the context of road safety. Firstly, the concept of ‘willingness to pay’ for road safety and its relation to the VoSL are explained. The VoSL approach will be compared to other approaches to monetise safety effects, in particular the human capital approach and ‘quality adjusted life years’. Secondly, methods to estimate the VoSL and their applicability to road safety will be discussed. Thirdly, the paper reviews the VoSL estimates that have been found in scientific research and compares them with the values that are used in policy evaluations. Finally, a VoSL study in the Netherlands will be presented as a case study, and its applicability in policy evaluation will be illustrated.