Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

Systems engineering: A problem-solving approach to improving program performance

Evaluation methodologies have generally emphasized the measurement and assessment of program merit, as reflected by the achievement of program objectives. Less attention has been devoted to analyzing and explaining the reasons for deficient performance and, more importantly, what needs to be modified to improve performance. While evaluators do identify performance problems, the identified problems are not always analyzed. Consequently, the information reported to decision makers may be incomplete, inconclusive, and of limited utility in the decision-making process. This article describes an evaluation methodology based on problem solving techniques which can be effective and efficient in defining and analyzing problems which impair program performance. The model can be applied in evaluations where the purpose is to provide decision makers with information and recommendations to improve program performance and provide an alternative to those evaluation models which emphasize the assessment of program merit. The model also provides a potentially unique cost/benefit methodology for estimating the potential worth of improving program performance (i.e., solving the problem).     

The inherently political nature of program evaluators and evaluation research

This paper discusses the importance of addressing the political dynamics inherent in any evaluation research process. The theoretical orientation contributing to the denial and avoidance of politics is articulated and contrasted with one which posits politics as normative and useful. A connection is drawn between stakeholder and case study approaches to evaluation. It is argued that both are effective routes to evaluations which facilitate decision making. A case example of a study of Community Development Block Grant agencies is used to detail problems accompanying avoidance of politics. Guidelines for effectively addressing and using politics are offered.     

A user-focused model for the utilization of evaluation

It is proposed that the present low level of utilization of evaluation findings is traceable in part to their failure to address directly the information needs of a clearly specified decision maker. An alternative model proceeding from such specification is proposed here, with evaluation closely interwoven with the on-going innovation process. The model suggests a number of implications for the organizational role of evalutors, for the design of evaluations, and for directions for methodological development. In particular, we suggest that evaluation designs be assessed against possible “threats to utility” as well as against the traditional “threads to validity.”     

Ethical problems in program evaluation: Advice for trapped evaluators

This paper argues that ethical problems in program evaluation are increased when conflicting or incompatible evaluation models are applied concurrently. Three models are illustrated: an Amelioration Model, generating better information for a program's own decision makers; an Accountability Model, focusing on public data disclosure and citizen participation in evaluation; and an Advocacy Model, in which the evaluation is designed to advance the program's interests in the competition for resources. Ethical problems inherent in each model, and arising from combining models are presented, along with case examples and discussion. The authors suggest several directions that program evaluators can consider to reduce their vulnerability to ethical problems.     

Pre- and post-planned evaluations are complements not alternatives: A reply to Strasser and Deniston

Strasser and Deniston's own analysis shows that post-planned evaluations are unsuitable substitutes for pre-planned evaluations. When viewed as post-experimental interviews, however, post-planned evaluations can produce valuable information which complements traditional experimental and quasi-experimental evaluations.     

Resolving conflicting demands on evaluation: A test demonstration of videotex for farmers

Evaluating an innovation for federal, state, and local policymakers and program managers alike entails conflicting demands on the evaluation study. Policymakers at federal, state, and local levels are best assisted by impact evaluations, whereas state and local program managers are best assisted by process evaluations. In-house evaluators often have an advantage in conducting process evaluations; external evaluators generally have an advantage in conducting impact evaluations. A cost-effective approach may be to combine in-house process evaluation and external impact evaluation. This dual approach was found to reduce conflicting demands on the evaluation of an experimental videotex system for agricultural producers.     

Child Custody Evaluations in Cases Involving Sexual Abuse: A View from the Bench

SUMMARY

Psychological evaluation and testimony play a critical role in child sexual abuse cases, especially in custody and visitation cases. While the evaluation generally cannot determine whether sexual abuse has occurred, it can provide useful information to assist the court in deciding matters of custody and access in the face of the allegations. Judges need to know the standard of practice for the development of expert opinion in order to be able to evaluate testimony offered by custody evaluators. The informed judge can develop useful input by outlining expectations within the order for evaluation and actively regulating the gate for admission of expert testimony. Elements of a model order for custody evaluations addressing allegations of sexual abuse are proposed.     

Comparative influence of political, cost/benefit, and statistical information on administrative decision making.

The influence of three types of evaluation information was studied in simulated decisionmaking situations within a community mental health center setting. Twelve administrators reviewed one of two decision problems (direct services or indirect services funding) and were then presented political, cost/benefit and statistical information. After each presentation, the administrators rated the importance of each type of information for decision making. Only the cost/benefit data were rated as significantly influencing their decisions. The influence of each type was not dependent upon the decision problem, nor to length of time in administrative positions. The results were interpreted in terms of “who says what, how, and to whom. ” If evaluators desire a greater impact on the decision making behavior of community health administrators, they should consider employing a pay off-based evaluation strategy and providing cost/benefit information.     

The use of metaevaluation to plan evaluations of conservation programs

The U.S. Department of Energy (DOE) is sponsoring and will be required to evaluate a wide range of conservation programs. The development of evaluation plans for such programs can often be improved by reviewing evaluations that have been done on similar programs. The Residential Conservation Service (RCS), sponsored by DOE, is discussed as an example of how conducting metaevaluations can improve evaluation planning. Many utilities have sponsored programs that are similar to the RCS. A review of the utility program evaluations suggested key methodological issues and data problems that should be considered in the development of an RCS evaluation plan. The implications of these metaevaluation findings for an RCS evaluation plan are discussed.     

Performance Evaluation in Non-Governmental Organizations (NGOs): An Analysis of Evaluation Models and their Applications in Brazil

The objective of this article is to deepen the debate surrounding the practice of evaluation as performed by Brazilian NGOs. In order to accomplish this goal, we examined the results of an investigation conducted between 2007 and 2009 that had two stages: first, a national study and second, a quantitative and qualitative study made in the state of Santa Catarina. We also considered secondary sources of data from two important studies on the national level that were conducted in 2008 and 2009 by the Fonte Institute, with partnership of the Itaú Social Foundation. The results showed that most NGOs use internal rather than external evaluation and conduct evaluations focused on projects and programs. Furthermore, they used evaluation mainly to help decision making about projects. Consequently, the majority of NGOs focused their evaluations on results and were not concerned with creating spaces for self-reflection and learning.     

Zero effects in substance abuse programs. Avoiding false positives and false negatives in the evaluation of community-based programs

The evaluation of community-based programs poses special design and analysis problems. The present article focuses on two major types of errors that can occur in such evaluations: false positive--incorrectly declaring a program to be effective--and false negatives--incorrectly declaring a program to be ineffective. The evaluation of a national demonstration of community-based programs to reduce substance abuse, Fighting Back, is used to illustrate several approaches to reduce the probability of errors. Both those errors that are affected by the design and those by analytic approaches are considered. Ways to assess multiple outcomes and to match the complexity of the program with design and analytic strategies are proposed. Community trials are complex interventions, and, although they can provide very useful information, their outcomes have to be understood in terms of the constructs they test and the contexts within which they are carried out.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.     

Using macro variables in program evaluation

This paper discusses the importance of using macro-level variables for program analysis. It is argued that three types of macro variables--cultural, economic, and demographic--provide a context for evaluation, and offer alternative explanations for the success or failure of a program. The problems of causality and availability of data are discussed. Macro variables are seen to be especially useful in "meta evaluation," and in integrating the results of specific program evaluations into general conclusions.     

Exploring the Potential of Shadow Controls in the Evaluation of Children’s Services

This article describes and presents a framework for an under‐used evaluation technique in the context of an evaluation of a programme for disaffected young people. Shadow controls—the use of expert judgement to estimate the success of a programme—are often dismissed in research design as an unreliable form of comparison, but can be useful in situations where there is limited scope for a control group or to enhance the causal inference attributable to non‐experimental evaluations. The exercise described uses a practice tool as a structure for making predictions about the situations of the young people on the programme, assuming they do not receive an intervention. These predictions (shadow controls) are then compared to outcome data for the young people at the end of the programme. The results of the exercise provide some important messages about the programme’s effectiveness and the potential for strengthening non‐experimental evaluation methods. The article also discusses how the method can usefully inform evaluations of social programmes and encourage agency and user collaboration.     

A contingent approach to strategy and tactics in formative and summative evaluations

A contingent approach to strategic and tactical choices in formative and summative evaluations is presented. Choices regarding evaluation phases and tactics were found to be dependent on evaluation goals (e.g., whether the evaluation has a formative or summative purpose) and contextual variables. A simulation was designed to pool the judgments of evaluators and users of evaluations to see how they felt evaluation strategy and tactics should change as the situation changes so as to increase the likelihood of evaluation success. Eight panels were constructed to examine eight different situations. The next step in this research is to determine whether following the pooled judgments does lead to greater evaluation success.     

Experiences of Custody Evaluation: Perspectives of Young Adults Who Were the Subject of Family Court Proceedings as Children

ABSTRACT

Court-ordered custody evaluations are conducted primarily to assist courts in making decisions regarding the best interests of children in the context of parental disputes over custody and access. They also represent the most common means through which children's wishes in relation to custody and access are presented as evidence. This article reports on a qualitative research study of young adults' recollections of participating in custody evaluations ordered by the Family Court of Australia. In-depth interviews were conducted with 18 young adults, aged 18 to 26. Two major areas of inquiry are addressed in this article: (1) participants' recollections of interviews conducted by social workers and psychologists for the purpose of custody evaluation, and (2) their suggestions for ways in which professionals working with children in this context might make the process easier for children. Taken together, their experiences point to some useful principles for practice in this field.     

The manifestations of politics in evaluation: An exploratory study across the evaluation process

Evaluation has been described as a political act. Programs and policies are generated from a political process, and the decision to evaluate and how to use the evaluation are manifestations of the political dynamic. This exploratory study was conducted with practicing evaluators to understand what they view as political situations in the evaluation process and how they responded to these situations. Findings suggest that, in relation to the potential evaluation phases in which each respondent has been involved, evaluations are susceptible to politics when initially attempting to identify stakeholders and when it’s time to report the evaluation findings. Evaluators have also developed multiple strategies for dealing with these situations, including finding allies for the evaluation and working to explain the evaluation process and its implications. We hope that this study will help to inform novice and expert evaluators about the various political situations they may encounter in their practice.     

Reflections on experiential learning in evaluation capacity building with a community organization,Dancing With Parkinson’s

This paper discusses what was learned about evaluation capacity building with community organizations who deliver services to individuals with neurological disorders. Evaluation specialists engaged by the Ontario Brain Institute Evaluation Support Program were paired with community organizations, such as Dancing With Parkinson’s. Some of the learning included: relationship building is key for this model of capacity building; community organizations often have had negative experiences with evaluation and the idea that evaluations can be friendly tools in implementing meaningful programs is one key mechanism by which such an initiative can work; community organizations often need evaluation most to be able to demonstrate their value; a strength of this initiative was that the focus was not just on creating products but mostly on developing a learning process in which capacities would remain; evaluation tools and skills that organizations found useful were developing a theory of change and the concept of heterogeneous mechanisms (informed by a realist evaluation lens).