Revisiting a theory of negotiation: the utility of Markiewicz (2005) proposed six principles

People invited to participate in an evaluation process will inevitably come from a variety of personal backgrounds and hold different views based on their own lived experience. However, evaluators are in a privileged position because they have access to information from a wide range of sources and can play an important role in helping stakeholders to hear and appreciate one another's opinions and ideas. Indeed, in some cases a difference in perspective can be utilised by an evaluator to engage key stakeholders in fruitful discussion that can add value to the evaluation outcome. In other instances the evaluator finds that the task of facilitating positive interaction between multiple stakeholders is just ‘an uphill battle’ and so conflict, rather than consensus, occurs as the evaluation findings emerge and are debated.As noted by Owen [(2006) Program evaluation: Forms and approaches (3rd ed.). St. Leonards, NSW: Allen & Unwin] and other eminent evaluators before him [Fetterman, D. M. (1996). Empowerment evaluation: An introduction to theory and practice. In D. M. Fetterman, S. J. Kaftarian, & A. Wandersman (Eds.), Empowerment evaluation: Knowledge and tools for self-assessment and accountability (pp. 3–46). Thousand Oaks, CA: Sage Publications; Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage Publications; Stake, R. A. (1983). Stakeholder influence in the evaluation of cities-in-schools. New Directions for Program Evaluation, 17, 15–30], conflict in an evaluation process is not unexpected. The challenge is for evaluators to facilitate dialogue between people who hold strongly opposing views, with the aim of helping them to achieve a common understanding of the best way forward. However, this does not imply that consensus will be reached [Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage]. What is essential is that the evaluator assists the various stakeholders to recognise and accept their differences and be willing to move on.But the problem is that evaluators are not necessarily equipped with the technical or personal skills required for effective negotiation. In addition, the time and effort that are required to undertake this mediating role are often not sufficiently understood by those who commission a review. With such issues in mind Markiewicz, A. [(2005). A balancing act: Resolving multiple stakeholder interests in program evaluation. Evaluation Journal of Australasia, 4(1–2), 13–21] has proposed six principles upon which to build a case for negotiation to be integrated into the evaluation process. This paper critiques each of these principles in the context of an evaluation undertaken of a youth program. In doing so it challenges the view that stakeholder consensus is always possible if program improvement is to be achieved. This has led to some refinement and further extension of the proposed theory of negotiation that is seen to be instrumental to the role of an evaluator.     

The evaluation and monitoring of a Medicaid second surgical opinion program.

The evaluation of newly implemented and controversial programs often present real problems for the evaluator. When a program is accompanied by a legislatively mandated evaluation, the issues become even more acute. The evaluator, wanting to impact on policy, must strike a balance between the adherence to principles of scientific inquiry, yet be responsive to the needs of policymakers. This paper describes the evaluation and follow-up monitoring of a recently established second opinion program for Medicaid recipients. The program was to terminate in 17 months unless extended by the Wisconsin Legislature. The evaluation found, among other things, marked reductions in the use of surgical services after implementation of the second opinion program. Based in large part on the evaluation, the program was extended with a provision for continued monitoring. The paper discusses the factors that contributed to the evaluation having a major impact on the decision to extend the program.     

Institutionalising the evaluation function: A South African study of impartiality,use and cost

PurposeThis article explores the implications of outsourcing the evaluation function in South Africa, a context where there is a mismatch between evaluation supply and demand. It unpacks the tradeoffs between internal and external evaluation, and challenges some commonly held assumptions about both.ApproachBased on experiences as an internal evaluator, external evaluator, evaluation manager, and building evaluation capacity, the author explores how each role changes when evaluation is a scarce skill, and looks at implications outsourcing has for both the organization, and the evaluation.FindingsThe purpose of the evaluation must drive the decision to outsource. However, with changing models of collaboration, there may be hybrid options that allow organizations to build evaluation capacity.Practical implicationsOrganisations are faced with a trade-off between commissioning an evaluation, and building internal evaluation capacity. To better understand each approach, it is important to consider the purpose and context of the evaluation. This shifts some commonly held assumptions about internal and external evaluations. Re-examining these assumptions will help organizations make a more informed decision about an evaluation approach.Originality/valueThe field of evaluation is particularly concerned with evaluation use. Most of the literature on this has focused on the approach of individual evaluators, and insufficient attention has been paid to the institutional architecture of the evaluation. This article considers how some of the organisational structures around an evaluation contribute to evidence use, and the case study of South Africa also shifts the focus to the central but overlooked role of context in the debate.     

Legislative history and evaluation

Legislative histories can be done with those evaluative studies which are based on programs which have a history in the legislature. ESEA Title I, Title IV, and the Title I Technical Assistance Centers are examples of programs which have their direct source in federal legislation. The evaluator could ask whether the intent (aims and objectives) of a program is the same as the intent of the legislation or could ask whether the outcomes of a program were the same as those intended by the legislature. The purpose of this paper is to introduce the reader to the doing of legislative history in an evaluation study. The paper introduces the reader to five major sources for legislative history. The paper then discusses seven acceptable sources of information. Following this is a five-step guide to doing a legislative history.     

Prerequisites of EAP Evaluation:

This chapter's main contention is that EAP evaluators, coordinators and administrators need to spend more time in thoughtfully planning EAP evaluations as such activities directly influence the quality of any subsequent evaluation. The chapter outlines four prerequisites to be considered in this regard. These are: (1) a sincere commitment to evaluate the program, (2) answering the why, when, and how of the evaluation; (3) the goals and objectives of the EAP require specification; and (4) information should be in retrievable form. These prerequisites are discussed within the context of a cooperative planning process between the administration, the EAP coordinator and EAP evaluator(s), and are described as to how they may directly influence the EAP evaluation.     

The operation was a success but the patient died: A critique of “the implementation and evaluation of a problem-solving training program for adolescents”

This critique examines George S. Tellado's account of his evaluation of a problem-solving program for adolescents in a public junior high school in Pennsylvania. The program was apparently successful (i.e., it was shown by the evaluation to improve students' problem-solving capacities), and the evaluation was, for the most part, technically sound, but the program did not survive. This critique suggests that by examining the problem-solving program as an organizational change intervention, several problems can be uncovered regarding the program's design, implementation, evaluation, and institutionalization. In particular, it appears that the school system's staff did not have sufficient ownership of the program to understand it or to wish to carry it on after the program's originator and evaluator were no longer present. This article addresses these problems in detail and recommends techniques from the field of organizational change consultation to increase the likelihood that such programs will live on into the future.     

The LINC Project

Abstract

The Linking Intergenerational Networks in Communities (LINC) Project, a statewide, collaborative, intergenerational program, funded in 1995 by the V. Kann Rasmussen Foundation, included Clemson University, the Bureau of Senior Services for South Carolina, the South Carolina Department of Education, and the United Way of South Carolina. In this paper, the author focuses on what the partnering groups learned during the five-year project about the process of creating and implementing a collaborative statewide intergenerational program that included a public university, the state offices of aging and education, and a private fundraising and public service agency. The author, who during part of the project was the evaluator at Generations Together at the University of Pittsburgh and who consulted with the LINC project on the project evaluation from 1998 to 2000, uses the literature on collaboration to describe how the four partners worked together to form an intergenerational collaborative organization in South Carolina.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

Ethical problems in program evaluation: Advice for trapped evaluators

This paper argues that ethical problems in program evaluation are increased when conflicting or incompatible evaluation models are applied concurrently. Three models are illustrated: an Amelioration Model, generating better information for a program's own decision makers; an Accountability Model, focusing on public data disclosure and citizen participation in evaluation; and an Advocacy Model, in which the evaluation is designed to advance the program's interests in the competition for resources. Ethical problems inherent in each model, and arising from combining models are presented, along with case examples and discussion. The authors suggest several directions that program evaluators can consider to reduce their vulnerability to ethical problems.     

Ethical dilemmas in evaluating programs with family court related clients

Two ethical dilemmas are discussed; breaching client confidentiality in order to intervene in the client's illegal behavior, and defining the role of evaluator as expert witness. The evaluation context concerns a combined program development and evaluation project with clients involved in family court proceedings for child abuse or neglect. Different perspectives, approaches, and experiences in resolving each dilemma are reviewed. The conclusion argues that approaches to ethical dilemmas in evaluation projects must go beyond techniques to disguise data, to include an assessment of the values and value commitments connected to the different roles an evaluator must perform.     

Goal-oriented evaluation for program planning: A case study

Utilizing a contextual model of evaluation, a goal-oriented method was applied to the Health Psychology program, a doctoral program in its early stages at the University of California, San Francisco. There were five stages involved in implementing this method: (1) clarification of the goals and objectives of the program, (2) prioritizing the objectives, (3) judging the attainment of the objectives, (4) organization of faculty/student input, and (5) feedback to the program management. All faculty members and students were invited to participate as selfevaluators in this evaluation effort. The results indicated that there were significant differences between the faculty group and the student group on their ranking and rating of the importance of specific educational and resource objectives. A one year follow-up was obtained by interviewing the director of the program to assess the impact of the project on program planning. The advantages and disadvantages of the approach were discussed in light of this attempt to analyze a new educational program.     

Expanding evaluator competency research: Exploring competencies for program evaluation using the context of non-formal education

The overlap of competencies between general program evaluation and specific contexts or content will always be reality because evaluators may need unique competencies to answer evaluation questions for particular contexts or content areas. Limited research exists that explores the essential competencies required by professionals who use evaluation as one part of their job portfolio, which leaves unanswered questions regarding the applicability of current evaluator competency models in such settings. We used a modified three-round Delphi technique to identify evaluator competencies for non-formal educators in Cooperative Extension (CE). Our panelists identified 36 competencies in the non-formal educational programming context for CE educators that they considered important to be included in evaluation capacity building efforts. We categorized our 36 identified competencies from the Delphi study into the five competency domains proposed by the American Evaluation Association. Our findings provide information to help guide professional development among non-formal educators related to program evaluation.     

Applicability of standards for evaluations of educational programs, projects and materials in an international setting

This paper starts with a short discussion of the book Standards for Evaluations of Educational Programs, Projects and Materials by the Joint Committee on Standards for Educational Evaluation (McGraw-Hill, 1981). It then goes on to examine the usefulness in countries other than the United States of these standards and argues that evaluation of “programs” is more complicated and more politically biased than evaluation of “projects” and “materials.” In Europe, evaluation of educational programs (school systems) has proved to be highly correlated with the overriding objectives of different political parties, e.g., conservative, liberal, socialist, whereas educational policy in the United States is less bound to political parties, varying rather from case to case. In order to show how generally agreed upon educational goals can be interpreted according to different values, the author has reanalysed some studies of mean achievement in reading comprehension and science for 10- and 14-year-old students in 15 countries (IEA Project, 1973). The rank order of these countries is changed if, instead of national mean achievement, one takes proportion of students under a minimum level or above a prescribed top level as the criterion. Also the national (program) standard is evaluated in terms of between-school variance in results and in proportion of single-school units with exceptionally low or exceptionally high mean results. The conclusion is that the book on standards for evaluations of educational programs provides a good checklist of prerequisites for a reliable and valid evaluation. However, it is also concluded that—due to differences in values of program outcomes—such standards do not guarantee that the result of the evaluation will be indisputable.     

Collaborative evaluation of a high school prevention curriculum: How methods of collaborative evaluation enhanced a randomized control trial to inform program improvement

Blending high-quality and rigorous research with pure evaluation practice can often be best accomplished through thoughtful collaboration. The evaluation of a high school drug prevention program (All Stars Senior) is an example of how perceived competing purposes and methodologies can coexist to investigate formative and summative outcome variables that can be used for program improvement. Throughout this project there were many examples of client learning from evaluator and evaluator learning from client. This article presents convincing evidence that collaborative evaluation can improve the design, implementation, and findings of the randomized control trial. Throughout this paper, we discuss many examples of good science, good evaluation, and other practical benefits of practicing collaborative evaluation. Ultimately, the authors created the term pre-formative evaluation to describe the period prior to data collection and before program implementation, when collaborative evaluation can inform program improvement.     

An evaluation u-turn: From narrow organisational objectives to broad accountability

The development of evaluation scholarship progresses, but the slow and incremental steps taken are largely refinements on a direction of thinking that has become established and virtually taken for granted. Most frameworks and models which have gained recognition in the academy and practice follow programme logic methodology. They also point to the achievement of organisational objectives as being the principal, but not always sole, purpose of public relations programmes. The achievement of organisational objectives has therefore been the focus of evaluation.Parallel, to this, a second strand of thinking in the scholarship and practice has questioned the assumption that the meeting of organisational objectives is paramount and whether organisations alone should determine the measures of success. Other perspectives such as stakeholder judgements of success and the obligation of organisations to meet societal expectations have been discussed as legitimate ways in which to evaluate programmes and activities.This paper takes forward this second strand by proposing that the evaluation debate should be ‘flipped’. Instead of organisational objectives being a primary denominator for evaluation, the paper proposes that whether organisations discharge their accountabilities to stakeholders and society is of equal, if not more weight. An organisation will succeed only if it sustains its licence to operate. To achieve that it has to meet the reasonable expectations that licence-givers have and to obtain their active consent to support its actions. The key questions for public relations are therefore: who/what holds us to account? What are their expectations? How are their expectations fulfilled? Drawing on the management and public relations literature to answer these questions, the paper calls for a radical ‘turn’ from the prevailing thinking, which the authors suggest is more about public relations explaining and justifying itself, towards a model that aligns with contemporary management thinking and practice.The paper proposes a model and process that provides a way forwards based not only on organisational aspirations, but on a hierarchy of obligations: to society, stakeholders, the organisation, and to the professional function which runs campaigns. It offers a principles based evaluation approach. In doing so, it takes the evaluation debate in a new direction and thereby adds to theory-building. It also offers a new way forwards for the practice which has become increasingly mired in developing more comprehensive lists of indicators and metrics and expanding taxonomies of evaluation terms.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

A case study of action research in probation : The improved correctional field services project

This case study of the Improved Correctional Field Services Project describes the pre-implementation history of an action research project—in this case a project to improve probation services. That history encompasses a period of about five years (1974–1978). ICFS represented a major initiative in probation programming by the now defunct Law Enforcement Assistance Administration. It was intended to be a test of some of the principle ideas for making probation more effective. Its history illustrates the pitfalls that may confront a program developer and evaluator. Further, it illustrates how concepts and objectives can change within the life-time of a given project.     

Exploring Three Functions in Child Custody Evaluation for the Relocation Case: Prediction,Investigation, and Making Recommendations for a Long-Distance Parenting Plan

SUMMARY

The psycho-legal dilemmas posed by child custody relocation cases are discussed in terms of the four decisional alternatives facing the court and evaluator. Different legal contexts for relocation are reviewed in terms of their implications for the custody evaluation. Complexities involved in the evaluator's function of making predictions for the court are presented. The need to conduct careful investigation on both risk and pragmatic factors is highlighted by case illustrations. The obstacles of crafting of long distance parenting plans that will be in the best interests of the child are presented as governed by the goal of harm mitigation.