Handling problems of ecological fallacy in program planning and evaluation

The issue of ecological fallacy is highlighted in the use of social area analysis as a method of program planning and evaluation. Data on psychiatric admissions from one catchment area in New York City show misleading cross level generalization to be a real and ever present danger that planners and evaluators should not ignore. A simple method of assessing the degree and major sources of difference between higher (geographic) and lower (sociological) level relationships is illustrated. Suggestions are made regarding ways of clarifying cross level generalizations.     

The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.     

Involving communities in the evaluation of programmes with ‘at risk’ children and young people

This article explores the relationships between evaluators and those who take part in community‐based programmes. It draws specifically on experience gained from the continuing evaluation of the Communities that Care (CTC) programme in Britain. Different community perspectives towards evaluation are discussed, highlighting the challenges these create for evaluators. In the final sections examples are used to identify different strategies used on the CTC evaluation, showing how ‘trust’, ‘boundary setting’, ‘independence’ and ‘identification of benefits’ are central to including community participants in evaluation. Copyright © 2001 John Wiley & Sons, Ltd.     

The value of experiential approaches to evaluation training

The training of evaluators is an important task—important because of the good that well trained evaluators can do and because of the mischief that poorly trained evaluators can cause. There is broad agreement on the importance of producing evaluators who are methodologically versatile and organizationally savvy. To accomplish these educational ends it is necessary to supplement classroom based instruction with experiential approaches. This paper describes a theoretical model of an effective experiential educational program for training evaluators, assesses two operating programs against the requirements of the model, and reports the results of a quasi-experimental evaluation of the effectiveness of the programs in producing important educational outcomes.     

Aligning collaborative and culturally responsive evaluation approaches

The authors, three African-American women trained as collaborative evaluators, offer a comparative analysis of collaborative evaluation (O'Sullivan, 2004) and culturally responsive evaluation approaches (Frierson, Hood, & Hughes, 2002; Kirkhart & Hopson, 2010). Collaborative evaluation techniques immerse evaluators in the cultural milieu of the program, systematically engage stakeholders and integrate their program expertise throughout the evaluation, build evaluation capacity, and facilitate the co-creation of a more complex understanding of programs. However, the authors note that without explicit attention to considerations raised in culturally responsive evaluation approaches (for example, issues of race, power, and privilege), the voices and concerns of marginalized and underserved populations may be acknowledged, but not explicitly or adequately addressed. The intentional application of collaborative evaluation techniques coupled with a culturally responsive stance enhances the responsiveness, validity and utility of evaluations, as well as the cultural competence of evaluators.     

Aligning collaborative and culturally responsive evaluation approaches

The authors, three African-American women trained as collaborative evaluators, offer a comparative analysis of collaborative evaluation (O'Sullivan, 2004) and culturally responsive evaluation approaches (Frierson et al., 2002, Kirkhart and Hopson, 2010). Collaborative evaluation techniques immerse evaluators in the cultural milieu of the program, systematically engage stakeholders and integrate their program expertise throughout the evaluation, build evaluation capacity, and facilitate the co-creation of a more complex understanding of programs. However, the authors note that without explicit attention to considerations raised in culturally responsive evaluation approaches (for example, issues of race, power, and privilege), the voices and concerns of marginalized and underserved populations may be acknowledged, but not explicitly or adequately addressed. The intentional application of collaborative evaluation techniques coupled with a culturally responsive stance enhances the responsiveness, validity and utility of evaluations, as well as the cultural competence of evaluators.     

Visual evaluation using microcomputer-based imaging technology

Microcomputer-based imaging technology reestablishes the direct link between evaluation and environmental design by allowing evaluators and urban designers to visually explore and prospectively evaluate alternative simulations of design proposals. This article describes the microcomputer hardware and software used to create simulations and how these simulations are being used in planning education at the author's university. Imaging technology is presented as a useful and promising addition to the existing repertory of evaluation techniques.     

Evaluating a management and organization development effort in mental health agencies.

Although the use of management and organization development techniques in health and human service settings is increasing, there are few systematic studies of their effects. Those which do exist are problematic in many respects. An 18-month program incorporating both management and organization development is described along with the results of its evaluation, which were positive and well-utilized. A discussion of the methods employed, the problems encountered, and the outcomes obtained illustrates that these evaluation efforts are needed and can be useful to program evaluators involved in planned-change efforts.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

Resolving conflicting demands on evaluation: A test demonstration of videotex for farmers

Evaluating an innovation for federal, state, and local policymakers and program managers alike entails conflicting demands on the evaluation study. Policymakers at federal, state, and local levels are best assisted by impact evaluations, whereas state and local program managers are best assisted by process evaluations. In-house evaluators often have an advantage in conducting process evaluations; external evaluators generally have an advantage in conducting impact evaluations. A cost-effective approach may be to combine in-house process evaluation and external impact evaluation. This dual approach was found to reduce conflicting demands on the evaluation of an experimental videotex system for agricultural producers.     

Cognitive complexity of evaluator competencies

A critical aspect of evaluator education and professional learning is to educate evaluators who know the major evaluation models and learn how to manage relationships and solve complex problems when conducting, critiquing, developing and interpreting evaluations. The American and Australian Evaluation Associations have specified desired evaluator competencies, although developing a core curriculum for evaluation still seems elusive. It is suggested that these various competencies can be considered in terms of their levels of cognitive complexity.A model of cognitive complexity is utilised to explore the tasks and thinking of evaluators, leading to an important distinction between ‘knowing that’ and ‘knowing how’ in relation to evaluation tasks. As an illustration of this posited relationship, the Australian ‘Evaluators Professional Learning Competencies’ were coded according to their cognitive complexity. Two-thirds of these competencies were classed as ‘knowing that’ or surface thinking, and one third were classified as ‘knowing how’ or deeper thinking. A taxonomy is offered as a method to understand models of learning necessary for evaluator education and training, as well as for further development of professional evaluator competencies.     

Analyzing mental health evaluation: moral and ethical dimensions

Now that evaluators have been sensitized to the importance of moral and ethical issues in their work, it is time to move beyond generalities and examine the moral and ethical implications of specific evaluation models in specific settings. This paper proposes a framework that can be used to examine moral and ethical dimensions of evaluation and illustrates it by analyzing a selected model of mental health evaluation. Such a systematic identification of moral issues can improve evaluation practice both proactively, by shaping the training of evaluators, and retrospectively, by contributing to meta-evaluation.     

Mental health program evaluation: Where you start?

Seduced by the elegance, sophistication and jargon characteristic of new developments in applied research, practicing evaluators are in danger of becoming distracted from achievement of their evaluative goal: that of providing timely, reliable and useful data to program management to facilitate rational data-based decision-making. To avoid such seduction, the author advocates adaptation of simple, inexpensive and efficient evaluative methods tailored to organizational needs and presentation of findings in language readily understood by constituent groups. Likewise, insistance by evaluators upon maintenance of a strict role boundary — that is, limiting role functioning exclusively to those tasks considered evaluation — may be dysfunctional. Rather than detract from accomplishment of his goals, the author maintains that such participation allows development of skills, insights and staff relationships which enhance the evaluator's effective functioning within the organizational context and contribute to organizational acceptance and use of his evaluative results.     

Organizing evaluation to serve the needs of program planners and managers

Evaluation is often both conceptually and functionally separate from program planning and operations management. Though program managers may not see themselves as served well by the work of evaluation staffs, there are a number of potentially beneficial side effects to be realized from close collaboration between program administrators and evaluators. These flow only when the evaluation task is broadly defined and when both administrators and evaluators recognize their interdependence. Particular functional benefits to management from evaluation are described. Characteristics of interorganizational relations which might increase the probability of their attainment are then outlined and discussed.     

Assumption-aware tools and agency; an interrogation of the primary artifacts of the program evaluation and design profession in working with complex evaluands and complex contexts

Like artisans in a professional guild, we evaluators create tools to suit our ever evolving practice. The tools we use as evaluators are the primary artifacts of our profession, reflect our practice and embody an amalgamation of paradigms and assumptions. With the increasing shifts in evaluation purposes from judging program worth to understanding how programs work, the evaluator’s role is changing to that of facilitating stakeholders in a learning process. This involves clarifying purposes and choices, as well as unearthing critical assumptions. In such a role, evaluators become major tool-users and begin to innovate with small refinements or produce completely new tools to fit a specific challenge or context.We interrogate the form and function of 12 tools used by evaluators when working with complex evaluands and complex contexts. The form is described in terms of traditional qualitative techniques and particular characteristics of the elements, use and presentation of each tool. Then the function of each tool is analyzed with respect to articulating assumptions and affecting the agency of evaluators and stakeholders in complex contexts.     

Developmental trends in program evaluation

It is suggested that the ontogeny of program evaluators replicates the phytogeny of the profession of program evaluation. A trend across the purposes of program evaluation from program amelioration to program advocacy and ultimately to program accountability is projected.     

Metaphysics and professional resistance to program evaluation

One of the most common and most difficult problems confronting program evaluators is the resistance they encounter when attempting to implement evaluation efforts. The philosophical and psychological foundations of this resistance are explored. Problems in the evaluative paradigm are presented. Value differences between evaluators and practitioners are highlighted. Emotional bases for practitioner behavior are clarified and types and loci of resistance are outlined. Suggestions for coping strategies are discussed.     

Pathways to becoming an internal evaluator: Perspectives from the Australian non-government sector

There is a lack of clarity around intra-organisational evaluation roles and pathways into these roles in non-government organisations (NGOs). This article presents three auto-narratives from the authors who are working as internal evaluators in the NGO sector. We examine this phenomenon of role ambiguity by exploring our evaluation journeys and struggles to find identities in the formal evaluation community. Findings from the auto-narratives identify implications for the evaluation field regarding professionalisation. This article explores how aspects of professionalisation, such as clarification of roles and tasks of internal evaluators, could facilitate their recruitment, assess credibility and guide career trajectory. Elucidating internal evaluation career pathways contributes to the evaluation discipline by providing information relevant for evaluation capacity building, evaluator training, and the professionalisation movement.     

The manifestations of politics in evaluation: An exploratory study across the evaluation process

Evaluation has been described as a political act. Programs and policies are generated from a political process, and the decision to evaluate and how to use the evaluation are manifestations of the political dynamic. This exploratory study was conducted with practicing evaluators to understand what they view as political situations in the evaluation process and how they responded to these situations. Findings suggest that, in relation to the potential evaluation phases in which each respondent has been involved, evaluations are susceptible to politics when initially attempting to identify stakeholders and when it’s time to report the evaluation findings. Evaluators have also developed multiple strategies for dealing with these situations, including finding allies for the evaluation and working to explain the evaluation process and its implications. We hope that this study will help to inform novice and expert evaluators about the various political situations they may encounter in their practice.     

Looking Forward Not Looking Back: The Evaluation of Community Living

This article outlines the development of evaluative methods within the field of mental handicap services. It suggests that evaluators have too often taken for granted the tacit assumptions underlying service models, using these to determine the criteria for evaluating success. It argues that evaluators should be more explicit about the values they adopt, and what patterns of life services should be enabling people to achieve. By more closely linking evaluation criteria to what people with a mental handicap should achieve, evaluation can become more effective in promoting positive change in service models. The article describes an approach developed by the authors for the evaluation of the 'All-Wales Strategy for the Development of Services for Mentally Handicapped People'.