Teaching graduate-level evaluation sociology through a departmental evaluation

The Department of Sociology at the University of Texas at Austin conducted an internal evaluation of its graduate program through an applied methodology graduate course. This article reports on the effectiveness of the format for teaching research methods, the utilization of the evaluation's findings, and the conditions that need to occur for a similar study to take place. Through the course, graduate students gained experience using multiple methods in evaluation research as well as acquiring an inside perspective on the department and the profession. Although some problems of conducting an internal evaluation are discussed, we found that with departmental support the evaluation was an effective method of promoting internal discourse and change.     

Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches

Collaborative Evaluation systematically invites and engages stakeholders in program evaluation planning and implementation. Unlike "distanced" evaluation approaches, which reject stakeholder participation as evaluation team members, Collaborative Evaluation assumes that active, on-going engagement between evaluators and program staff, result in stronger evaluation designs, enhanced data collection and analysis, and results that stakeholder understand and use. Among similar "participant-oriented" evaluation approaches (Fitzpatrick, Sanders, & Worthen, 2011), Collaborative Evaluation distinguishes itself in that it uses a sliding scale for levels of collaboration. This means that different program evaluations will experience different levels of collaborative activity. The sliding scale is applied as the evaluator considers each program's evaluation needs, readiness, and resources. While Collaborative Evaluation is a term widely used in evaluation, its meaning varies considerably. Often used interchangeably with participatory and/or empowerment evaluation, the terms can be used to mean different things, which can be confusing. The articles use a comparative Collaborative Evaluation Framework to highlight how from a theoretical perspective, Collaborative Evaluation distinguishes itself from the other participatory evaluation approaches.     

Priority setting for evaluation: Developing a strategic evaluation portfolio

Resources for evaluation are frequently scarce and best use should be made of them to deliver against the typical purposes of an evaluation function to (i) enhance accountability and (ii) promote operational improvement and learning. This paper presents a method for analyzing and prioritizing potential evaluations to improve the selection of a portfolio of activities that give the greatest pay-off. The method establishes the relative priority of ‘evaluation opportunities’ against criteria that relate to the usual primary purposes of an evaluation function. The method was developed in the context of a multilateral organization but is of general utility to the wider evaluation community and, with suitable adaptation can be applied to help ensure that scarce evaluation resources are used to their best advantage.A range of benefits are expected to accrue to an organization from adopting a more thorough, analytical priority setting process. These include:

• •Enhancing the relevance of evaluations to the wider organizational agenda
• •Providing structured justification for allocating funds to evaluation activities
• •More transparent, predictable decision-making
• •Enabling realignment of evaluation priorities with changing needs and circumstances
• •Improved credibility with stakeholders
• •Providing a clear direction and sense of purpose to evaluation staff
• •Documenting a clear framework for the development of an organization's evaluation portfolio.

The paper describes a priority setting method, including the key criteria that are used to assess ‘evaluation opportunities’, and presents different analyses of an evaluation portfolio. Examples from a practical application of the approach to the preparation of an evaluation work plan in a multilateral environmental agency are given.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

Visionary evaluation: Approaching Aboriginal ontological equity in water management evaluation

The 2017 Traditional Owner evaluation of the implementation of the Murray-Darling Basin Plan developed an approach to evaluation that tested the use of Standpoint Theory in the field of natural resource management. This methodological choice was intended to enable First Nation approaches to data generation and use in equal measure to non-indigenous approaches. The method is implemented as a nested, up-hierarchy of scale, enabling a pan-optican dimension of vision from "below" and "above". The paper does not present the evaluative results regarding the implementation of the Plan because that information is co-owned by the participating Nations for their uses. Instead, and in respect of that arrangement, the paper presents the evaluation practices funded by the Murray-Darling Basin Authority. The methodology was negotiated and implemented with the Nations in the pilot study as a co-production across cultural boundaries. The approach was then evaluated by the participants, and these results are reported. All those reviewing the methodology were directly involved in some aspect of the evaluation, 64 % of whom identified as Traditional Owners, 67 % of whom were involved in high level decision-making about the evaluation approach. Traditional Owners rated cultural competence of the tested approach at 68 %, the benefits of the approach at 75 %, satisfaction with the standard of the evaluation at 72 %, and satisfaction with complying with the Basin Plan’s requirements for evaluation at 78 %. Recommendations for broader engagement and better science communication are made.     

Politics in evaluation: Politically responsive evaluation in high stakes environments

The role of politics has often been discussed in evaluation theory and practice. The political influence of the situation can have major effects on the evaluation design, approach and methods. Politics also has the potential to influence the decisions made from the evaluation findings. The current study focuses on the influence of the political context on stakeholder decision making. Utilizing a simulation scenario, this study compares stakeholder decision making in high and low stakes evaluation contexts. Findings suggest that high stakes political environments are more likely than low stakes environments to lead to reduced reliance on technically appropriate measures and increased dependence on measures better reflect the broader political environment.     

Web accessibility: evaluation of a website with different semi-automatic evaluation tools

It is essential that access to information must to be universal given the importance of having a truly global Internet. This study proposes to evaluate a government website with two semiautomatic accessibility evaluation tools. The results have demonstrated that the use of more than one semi-automatic assessment tool can provide enhanced results. The differences in results and presentation in the reports generated by the two tools indicate that there is more complete results by combining tests with different accessibility evaluation tools.     

Using a community-created multisite evaluation to promote evaluation use across a sector

Multisite evaluation has been cited as a strategy for building knowledge about a program's success and enhancing evaluation capacity and evaluation use. Even so, the extent to which data from these efforts are made available to individual project participants varies. This article describes a community-created multisite evaluation that was designed to overcome this limitation. Based on the negotiated centralized evaluation model and implemented within the context of a community of practice, the EvalFest community-created multisite evaluation has used shared measures to collect data from 30,000 participants from partner sites across the community over three years. This article describes the design characteristics of the community-created multisite approach, and presents results related to partners’ evaluation use that demonstrate the early success of this method. We conclude by considering how this strategy might be applied to other sectors.     

Logic and evaluation theory

With this series of papers, evaluators are being called to substantiate the rationale and warrant for their own evaluative actions in ways parallel to how evaluators question the logic of program interventions, both as designed and as implemented. This endeavor is timely, appropriate, and important. In these comments, I raise modest questions about the logical constitution of an evaluation theory and about what is missing from a textual reading alone of such theory.     

Advances in collaborative evaluation

Collaborative evaluation is an approach that offers, among others, many advantages in terms of access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings. In the last decade, collaborative evaluation has grown in popularity along with similar participatory, empowerment, and utilization-focused evaluation approaches. Increasingly, it has brought together evaluators and stakeholders from different sectors, disciplines, and cultures to exchange knowledge on how collaboration can be used as a strategic tool for fostering and strengthening evaluation practice. O'Sullivan (2004) and Rodríguez-Campos, 2005, Rodríguez-Campos, 2008 brought forward collaborative evaluation conceptual frameworks that distinguished it from other approaches. This article updates those conceptual frameworks and discusses key collaborative evaluation advances in order to further clarify and facilitate engagement in sound practice.     

Program evaluation in rehabilitation

Increased accountability and more encompassing accreditation standards have heightened interest in the role that evaluation research plays in improving the quality of life and independence for people with disabilities. The field of Program Evaluation is a valuable resource for conducting studies that meet the standards and information needs in the field of rehabilitation. This article describes the nature of evaluation research and illustrates its contribution to contemporary rehabilitation practice.     

Legislative history and evaluation

Legislative histories can be done with those evaluative studies which are based on programs which have a history in the legislature. ESEA Title I, Title IV, and the Title I Technical Assistance Centers are examples of programs which have their direct source in federal legislation. The evaluator could ask whether the intent (aims and objectives) of a program is the same as the intent of the legislation or could ask whether the outcomes of a program were the same as those intended by the legislature. The purpose of this paper is to introduce the reader to the doing of legislative history in an evaluation study. The paper introduces the reader to five major sources for legislative history. The paper then discusses seven acceptable sources of information. Following this is a five-step guide to doing a legislative history.     

Management consulting and evaluation

The basic functions and procedures of management consulting are contrasted here with formal program evaluation. A case example further illustrates the differing approaches used in programmatic problem solving. The similarities and differences between these two professional specializations are highlighted in order to understand better the unique but complementary contribution each makes to the improvement of program operations.     

Advances in collaborative evaluation

Collaborative evaluation is an approach that offers, among others, many advantages in terms of access to information, quality of information gathered, opportunities for creative problem-solving, and receptivity to findings. In the last decade, collaborative evaluation has grown in popularity along with similar participatory, empowerment, and utilization-focused evaluation approaches. Increasingly, it has brought together evaluators and stakeholders from different sectors, disciplines, and cultures to exchange knowledge on how collaboration can be used as a strategic tool for fostering and strengthening evaluation practice. O'Sullivan (2004) and Rodríguez-Campos (2005, 2008) brought forward collaborative evaluation conceptual frameworks that distinguished it from other approaches. This article updates those conceptual frameworks and discusses key collaborative evaluation advances in order to further clarify and facilitate engagement in sound practice.     

Leverage and evaluation effectiveness

Weaknesses in evaluations often can be traced to structural limitations in the positions of evaluation researchers. Conventional human relations techniques often are an insufficient basis for securing strong support for evaluation research. Strategies for increasing evaluation research leverage are reviewed. Alignment of evaluation research with regulatory bodies with authority to suspend public program expenditures is advocated. Several likely obstacles in the development of the regulatory evaluation model are anticipated and addressed.     

Predictive validity of a functional capacity evaluation: the physical work performance evaluation

The predictive validity of a Functional Capacity Evaluation, the Physical Work Performance Evaluation (PWPE), was examined in 30 workers compensation patients with musculoskeletal dysfunction who participated in an industrial rehabilitation program. Return-to-work recommendations were made by comparing the discharge PWPE to the physical requirements of the job. Kappa coefficients were used to compare PWPE recommendations to actual work at discharge, 3 months and 6 months. Substantial agreement was found between PWPE recommendations at all three follow-up points. This study provides preliminary evidence in support of the predictive validity of the PWPE for making return-to-work recommendations at the conclusion of an industrial rehabilitation program.     

Theory-based evaluation of a comprehensive Latino education initiative: an interactive evaluation approach

Latino student access to higher education has received significant national attention in recent years. This article describes a theory-based evaluation approach used with ENLACE of Hillsborough, a 5-year project funded by the W.K. Kellogg Foundation for the purpose of increasing Latino student graduation from high school and college. Theory-based evaluation guided planning, implementation as well as evaluation through the process of developing consensus on the Latino population of focus, adoption of culturally appropriate principles and values to guide the project, and identification of strategies to reach, engage, and impact outcomes for Latino students and their families. The approach included interactive development of logic models that focused the scope of interventions and guided evaluation designs for addressing three stages of the initiative. Challenges and opportunities created by the approach are discussed, as well as ways in which the initiative impacted Latino students and collaborating educational institutions.     

Open systems evaluation and the logic model: Program planning and evaluation tools

This paper describes two case management projects and efforts to evaluate outcomes. Lessons learned from the first effort suggest that a tool referred to as the logic model and a particular approach to evaluation called the “open systems” model might be useful to planners and program evaluators. The logic model provides a means of presenting conditions a program is intended to address, activities that constitute a program, short term outcomes resulting from program activities and long term impacts. The open systems perspective emphasizes evaluation as a tool to be used to achieve program objectives as opposed to establishing cause and effect relationships. This paper also describes how the logic model and open systems evaluation were used to facilitate the development of an evaluation plan for a project designed to assist homeless, substance abusing, pregnant women.