Evaluation readiness: improved evaluation planning using a data inventory framework

Factors intrinsic to many programs, such as ambiguously stated objectives, inadequately defined performance measures, and incomplete or unreliable databases, often conspire to limit the evaluability of these programs. Current evaluation planning approaches are somewhat constrained in their ability to overcome these obstacles and to achieve full preparedness for evaluation. In this paper, the concept of evaluation readiness is introduced as a complement to other evaluation planning approaches, most notably that of evaluability assessment. The basic products of evaluation readiness--the formal program definition and the data inventory framework--are described, along with a guide for assuring more timely and appropriate evaluation response capability to support the decision making needs of program managers. The utility of evaluation readiness for program planning, as well as for effective management, is also discussed.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

A response to the BPA evaluation plan: Some modest but utilitarian considerations

In response to the complex and expensive Bonneville Power Administration (BPA) evaluation plan, this article attempts to identify some pragmatic issues and alternative approaches to conservation program evaluation. In particular, the need for simpler, lower-cost evaluation options and the importance of process evaluation to program management are emphasised. This is followed by the delineation of several key principles to be observed in conducting a more modest, but still methodologically sound and managerially useful, evaluation of energy conservation programs.     

The evaluation and monitoring of a Medicaid second surgical opinion program.

The evaluation of newly implemented and controversial programs often present real problems for the evaluator. When a program is accompanied by a legislatively mandated evaluation, the issues become even more acute. The evaluator, wanting to impact on policy, must strike a balance between the adherence to principles of scientific inquiry, yet be responsive to the needs of policymakers. This paper describes the evaluation and follow-up monitoring of a recently established second opinion program for Medicaid recipients. The program was to terminate in 17 months unless extended by the Wisconsin Legislature. The evaluation found, among other things, marked reductions in the use of surgical services after implementation of the second opinion program. Based in large part on the evaluation, the program was extended with a provision for continued monitoring. The paper discusses the factors that contributed to the evaluation having a major impact on the decision to extend the program.     

Ethical problems in program evaluation: Advice for trapped evaluators

This paper argues that ethical problems in program evaluation are increased when conflicting or incompatible evaluation models are applied concurrently. Three models are illustrated: an Amelioration Model, generating better information for a program's own decision makers; an Accountability Model, focusing on public data disclosure and citizen participation in evaluation; and an Advocacy Model, in which the evaluation is designed to advance the program's interests in the competition for resources. Ethical problems inherent in each model, and arising from combining models are presented, along with case examples and discussion. The authors suggest several directions that program evaluators can consider to reduce their vulnerability to ethical problems.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Problems in being responsive: Reflections on an evaluation of a program for training motorcycle riders

This paper discusses some methodological issues arising from an evaluation of the Queensland Road Safety Council Motorcycle Training Program. The basic orientation of the evaluation was responsive/illuminative; responsive in the sense that the evaluation proceeded interactively, with appropriate issues and procedures identified and modified progressively through direct observation of the program and discussion with the participants; illuminative in the sense that the final report attempted to provide a detailed account of the program operation and of its strengths and weaknesses as perceived by various stakeholders. An account is given of the main considerations in designing, implementing, and reporting the evaluation. Implications for future evaluations are discussed.     

Organizing evaluation to serve the needs of program planners and managers

Evaluation is often both conceptually and functionally separate from program planning and operations management. Though program managers may not see themselves as served well by the work of evaluation staffs, there are a number of potentially beneficial side effects to be realized from close collaboration between program administrators and evaluators. These flow only when the evaluation task is broadly defined and when both administrators and evaluators recognize their interdependence. Particular functional benefits to management from evaluation are described. Characteristics of interorganizational relations which might increase the probability of their attainment are then outlined and discussed.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

The teaching-family model: Research and dissemination in a service program

The Teaching-Family Model serves as an example of how research can be used as feedback to change a residential treatment program for youths. Further, research can serve to modify training and evaluation of the program as well. The feedback loop established by continual research and evaluation serves to improve program quality, thus facilitating dissemination as the model is adopted by more agencies.From 1967 to 1980, the Teaching-Family Model expanded from one group home in Kansas to more than 150 homes across the United States. Through both successes and failures, proponents of the Teaching-Family Model learned that research and evaluation can and must be a part of the treatment delivery system rather than an occasional adjunct.     

Evaluating utility conservation programs: The BPA plan

This paper summarizes a detailed plan for evaluation of the residential energy conservation programs offered by the Bonneville Power Administration (BPA). The major purpose of the evaluation is to determine the energy savings attributable to the programs and to identify differences between program participants and nonparticipants. The focus of this paper is on the sampling issues associated with reliable estimates of energy savings. The methodology attempts to correct for the self-selection bias likely to occur in conservation program evaluation by using intentional nonrandom sampling. In addition, the paper suggests a variety of analytical approaches to analyzing the data with respect to energy savings.The proposed design, using four different groups to estimate program effects, is a comprehensive endeavor. This is appropriate for BPA because of the substantial investment it is making in conservation. However, the design is flexible and allows for use offewer groups to reduce evaluation costs.     

Applicability of multiple matrix sampling to estimating effectiveness of educational programs

When evaluating the effectiveness of an educational program, a sampling strategy known as “multiple matrix sampling” is particularly effective and efficient when the goal of the evaluation is estimating group (as opposed to individual) performance. Although this technique involves sampling both program participants and items and, subsequently, different participants being assessed over different sets of items, the results may be used to estimate those which would have been obtained had all participants been administered all items. The technique is here described in some detail, with advantages and disadvantages discussed, and examples of its application are given.     

Resolving conflicting demands on evaluation: A test demonstration of videotex for farmers

Evaluating an innovation for federal, state, and local policymakers and program managers alike entails conflicting demands on the evaluation study. Policymakers at federal, state, and local levels are best assisted by impact evaluations, whereas state and local program managers are best assisted by process evaluations. In-house evaluators often have an advantage in conducting process evaluations; external evaluators generally have an advantage in conducting impact evaluations. A cost-effective approach may be to combine in-house process evaluation and external impact evaluation. This dual approach was found to reduce conflicting demands on the evaluation of an experimental videotex system for agricultural producers.     

On considering the unintended impact of evaluation: reactive distortions in program goals and activities

The relationship between program planning and evaluation can be viewed as bidirectional; that is, evaluation methods, procedures, instruments, and criteria not only are determined by, but also influence, program goals and activities. Within the human services context, several factors or sources of reactivity between evaluation and program planning can be identified. These involve (a) quantification of goals and activities, (b) preferences by different audiences for various kinds of evaluation data, (c) values and evaluation criteria, and (d) evaluation requirements and resource availability. Effects of these reactive features are discussed and illustrated with examples drawn from mental health evaluation and accountability practices. It is argued that for evaluation to be a credible and useful practice, evaluators should plan their efforts and assess their own effectiveness within the larger context of human service systems.     

Leverage and evaluation effectiveness

Weaknesses in evaluations often can be traced to structural limitations in the positions of evaluation researchers. Conventional human relations techniques often are an insufficient basis for securing strong support for evaluation research. Strategies for increasing evaluation research leverage are reviewed. Alignment of evaluation research with regulatory bodies with authority to suspend public program expenditures is advocated. Several likely obstacles in the development of the regulatory evaluation model are anticipated and addressed.     

Management consulting and evaluation

The basic functions and procedures of management consulting are contrasted here with formal program evaluation. A case example further illustrates the differing approaches used in programmatic problem solving. The similarities and differences between these two professional specializations are highlighted in order to understand better the unique but complementary contribution each makes to the improvement of program operations.     

An evaluator of the future

For the evaluator who is part of the management of a human service organization, future expectations will be more demanding than at present. Operating staff, managers, and funders will already be familiar with the methods and the usefulness of capable program evaluation. Improved information tools will help the evaluator meet this challenge, by keeping the organization visible and manageable. A more fundamental change will be an improved “language of accountability”. Program standards, accreditation, and review now focus on structure and process characteristics. What is emerging is a growing consensus on definitions of client problems and program objectives in relation to these problems. This consensus spurs psychometric research on measures of problems and of the attainment of program objectives. Dependable, standard measures and meaningful comparative data will make the evaluation of program effectiveness possible.