A response to the BPA evaluation plan: Some modest but utilitarian considerations

In response to the complex and expensive Bonneville Power Administration (BPA) evaluation plan, this article attempts to identify some pragmatic issues and alternative approaches to conservation program evaluation. In particular, the need for simpler, lower-cost evaluation options and the importance of process evaluation to program management are emphasised. This is followed by the delineation of several key principles to be observed in conducting a more modest, but still methodologically sound and managerially useful, evaluation of energy conservation programs.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.     

An alternative evaluation approach for the problem-solving training program: A utilization-focused evaluation process

A utilization-focused approach in evaluating this program would have placed more emphasis on clearly identifying evaluation users, their information needs, and likely use of findings. With the advantages of hindsight, it appears that such an approach might have led to greater focus on critical implementation issues that affected the eventual decision not to continue the program despite seemingly positive outcomes evaluation findings. Other methods options are also discussed, including the potential usefulness of critical case studies. Finally, there is discussion of how to prepare decision makers for utilization and work with them to interpret and apply findings.     

The goal-oriented approach to evaluation: Critique and case study from drug abuse treatment

The specification and measurement of program goals remains central to most evaluation research strategies, yet procedures for implementing this approach are not well-articulated. It is the purpose of this discussion to describe a stepwise procedure for programmatic goal setting and monitoring used in a demonstration drug treatment program for women. Three implementation steps are described: (a) goal setting, (b) checking for consistency, (c) monitoring and feedback. The advantages and limitations of this approach are discussed and useful complementary measurement strategies are suggested.     

Complexity of process and role in the evaluation of an interdisciplinary mental health education program: The deacon's masterpiece

Research and literature on program evaluation must attend to problems of application, which can have major effect on the outcome. Increasing numbers of innovative, multifaceted programs operating in complex settings are presenting for evaluation. Evaluation process must become flexible, creative, and multifaceted in order to produce valid and comprehensive results, and to meet the needs of this field. Evaluators and evaluation, the evaluated program, and the environment become a mutually interacting system in the process of evaluation. Evaluation goals and roles must be conceptualized, agreed upon, sanctioned, and appropriately implemented or conflict and ineffectiveness will result. Three models of evaluation emerge: objective and independent, objective but serving one interest, and pseudo-scientific public relations. Evaluation of an interdisciplinary mental health education program illustrates these issues. Evaluation technology, process, and roles must all be attended to or the weak link undermines the rest.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

Where local and national evaluators meet: unintended threats to ethical evaluation practice

The ethical work of program evaluators is based on a covenant of honesty and transparency among stakeholders. Yet even under the most favorable evaluation conditions, threats to ethical standards exist and muddle that covenant. Unfortunately, ethical issues associated with different evaluation structures and contracting arrangements have received little attention in the evaluation research literature. This article focuses on the unintended ethical threats associated with multitiered evaluation contexts. After briefly reviewing the various frames through which evaluation theory and ethics are commonly viewed, we discuss ethical challenges associated with multitiered evaluation designs including examples drawn from our evaluation projects. The article concludes with specific findings and recommendations for evaluators, grantors, grantees, and researchers.     

The use of metaevaluation to plan evaluations of conservation programs

The U.S. Department of Energy (DOE) is sponsoring and will be required to evaluate a wide range of conservation programs. The development of evaluation plans for such programs can often be improved by reviewing evaluations that have been done on similar programs. The Residential Conservation Service (RCS), sponsored by DOE, is discussed as an example of how conducting metaevaluations can improve evaluation planning. Many utilities have sponsored programs that are similar to the RCS. A review of the utility program evaluations suggested key methodological issues and data problems that should be considered in the development of an RCS evaluation plan. The implications of these metaevaluation findings for an RCS evaluation plan are discussed.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

Challenges in evaluating programs serving homeless families

Homeless families, usually headed by single mothers, are the fastest growing segment of the homeless population. Although families with children make up a large percentage of those experiencing homelessness, there have been few studies on the effectiveness of services for homeless families offered by shelters transitional housing programs, and other agencies. This article summarizes the research that has been conducted on families that are homeless and the limitations of this research for program evaluation. Challenges to program evaluation are identified: the lack of published studies in the area; the fact that program evaluation is rarely built into programs or funded; the methodological issues in tracking families; the difficulty in assessing the impact on families; implementation issues; the range of barriers to housing stability faced by homeless families; and changes in society at large that influence the results of evaluation. Six recommendations are made for meeting these challenges: evaluation from a coherent theory of change; setting reasonable and differential expectations for clients; strengthening linkages with other agencies that serve the same client population; increasing the emphasis on follow-up; utilizing prevention efforts to reduce the number of families that become homeless; and working for expanded low-income housing and expanded subsidies.     

Community-Based Program Research

ABSTRACT

At first glance, the worlds of program professionals and evaluators may seem quite separate. However, there are common issues whose resolution will enhance both program development and research on programs. Program professionals and evaluators have a great deal to learn from each other. As that learning occurs, both program delivery and evaluation research will benefit. Both are concerned about matters of validity, whether it pertains to the nature of the program intervention itself or to the nature of the assessment of the program intervention. To reinforce the development of program evaluation in partnership, this paper discusses key points about evaluation research relevant to both program professionals and researchers. These include the contextual influences on a program, the “readiness” of a program for evaluation, and whether the evaluation research “works” for the program. Underlying our discussion is a call for the development of partnerships around research on programs. In that light, this article elaborates the process steps that should be taken to build program evaluation partnerships, including a discussion of what evaluators and program professionals need to know about each other, and their respective values, interests, and professional perspectives.     

Designing useful evaluations: The stakeholder survey

The failure of program evaluations to provide policy decision-makers with useful information is a familiar complaint. This paper discusses an approach towards designing and implementing more useful evaluations — the stakeholder survey. Stakeholders are individuals, or groups, who have a direct interest in the program being evaluated. The central premise of the paper is that the information needs of stakeholders should be incorporated directly into the formulation of evaluation objectives, and the design of the evaluation to achieve those objectives. The paper presents a rationale for the stakeholder survey, describes its major components, and reports on the application of the approach in a large program evaluation. Key issues involved in applying the approach are also considered. The evidence from this study suggests that such surveys are not only feasible, but also of considerable value to the evaluation team.     

Systems engineering: A problem-solving approach to improving program performance

Evaluation methodologies have generally emphasized the measurement and assessment of program merit, as reflected by the achievement of program objectives. Less attention has been devoted to analyzing and explaining the reasons for deficient performance and, more importantly, what needs to be modified to improve performance. While evaluators do identify performance problems, the identified problems are not always analyzed. Consequently, the information reported to decision makers may be incomplete, inconclusive, and of limited utility in the decision-making process. This article describes an evaluation methodology based on problem solving techniques which can be effective and efficient in defining and analyzing problems which impair program performance. The model can be applied in evaluations where the purpose is to provide decision makers with information and recommendations to improve program performance and provide an alternative to those evaluation models which emphasize the assessment of program merit. The model also provides a potentially unique cost/benefit methodology for estimating the potential worth of improving program performance (i.e., solving the problem).     

Problems in being responsive: Reflections on an evaluation of a program for training motorcycle riders

This paper discusses some methodological issues arising from an evaluation of the Queensland Road Safety Council Motorcycle Training Program. The basic orientation of the evaluation was responsive/illuminative; responsive in the sense that the evaluation proceeded interactively, with appropriate issues and procedures identified and modified progressively through direct observation of the program and discussion with the participants; illuminative in the sense that the final report attempted to provide a detailed account of the program operation and of its strengths and weaknesses as perceived by various stakeholders. An account is given of the main considerations in designing, implementing, and reporting the evaluation. Implications for future evaluations are discussed.     

Quality of life in the evaluation of community support systems

As a result of the authors' experience in conducting an evaluation of the community support system (CSS) program in New York State, they have identified five reasons for focusing on quality of life (QOL) as a desired outcome for programs for the chronically ill. These reasons are presented and problems in QOL evaluative research are discussed. Psychological indicators are distinguished from social indicators of QOL, and two methods of operationalizing perceived QOL, the psychological well-being and life domains methods, are examined. A conceptual model is presented and the results of an exploratory study of the QOL of 118 chronic psychiatric patients receiving CSS services are presented.     

“Dirty” research in “real” places: A practitioner's guide to program evaluation in the human services

Traditional models of evaluation have often fallen short of their potential for improving services to clients. Using illustrations from a study of client turnover in group homes for troubled youth, this paper discusses practical ways to increase the impact of program evaluation in human service settings.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Putting youth development into practice: learning from an innovative fellowship program

Professionals who work with youth can have a tremendous impact on the development and life trajectory of these young people. This article reports on an effort to provide support and professional development for those who work with youth during nonschool hours in a youth development fellowship program. Combining intensive residency workshops and a six-month coaching experience, the fellowship was designed to teach participants how to integrate youth resilience and youth development perspectives into their work. The evaluation collected data from the fellows and their coaches during the course of the program. The study offers insights into the types of benefits achieved through this type of focused intervention with youth workers, as well as the challenges of conducting such an effort.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Die Kombination von Lebenslauf- und Biografieforschung

The combination of life course and biographical research can only be done within the framework of mixed-method-approaches. As there is no established research program for this, good practice examples of existing projects in which methods have been combined have to be reflected. In this sense, this article discusses three case studies regarding the complementarity of life course and biographical perspectives and possible synergies emerging from the combination. The discussion is based on three sequential mixed methods projects investigating identity-related questions: identity work and employment, family foundations and the construction of national belonging. Four different forms of synergies can be analyzed: a) Complementary understanding of process dynamics, b) complementary understanding of different types, c) complementarity to avoid one-sided misinterpretations and to understand the role of ambivalences for the course of life, and d) expanded possibilities for the evaluation of theoretical models. Finally, the position is justified that the combination of life course and biographical research doesn’t have necessarily to be linked to a paradigmatic standpoint when setting up the study. Qualitative and quantitative studies may be considered of equal importance without the necessity to decide which method is core and which is supplementary.