The evaluator as program consultant

This paper presents a model for the role of the evaluator in which the evaluator acts as a Program Consultant who operates in three domains: Program Development, Planned Change, and Evaluation Technology. This permits the performance of evaluations that have greater validity and utility. The model is presented both from the theoretical perspective of linkage and from the context of the evaluation of a Case Management program operated by a Community Action Agency. Each of the domains is discussed with examples from the Case Management Evaluation. The paper concludes with a discussion of the contexts in which the model is best applied.     

Response from a policymakers' perspective

The case in point (Tellado, 1984) needs to be defined in terms of a client for the research. The research may also have implications for early intervention to mitigate the problems uncovered in the case study. An important element in the development of problem-solving skills may be linguistic skills. Further research about this variable seems warranted.     

Teaching graduate-level evaluation sociology through a departmental evaluation

The Department of Sociology at the University of Texas at Austin conducted an internal evaluation of its graduate program through an applied methodology graduate course. This article reports on the effectiveness of the format for teaching research methods, the utilization of the evaluation's findings, and the conditions that need to occur for a similar study to take place. Through the course, graduate students gained experience using multiple methods in evaluation research as well as acquiring an inside perspective on the department and the profession. Although some problems of conducting an internal evaluation are discussed, we found that with departmental support the evaluation was an effective method of promoting internal discourse and change.     

An alternative evaluation approach for the problem-solving training program: A utilization-focused evaluation process

A utilization-focused approach in evaluating this program would have placed more emphasis on clearly identifying evaluation users, their information needs, and likely use of findings. With the advantages of hindsight, it appears that such an approach might have led to greater focus on critical implementation issues that affected the eventual decision not to continue the program despite seemingly positive outcomes evaluation findings. Other methods options are also discussed, including the potential usefulness of critical case studies. Finally, there is discussion of how to prepare decision makers for utilization and work with them to interpret and apply findings.     

Stakeholders as partners in evaluation: A stakeholder-collaborative approach

An emphasis in recent years on the utilization of evaluation has led to such innovations as stakeholder-based evaluation, which typically allows concerned groups to participate at the planning and review phases of a study. This paper presents a case study of a “stakeholder-collaborative” approach, in which the roles of participants were extended, to incorporate the design, administration, and interpretation of surveys and interviews, extensive data-gathering and development of major portions of the final report. An internal evaluator assigned to the group assumed the roles of facilitator and task manager, to enhance the validity of results. Benefits and disadvantages of the approach for this case are discussed, as well as utilization of the report produced. Six months after submission of the final report, four group members and two major users were interviewed regarding this approach and the report's use; all rated the process positively. Interviewee's suggestions for decreasing problems and maximizing benefits are summarized.     

Evaluation case histories as a parallel to legal case histories: Accumulating knowledge and experience in the evaluation profession

Legal case histories are a basic and essential part of the law. It is proposed that evaluation case histories would be valuable to the evaluation profession in that they would serve both to accumulate knowledge and experience and as examples of how to do evaluations. Part I of the paper is a discussion of the nature, place and use of case histories in law. The format of a reported case, the use of cases by attorneys and how case decisions are reported and categorized are also discussed. Part II of the paper tries to envision the nature and use of evaluation case histories as a parallel to legal case histories. Matters discussed include the possible function and use of evaluation case histories, the difference between an evaluation report and an evaluation case history, information an evaluation case history could contain, and how case histories might be reported and categorized.     

The case study evaluation: A means for managing organizational and political tensions

This study of evaluation utilization identified organizational, political, and practical arrangements which facilitated wide use of The Interim Report Evaluation in policy making for California's Early Childhood Education program. In a program fraught by tensions, where evaluations had been political tools, this evaluation was special. The research used a field study approach. Analysis was guided by literature on organizations, policymaking, and evaluation utilization. It identified techniques for maintaining political support, marshalling organizational resources, and designing and disseminating an evaluation that used an ethnographic approach and that was directly applied to policy deliberations. Such techniques have significance when evaluators and researchers need to convince policymakers of their worth. This case study adds to knowledge of the research/policy intersect, on ethnographic evaluation, and on state education policymaking.     

Using multiple lines of evidence and independent teams to enhance the credibility of evaluations

The utilization of evaluation findings is an ongoing issue among evaluators. This article presents an approach to conducting evaluations that can be used to enhance the credibility of findings and to facilitate the production of adequate documentation in support of evaluation findings. The concepts of multiple lines of evidence and the use of independent teams are discussed. Use of this approach is expected to improve the utilization of evaluation findings.Two program evaluations that were undertaken at Canada's federal department of Consumer and Corporate Affairs are presented. To undertake the data collection and analysis, these studies were broken into separate “evaluation modules, ” each one undertaken by an independent team. The teams reported to an in-house study director, responsible for weighing the evidence and findings of each of the teams and writing the final evaluation report.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.     

Sociocultural Researchers Can Shape the Future of Education

It has been argued that self-generated explanation (self-explanation) is an important tool in the acquisition of knowledge. However, the rationalist and the empiricist approaches, with which self-referential activity has usually been discussed, makes the phenomenon of self-explanation problematic. In this article, I examine self-explanation as a private case of the classical learning paradox, which is associated with self-referential activity, and consider an activity oriented solution to this difficulty. From a practical perspective, I suggest a reframing of self-reference in Activity Theory in order to provide researchers with a method for analyzing and supporting bootstrapping phenomena, such as the self-explanation effect, in learning and problem solving.     

Long over short term

Abstract

Managers and organizations are learning that dealing with ecological issues seriously is part of their responsibility. In this article, the inherent rationality of an ecologically oriented approach to management is highlighted. For this purpose, a conceptual framework for a comprehensive fitness of organizations is presented. Based on this framework and on case studies in small businesses a simulation model has been developed. The structure of this model is outlined. Then, on the basis of simulation experiments, several aspects of ecological management are discussed. The focus is on the role of pre-control, which substantiates the primacy of the long-term perspective. In this context, the trade-offs between short- and longer-term decisions, as well as the vulnerability of one-sided strategies are discussed.     

Paradoxes of risk management: Social responsibility and self-exclusion in Dutch casinos

This article deals with the basic contradictions of risk management and responsibility regarding problem gambling and self-exclusions, and draws special attention to the role knowledge production and science play in the construction and evaluation of gambling risks. This remarkable case of corporate social responsibility (CSR) is analyzed from a risk-governance perspective. It is grounded on a case study of the CSR practice of the Dutch casino monopoly, in particular the problem gambling mitigation and self-exclusion program of this corporation. The intentions and operations of this ‘responsible gambling’ practice constitute new business values and working norms. At the same time empirical evaluations of the self-exclusion program reveal that despite all the efforts put in the risk-management strategies, a significant part of the problem cannot be addressed at all. This points toward a major paradox of risk management. Overall the article highlights the ambiguous nature of risk management and responsible gambling.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Reducing Child Protection Error in Social Work: Towards a Holistic-Rational Perspective

Social work in the UK remains embroiled in concerns about child protection error. The serious injury or death of vulnerable children continues to evince much consternation in the public and private spheres. Governmental responses to these concerns invariably draw on technocratic solutions involving more procedures, case management systems, information technology and bureaucratic regulation. Such solutions flow from an implicit use of instrumental rationality based on a ‘means-end’ logic. While bringing an important perspective to the problem of child protection error, instrumental rationality has been overused limiting discretion and other modes of rational inquiry. This paper argues that the social work profession should apply an enlarged form of rationality comprising not only the instrumental-rational mode but also the critical-rational, affective-rational and communicative-rational forms. It is suggested that this combined, conceptual arsenal of rational inquiry leads to a gestalt that has been termed the holistic-rational perspective. It is also argued that embracing a more rounded perspective such as this might offer greater opportunities for reducing child protection error.     

Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

An examination of methods in the evaluation of volunteer programs.

Although many volunteer programs have been called "successful" and the use of paraprofessionals urged, few program evaluations with a benefit-cost analysis have been reported. Several major studies of volunteer utilization are reviewed, and particular needs with reference to program evaluation are discussed. Three alternative evaluation methodologies are proposed. The importance of utilizing a comprehensive evaluation incorporating all three methodologies is discussed. An ongoing volunteer program in a community mental health center is used as an illustration.     

Software development for the evaluation of the ergonomic compatibility on the selection of advanced manufacturing technology

Advanced Manufacturing Technology (AMT) is one of the most relevant resources that companies have to achieve competitiveness and best performance. The selection of AMT is a complex problem which involves significant amount of information and uncertainty when multiple aspects must be taken into consideration. Actual models for the selection of AMT are found scarce of the Human Factors and Ergonomics perspective which can lead to a more complete and reliable decision. This paper presents the development of software that enhances the application of an Ergonomic Compatibility Evaluation Model that supports decision making processes taking into consideration ergonomic attributes of designs. Ergonomic Compatibility is a construct used in this model and it is mainly based in the concept of human-artifact compatibility on human compatible systems. Also, an Axiomatic Design approach by the use of the Information Axiom was evolved under a fuzzy environment to obtain the Ergonomic Incompatibility Content. The extension of this axiom for the evaluation of ergonomic compatibility requirements was the theoretical framework of this research. An incremental methodology of four stages was used to design and develop the software that enables to compare AMT alternatives by the evaluation of Ergonomic Compatibility Attributes.     

Clinical implications of attachment narratives

This paper discusses attachment theory and research with emphasis on the empirical studies of Many Main. Main's work emphasizes the relationship between an individual's narrative regarding attachment relationships and their underlying internal working model of attachment. The internal working model of attachment is a representation of attachment experience and a set of rules for processing attachment experience. This paper describes various categories of attachment narrative construction and presents a case history that is discussed from an attachment theory perspective.     

Justifying conclusions in naturalistic evaluations: An interpretive perspective

Use of an interpretive perspective in naturalistic inquiry is one way to reconcile assumptions and practice in naturalistic evaluations, particularly concerning the justification of evaluation conclusions. In this paper I conclude that naturalistic evaluation based on an interpretive perspective cannot appeal to external methodological criteria, like trustworthiness criteria, for protection against bias in reaching evaluation conclusions. I suggest that claims for evaluation conclusions depend on the evaluator as interpreter — using his or her skills of persuasion and analysis to mediate among perspectives, selected evidence and claims for conclusions.