On participant-observation as a component of evaluation. Strategies, constraints and issues

Described here is a participant-observation strategy employed as one portion of the evaluation research at Family House, a residential treatment program for alcoholic mothers and their children. Major objectives of this leg of a two-legged research design combining this approach with a more standard pre- and post-test strategy were to document and analyze the processes of social development in the new "small society." The ethnographer acted as consultant, with "raw data" generated in the form of daily observations dictated on the rotating basis by staff members over a two-year period. The resulting narrative was coded and analyzed by the anthropologist, utilizing a phenomenological or interpretationist perspective which allowed the "native categories" of significance to emerge and guide the analysis. This novel research strategy is compared with standard ethnographic fieldwork, as well as with pre- and post-test design, in terms of its strengths and weaknesses. Strengths include access to an authentic "inside view" of Family House reality, as well as an unusually rich longitudinal record of daily life in the facility. Problems include danger of contaminating the data by means of the "feedback" provided to reward staff for their cooperation in the research effort. The author concludes, however, that since reactive effects are present in all designs, and since "feedback" is a normal feature of therapeutic milieux, validity is not unusually threatened. Finally, the purposes of evaluation research are scrutinized.     

Evaluation in the community mental health centers program: A bold new reproach?

The Federal Community Mental Health Centers Program (CMHC)-from 1963 to 1981-was heralded as a revolution in mental health care. Championed by many, and severely criticized by others, the actual impact of the program on the nation's mental health remains unclear. The authorization to evaluate the CMHC Program came originally from congressional legislation (PL 90-174), and later from the policies and regulations of NIMH under a series of Federal laws, notably PL 94-63. From 1976-1980, two dominant evaluation strategies were prevalent: funds expended by NIMH each year for studies of CMHC services or program-wide evaluations, and a much larger expenditure by CMHCs to conduct their own, independent evaluations following federal guidelines. As the Center's Program was turned over to the states in the form of block grants (PL 97-35), a group of professionals involved with setting and carrying out federal CMHC evaluation policy of both varieties met in public forum to debate the impact of these two evaluation approaches. While some participants cited gains in evaluation technology and impact upon local management of CMHCs, others found the lack of a coordinated and systematic approach to evaluating the CMHC Program to have been an opportunity missed. The impact of CMHC evaluation efforts are also discussed in terms of their major contribution to the field of evaluation research as a whole.     

A remedial strategy for data analysis

Professional evaluators are often called upon to analyze data produced by a catastrophically inadequate evaluation design. This problem is occurring more frequently as accountability pressures force program experts into evaluation activities for which they are not trained. A remedial strategy involving diagnosis of error, application of a corrective procedure and sensitization of program personnel of the need for a more sophisticated stance, is proposed as a solution. A case study is described and the contribution of a remedial strategy to improved evaluation is outlined.     

Connecting the dots: implementing and evaluating a network intervention to foster scientific collaboration and productivity

This article presents the design and implementation of a network intervention to foster scientific collaboration at a research university, and describes an experimental framework for rigorous evaluation of the intervention’s impact. Based on social network analysis of publication and grant data, an innovative type of research funding program was developed as a form of alteration of the university’s collaboration network. The intervention consisted in identifying research communities in the network and creating a new collaborative relation between pairs of unconnected researchers in selected communities. The new collaboration was created to maximally increase the overall cohesion of the target research community. In order to evaluate the impact of the program, we designed a randomized experiment with treatment and control communities based on the Rubin Causal Model approach. The paper describes the intervention design, reports findings from the program implementation, and discusses the statistical framework for future evaluation of the intervention.     

A model-guided process evaluation: Office-based prescribing and pharmacy dispensing of methadone

This article presents an exemplar of a model-guided process evaluation that specifies the treatment model, assesses its implementation, monitors the fidelity of the model throughout the project, assesses model exposure and absorption, and helps understand the program's intermediate effects (proximal outcomes) as well as final effects (distal outcomes). The New Mexico study on office-based prescribing and community pharmacy dispensing of methadone is a research demonstration project that phases a small group of female methadone maintenance patients out of methadone clinics and into a program where they will obtain their scheduled doses of methadone at pharmacies that work in collaboration with physicians and a social worker.The patient's methadone treatment will in this way become part of their overall health care. Early detection of problems of implementation (e.g., the omission of program content or the delivery of inaccurate information) enables the researcher to make adjustments before the problems become unmanageable and the integrity of the original research design is compromised. A model-guided process evaluation can critically inform health services research demonstrations designed for enabling continuous, ongoing feedback and improvement of client-related services.     

The goal-oriented approach to evaluation: Critique and case study from drug abuse treatment

The specification and measurement of program goals remains central to most evaluation research strategies, yet procedures for implementing this approach are not well-articulated. It is the purpose of this discussion to describe a stepwise procedure for programmatic goal setting and monitoring used in a demonstration drug treatment program for women. Three implementation steps are described: (a) goal setting, (b) checking for consistency, (c) monitoring and feedback. The advantages and limitations of this approach are discussed and useful complementary measurement strategies are suggested.     

Political and organizational preconditions for application of the standards for educational evaluation : The standards for educational evaluation seen in a wider perspective in relation to policy making and organization

Some of the standards for educational evaluation refer to the political and organizational context in which evaluation projects take place: “conflicting interests,” “political viability,” and various standards which have to do with the identification of the evaluation object and of the most important groups and persons involved. Whereas the Joint Committee on Standards for Educational Evaluation seems to focus on the level of the individual research project, the present authors feel that in addition, contextual influences on evaluations should be analyzed at a higher level of aggregation. Political and organizational factors at the macro level can even be seen as preconditions for the application of the standards at project level. This point of view is illustrated by referring to a study of the functioning of evaluation research within some innovative programs in Dutch education.     

An evaluator of the future

For the evaluator who is part of the management of a human service organization, future expectations will be more demanding than at present. Operating staff, managers, and funders will already be familiar with the methods and the usefulness of capable program evaluation. Improved information tools will help the evaluator meet this challenge, by keeping the organization visible and manageable. A more fundamental change will be an improved “language of accountability”. Program standards, accreditation, and review now focus on structure and process characteristics. What is emerging is a growing consensus on definitions of client problems and program objectives in relation to these problems. This consensus spurs psychometric research on measures of problems and of the attainment of program objectives. Dependable, standard measures and meaningful comparative data will make the evaluation of program effectiveness possible.     

Avoiding evaluation cooptation. Lessons from a renal dialysis evaluation.

An evaluation of the impact of terminating contracts to transport low-income patients for renal dialysis demonstrates how evaluators may become apologists for management and conduct studies too narrow for effective decision making. The evaluation found that patients continued to receive treatment; their death rates had not increased. Changes in their economic and emotional status were not studied; consequently, the evaluators erroneously concluded that the clients had not been seriously harmed. The article reviews the process the evaluation used. The authors conclude that to avoid cooptation, evaluators need independent information available through conducting and analyzing pilot studies and meeting with program constituents. To avoid overly narrow studies, the next level of management should participate in their design. To use the information effectively, evaluators need to see themselves as advisors rather than technicians.     

The use of participant observation in evaluation research

In the last several years considerable interest has developed in the use of qualitative methods in evaluation research. Recent works indicate that the tradition of qualitative research has much that will be useful to evaluation researchers. This paper does not seek to cover the range of qualitative methods available to the evaluation researcher but rather is concerned with one methodology in that tradition, participant observation. The paper describes the methodology of participant observation for those who are interested in its characteristics and for those who would like to consider it for evaluation or other types of research. Examples are given of how it has been used and references to additional sources of information are included as well. The paper is organized into three sections: characteristics of participant observation, when it is appropriate to use it, and specific techniques of the method.     

Evaluating a management and organization development effort in mental health agencies.

Although the use of management and organization development techniques in health and human service settings is increasing, there are few systematic studies of their effects. Those which do exist are problematic in many respects. An 18-month program incorporating both management and organization development is described along with the results of its evaluation, which were positive and well-utilized. A discussion of the methods employed, the problems encountered, and the outcomes obtained illustrates that these evaluation efforts are needed and can be useful to program evaluators involved in planned-change efforts.     

From Practice Evaluation to Agency Evaluation

Abstract

This paper describes the efforts that a family service agency undertook to develop a workable evaluation program as part of a United Way demonstration project. Integrating basic elements from single subject design and goal attainment scaling with program logic modeling, a simple system for collecting and compiling data on clinical outcomes was created that could provide feedback on the progress of individual cases to workers, and agency wide improvement rates to clinic administrators and funding agencies. The model that evolved is manageable because it is simple, was developed by the clinical staff, serves a clinical purpose, meets the expectations of funding sources, and actively engages clients, workers and supervisors in an ongoing process of monitoring progress. Sample forms used to gather data are presented along with the results and next steps for the evaluation process.     

Environmental assessment and evaluation research: examples from mental health and substance abuse programs.

Evaluation researchers have been broadening the traditional client input--"black box" treatment--client outcome paradigm that has guided many evaluations of mental health, substance abuse treatment, and other intervention programs. The points of expansion are in the areas of treatment implementation and treatment processes, as well as "extratreatment" influences on treatment selection, duration, and, especially, outcome. This review illustrates the application of environmental assessment procedures--particularly social climate measures--in four aspects of evaluation research suggested by the more comprehensive model: (a) evaluating treatment implementation; (b) exploring treatment processes; (c) identifying extratreatment influences on client posttreatment functioning; and (d) operationalizing outcome variables. Conceptual and methodological issues raised by these applications are discussed, and the benefits to be derived from an expanded model of evaluation research--especially the greater potential for program improvement--are considered.     

Consumer evaluation of community treatment

Development of community programs for offenders requires innovation in program design and evaluation. The Complex Offender Project (COP) provided intensive community treatment, and its clients' opinions were used as a component of its evaluation. Treated clients perceived the same needs for services as did other offenders, but more of them reported receiving the services. Compared to probation, COP was perceived to be more helpful in resolving problems, more continuously helpful, and of helping more to learn useful skills and achieve meaningful goals. The Complex Offender Project provides a model for program development, and the techniques of consumer evaluation are recommended as a means of evaluating such programs.     

Systems engineering: A problem-solving approach to improving program performance

Evaluation methodologies have generally emphasized the measurement and assessment of program merit, as reflected by the achievement of program objectives. Less attention has been devoted to analyzing and explaining the reasons for deficient performance and, more importantly, what needs to be modified to improve performance. While evaluators do identify performance problems, the identified problems are not always analyzed. Consequently, the information reported to decision makers may be incomplete, inconclusive, and of limited utility in the decision-making process. This article describes an evaluation methodology based on problem solving techniques which can be effective and efficient in defining and analyzing problems which impair program performance. The model can be applied in evaluations where the purpose is to provide decision makers with information and recommendations to improve program performance and provide an alternative to those evaluation models which emphasize the assessment of program merit. The model also provides a potentially unique cost/benefit methodology for estimating the potential worth of improving program performance (i.e., solving the problem).     

Leverage and evaluation effectiveness

Weaknesses in evaluations often can be traced to structural limitations in the positions of evaluation researchers. Conventional human relations techniques often are an insufficient basis for securing strong support for evaluation research. Strategies for increasing evaluation research leverage are reviewed. Alignment of evaluation research with regulatory bodies with authority to suspend public program expenditures is advocated. Several likely obstacles in the development of the regulatory evaluation model are anticipated and addressed.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

An evaluation of prototype school-based peer counseling program.

This study reports short-term outcome evaluation results of a school-based peer counseling (PRC) program as implemented by North Forsyth High School during January 1990-February 1991. The PRC program is evaluated in terms of: 1) the content of the program as evaluated by the PRC trainees; 2) the quality of the PRC instructor as perceived by these trainees; 3) several instrumental objectives deduced from the PRC training program; and 4) the degree to which the program was able to reach other students at North Forsyth during the time the PRC was in operation. In the absence of a control group, the treatment effect of the program is determined in terms of individual growth curve models. Through the multiple evaluation designs employed in this research, it has been learned that a significantly larger proportion of students in the PRC group had more favorable attitude toward the content of the PRC program when compared to the content of what they learned from their regular school curricula. Furthermore, a significantly larger proportion of students in the PRC program reported more favorable attitude toward their instructor than their regular school teachers. The PRC program was able to elicit an improvement of trainee self-esteem (p less than .01) and their social values (p less than .05) between pre and post of the Peer Training Evaluation Instrument. In terms of client access, 18.4 percent of the totality of potential clients at North Forsyth have been reached through sixty-one peer counselors sanctioned by the school.     

Register data in the evaluation and program planning of addiction treatment programs: Using Sweden as an example

Data from large-scale registers is often underutilized when evaluating addiction treatment programs. Since many programs collect register data regarding clients and interventions, there is a potential to make greater use of such records for program evaluation. The purpose of this article is to discuss the value of using large-scale registers in the evaluation and program planning of addiction treatment systems and programs. Sweden is used as an example of a country where register data is both available and is starting to be used in national evaluation and program planning efforts.The article focuses on possibilities, limitations and practicalities when using large-scale register data to conduct evaluations and program planning of addiction treatment programs. Main conclusions are that using register data for evaluation provides large amounts of data at low cost, limitations associated to the use of register data may be handled statistically, register data can answer important questions in planning of addiction treatment programs, and more accurate measures are needed to account for the diversity of client populations.