Involving constituents in agency priority setting: a case study

Many government agencies have "constituents" who believe the agencies should hear and consider their needs and desires in establishing program priorities. This article describes how one involves interested constituents (consumers, industry, medical professionals, and state organizations) in the agency's annual priority-setting process. This participation has produced a number of beneficial results-some expected and some unexpected.     

Developmental trends in program evaluation

It is suggested that the ontogeny of program evaluators replicates the phytogeny of the profession of program evaluation. A trend across the purposes of program evaluation from program amelioration to program advocacy and ultimately to program accountability is projected.     

A case study in defining developmental costs for quality assurance in mental health center programs

Issues related to the state-of-the-art of ambulatory mental health care review as well as to the resources necessary for planning and operating a quality assurance program may be hampering development of such programs in community mental health centers nationwide. Faced with the problem of responsible and prudent program management, it has become increasingly difficult for CMHC directors to implement innovative, but unproven, and often unpriced programs which have the potential to draw scarce professional resources away from direct service delivery. Pressure to implement CMHC program requirements that are not ordinarily considered revenue-producing in nature, for example, consultation and education services and quality assurance activities, may further create anxiety for mental health program directors, especially during periods when they are ostensibly confronted with the need to maximize CMHC revenues. Before mental health professionals and administrators of mental health programs can be expected to introduce quality assurance programs, they need to have some idea how much the system costs. The figures reported from three North Carolina mental health centers may offer some indication of what other mental health centers might expect to spend in developing a quality assurance program. Salary expenses for multidisciplinary staff participation in quality assurance activities are presented.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Skill acquisition and client movement indices. Implementing cost-effective analysis in rehabilitation programs.

This article outlines the development and implementation of a cost-effective approach to client and program evaluation. The indices presented summarize a client's acquisition of behavioral skills (Skill Acquisition Index) and progression along a continuum of becoming less dependent and more productive (Client Movement Index). Data are presented summarizing how the two non-monetary outcome measures have been used in formative program evaluation. The paper also discusses some of the problems involved in implementing cost-effective analyses in rehabilitation programs and suggests specific strategies to overcome the problems.     

Evaluation of the U.S. Air Force Family Support Center Program

This article provides an overview of the implementation and evaluation of Family Support Centers (FSC) in the U.S. Air Force. After describing the FSC model, the quantitative aspect of the evaluation is discussed, including the evaluation design, implementation methods, survey response rates, and procedures for data analysis. A summary of evaluation results also are presented as well as a discussion of administrative and methodological issues encountered in performing the evaluation. The results of the evaluation suggest that the FSC has become an integral part of the base human service delivery system, especially for married members and civilian spouses.     

An analysis of variables affecting length of stay in foster care

The purpose of this paper was to find out whether the relationship between type of contact with agency and/or child in foster care and average number of contacts over time makes any difference in length of stay in foster care. Four types of contacts were examined: Family—Agency, Family Initiated—Agency, Social Worker—Initiated, and Family—Child. Results indicated that there were critical numbers for each contact type found to be statistically significant that would tend to reduce the length of the child's stay in foster care. The importance of contact during the first month of placement was emphasized.     

Strengths, uses and problems of global scales as an evaluation instrument

A global scale can provided means for integrating a variety of client/patient assessment techniques to complement each other in a useful manner. Global scales have been applied to service planning and evaluation as well as being useful for the on-going communication of clients' clinical/functioning status. Global scales are attractive to many service programs because of their apparent ease of implementation and their apparent face validity. Furthermore, there is an extensive body of literature describing global scales as hightly reliable and valid when properly implemented and maintained. The literature also suggests that there are some serious pitfalls. While initial implementation of a global scale is easy, the maintenance of a reliable, valid and useful scale appears to require their active use in treatment planning, treatment review and clinical supervision processes. It also requires that more extensive multidimensional ratings be made at intake, review and termination. Furthermore, staff training and development sessions two or three times a year are needed to surface and deal with differences in clinician ratings. Given these sorts of supports, then, a global scale has been demonstrated as a useful tool in service program management as well as clinical process and outcome studies, including studies of cost effectiveness. Since the major use of a global scale is as an integrating construct, a quantitative model is offered to describe the relationship between global scale ratings and multidimensional facets influencing the global ratings.     

Referral to treatment: improving client-treatment match through process analysis.

This paper presents an expedient procedure for assessing referral counselor knowledge of the effectiveness of various treatment programs for different client types. Among the advantages of this technique are timely results, ease of application, and usefulness for process and formative evaluation. An empirical example using data from a drug abuse referral program is included to demonstrate one application of the technique. The discussion includes additional examples where the technique would have potential utility.     

Ethical dilemmas in evaluating programs with family court related clients

Two ethical dilemmas are discussed; breaching client confidentiality in order to intervene in the client's illegal behavior, and defining the role of evaluator as expert witness. The evaluation context concerns a combined program development and evaluation project with clients involved in family court proceedings for child abuse or neglect. Different perspectives, approaches, and experiences in resolving each dilemma are reviewed. The conclusion argues that approaches to ethical dilemmas in evaluation projects must go beyond techniques to disguise data, to include an assessment of the values and value commitments connected to the different roles an evaluator must perform.     

Techniques for the analysis of geographic data in evaluation

Although geographic relationships are often an important aspect of a program or system under evaluation, these relationships have frequently been ignored by evaluators. Standard techniques do exist, however, for the analysis of geographic data. This paper presents three such techniques, Geocode Analysis, Trend Surface Analysis, and Social Area Analysis, and discusses their use in evaluation. With these techniques, evaluators can more adequately deal with the questions of geographic relationships which arise in many studies.     

Measuring corporate social performance: The problem of values

Debates over the social performance of corporations have carried with them implicit evaluations of that performance. The apparent subjectivity and bias of these informal evaluations has led to the development of more formalized corporate social performance assessment methodologies — social audits. In this paper it is argued that these social audits are also influenced by subjectivity and bias considerations; and thru the case study of a social audit of a corporate safety program, the underlying value issues at the core of these influences are explored. The value implications of the various social audit methodologies in use are discussed; and several value sensitive stages/elements of social audit research and evaluation research in general are identified. Recommendations set forth for minimizing the negative consequences of value-influenced evaluations are a value-explicit evaluation approach, greater standardization of evaluation procedures, a dialectic evaluation approach, and better informed evaluation consumers.     

A systematic approach to increasing use of management-oriented program evaluation data

With the implementation of Management Information Systems (MISs) in human service delivery settings, came the expectation that managers would make data-based decisions. There is substantial evidence, however, that this has not occurred. The present authors describe a systematic approach to correcting this deficiency. The approach involves a behavioral definition of management information use, the identification of disruptions in the behaviors comprising information use, and an intervention designed to reduce disruptions and promote use. Conclusions about the effectiveness of this approach gained from its implementation in a community mental health center are described.     

Managing the implementation of innovations

Data from a field survey of acceptance and implementation of a new training system are used to assess the validity and utility of a conceptual model of implementation processes. Results demonstrate the model's applicability, provide evidence of its explanatory power, and suggest that the model can be usefully applied to guide implementation planning and monitoring. Such applications are expected to improve the return realized on investment in new technologies, policies, and programs.     

Evaluating Reagan's block grants. Informed opinions on essential issues and needed information resources

In this paper a set of common issues has been proposed to provide a framework for evaluating the effects of current and future block grant programs. By using these issues to identify current information needs and resources, the author attempts to promote the wider use of the available evaluation information on the current block grants and budget reductions, and to identify information gaps in these data bases.     

Inpatient psychiatric treatment of the difficult child: Common practices and their ethical implications

This article reviews and discusses the available research literature pertaining to the prevalence of seclusion and restraint procedures in inpatient psychiatric treatment facilities for children and adolescence. The rationales for use, potential effects, and factors found to influence the application of these more extreme clinical interventions are considered. The author suggests that the supposed therapeutic function of these types of practices has not been substantiated and their implementation can vary widely from one setting to another. Results from several studies suggest that clearly defined criteria for use and more observational studies of prevalence and precipitant factors could considerably refine these procedures and reduce potential abuses. The psychopharmacologic approach to the treatment of difficult children is examined as a sophisticated counteraggression strategy which has yet to demonstrate therapeutic efficacy. These clinical procedures, which are utilized in scores of inpatient settings, generate serious ethical questions for the health professional concerned with the legal rights of those minors labeled as deviant members of families or the society at large.