Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

Systems engineering: A problem-solving approach to improving program performance

Evaluation methodologies have generally emphasized the measurement and assessment of program merit, as reflected by the achievement of program objectives. Less attention has been devoted to analyzing and explaining the reasons for deficient performance and, more importantly, what needs to be modified to improve performance. While evaluators do identify performance problems, the identified problems are not always analyzed. Consequently, the information reported to decision makers may be incomplete, inconclusive, and of limited utility in the decision-making process. This article describes an evaluation methodology based on problem solving techniques which can be effective and efficient in defining and analyzing problems which impair program performance. The model can be applied in evaluations where the purpose is to provide decision makers with information and recommendations to improve program performance and provide an alternative to those evaluation models which emphasize the assessment of program merit. The model also provides a potentially unique cost/benefit methodology for estimating the potential worth of improving program performance (i.e., solving the problem).     

A collaborative university model for employee wellness

Universities are taking a more active approach in understanding and monitoring employees' modifiable health risk factors and chronic care conditions by developing strategies to encourage employees to start and sustain healthy behaviors. WellBama, the University of Alabama's signature health and wellness program, utilizes a collaborative model in partnership with select colleges and departments to implement strategies to improve employees' health status. The program provides onsite health screenings and assessments, timely health advising sessions, assistance in setting and monitoring individual health goals to promote improved health, and preventive examination referrals.     

A Collaborative University Model for Employee Wellness

Abstract

Universities are taking a more active approach in understanding and monitoring employees’ modifiable health risk factors and chronic care conditions by developing strategies to encourage employees to start and sustain healthy behaviors. WellBama, the University of Alabama's signature health and wellness program, utilizes a collaborative model in partnership with select colleges and departments to implement strategies to improve employees’ health status. The program provides onsite health screenings and assessments, timely health advising sessions, assistance in setting and monitoring individual health goals to promote improved health, and preventive examination referrals.     

A response to the BPA evaluation plan: Some modest but utilitarian considerations

In response to the complex and expensive Bonneville Power Administration (BPA) evaluation plan, this article attempts to identify some pragmatic issues and alternative approaches to conservation program evaluation. In particular, the need for simpler, lower-cost evaluation options and the importance of process evaluation to program management are emphasised. This is followed by the delineation of several key principles to be observed in conducting a more modest, but still methodologically sound and managerially useful, evaluation of energy conservation programs.     

Applicability of multiple matrix sampling to estimating effectiveness of educational programs

When evaluating the effectiveness of an educational program, a sampling strategy known as “multiple matrix sampling” is particularly effective and efficient when the goal of the evaluation is estimating group (as opposed to individual) performance. Although this technique involves sampling both program participants and items and, subsequently, different participants being assessed over different sets of items, the results may be used to estimate those which would have been obtained had all participants been administered all items. The technique is here described in some detail, with advantages and disadvantages discussed, and examples of its application are given.     

Social policy evaluation: An example from drinking age legislation

This article represents an exploration into issues relevant to the application of evaluation methodologies and theories to the social policy arena. A number of evaluation approaches and issues are explored in the context of an illustrative policy example — that of the legislative changes in minimum legal drinking age in the state of Michigan. This effort is directed towards the goal of furthering the solution of social problems through effective involvement of evaluation strategies in social policy development.     

Experimental and contextual models of program evaluation

Evaluation research is one of the most rapidly evolving fields of applied behavioral science. As demand for program assessment has increased, the number of alternative evaluation approaches has also grown. As a result, everyday practitioners have often lacked sufficient guidelines for the choice of appropriate evaluation strategies.The present paper articulates an underlying epistemological distinction between (a) experimental evaluation models which simplify program realities in generalizable analyses of discrete causes and effects, and (b) contextual evaluation models which holistically examine particular program operations. These two evaluation approaches are directed at different purposes and are applicable to different program settings. A topology of program characteristics (breadth of goals, scope of treatment, specificity of results, and clarity of theory) is developed and linked to the appropriateness of experimental and contextual evaluation.     

Tailored program evaluation: Past, present, future

This paper discusses measurement issues related to the evaluation of computer-tailored health behavior change programs. As the first generation of commercially available tailored products is utilized in health promotion programming, programmers and researchers are becoming aware of the unique challenges that the evaluation of these programs presents. A project is presented that used an online tailored health behavior assessment (HBA) in a worksite setting. Process and outcome evaluation methods are described and include the challenges faced, and strategies proposed and implemented, for meeting them. Implications for future research in tailored program development, implementation, and evaluation are also discussed.     

Standard setting: the crucial issues. A case study of accounting & auditing

A study of standard-setting efforts in accounting and auditing is reported. The study reveals four major areas of concern in a professional standard-setting effort: (1) issues related to the rationale for setting standards, (2) issues related to the standard-setting board and its support structure, (3) issues related to the content of standards and rules for generating them, and (4) issues that deal with how standards are put to use. Principles derived from the study of accounting and auditing are provided to illuminate and assess standard-setting efforts in evaluation.     

Quality of life indicators and program evaluation

Social indicators and program evaluation research have much in common. Quality of life work specifically raises the question of objective versus subjective variables, aggregate data versus distributional and individual issues, and the reference standards used in judgment and evaluation decisions. This article presents perspectives on these issues from the orientation of quality of life research and proposes that these approaches have relevance for program evaluation work.     

Skill acquisition and client movement indices. Implementing cost-effective analysis in rehabilitation programs.

This article outlines the development and implementation of a cost-effective approach to client and program evaluation. The indices presented summarize a client's acquisition of behavioral skills (Skill Acquisition Index) and progression along a continuum of becoming less dependent and more productive (Client Movement Index). Data are presented summarizing how the two non-monetary outcome measures have been used in formative program evaluation. The paper also discusses some of the problems involved in implementing cost-effective analyses in rehabilitation programs and suggests specific strategies to overcome the problems.     

Stakeholder management and program planning in German public theaters

Taking the example of a German public professional theater, this study is the first to analyze how stakeholder interests are managed in the process of program planning. We address the following questions: (1) What are the goals and interests of the most influential stakeholders? (2) How are these goals and interests reflected by theater management when deciding the season's program? (3) Which strategies does theater management apply to balance conflicting stakeholder goals and interests? Results of our qualitative study identify five decisions to be made with regard to program planning: selection of the season's theme, plays, stage directors, cast, and chronological order of productions. Four strategies of balancing conflicting stakeholder goals and interests are identified: setting priorities, combining stakeholder interests, focusing on neutral goals, and developing audiences.     

Evaluation in the community mental health centers program: A bold new reproach?

The Federal Community Mental Health Centers Program (CMHC)-from 1963 to 1981-was heralded as a revolution in mental health care. Championed by many, and severely criticized by others, the actual impact of the program on the nation's mental health remains unclear. The authorization to evaluate the CMHC Program came originally from congressional legislation (PL 90-174), and later from the policies and regulations of NIMH under a series of Federal laws, notably PL 94-63. From 1976-1980, two dominant evaluation strategies were prevalent: funds expended by NIMH each year for studies of CMHC services or program-wide evaluations, and a much larger expenditure by CMHCs to conduct their own, independent evaluations following federal guidelines. As the Center's Program was turned over to the states in the form of block grants (PL 97-35), a group of professionals involved with setting and carrying out federal CMHC evaluation policy of both varieties met in public forum to debate the impact of these two evaluation approaches. While some participants cited gains in evaluation technology and impact upon local management of CMHCs, others found the lack of a coordinated and systematic approach to evaluating the CMHC Program to have been an opportunity missed. The impact of CMHC evaluation efforts are also discussed in terms of their major contribution to the field of evaluation research as a whole.     

Accountability revisited: the arrival of client outcome evaluation

In the past few years, "accountability" for public mental health programs has become differentiated in the minds of not only program evaluators, but also program managers and funders, including state and local-level legislators. Increasingly, these officials are becoming concerned with more than just the numbers and targets of services delivered, and the cost involved, and are looking for evidence of positive outcome or impact on clients to justify program implementation and maintenance. This represents a significant move beyond the two accountability models that most recently seemed to be the focus of most formal accountability efforts--performance measurement and quality assurance. Pressures for implementing these two alternatives seem to have been reduced somewhat by the new federal Administration, but even prior to its advent there had been a rapid escalation in awareness of and concern for client outcome measurement among important audiences, including state and local mental health policy-makers and the U.S. Congress. This presents a major new opportunity and challenge for program evaluators at this new accountability focus continues to gather momentum.     

“Dirty” research in “real” places: A practitioner's guide to program evaluation in the human services

Traditional models of evaluation have often fallen short of their potential for improving services to clients. Using illustrations from a study of client turnover in group homes for troubled youth, this paper discusses practical ways to increase the impact of program evaluation in human service settings.     

A Preliminary Evaluation of the 1-2-3-Magic Parenting Program in an Australian Community Services Setting

Cognitive-behavioural parenting programs provide caregivers with alternative strategies to harsh disciplining practices but the choice of programs that are evidence-based in Australia is limited. Australian community service workers suggest that having a choice of evidence-based programs is important when matching programs to client needs, and that 1-2-3 Magic is a program used widely in Australia despite its small Australian evidence base. The efficacy of the 1-2-3 Magic program has previously been shown in a controlled trial in an Australian university research setting but little is known about its effectiveness in a “real-world” setting. This study examined a three-session group-format of the Australian 1-2-3 Magic & Emotion Coaching program in a typical metropolitan community service setting. Thirty-eight caregivers with children aged 2–6 years reported a significant decrease in disruptive child behaviour (ECBI), permissive parenting (PS), and parental depression and stress (DASS) from pre- to post-intervention, with results maintained at 3-month follow-up. While these findings suggest that a brief 1-2-3 Magic program is beneficial, it is important to note that methodological limitations (intervention-group only study design and possible confounding variables) do not allow ruling out alternative causes for these improvements. Further research will be needed to confirm that this program can generalise to “real-world” settings.     

Neighborhood perceptions and parent outcomes in family based prevention programs for Thai adolescents: the role of program engagement

Due to concerns over Thai adolescent risky behaviors, effective prevention strategies are needed. Determining the role neighborhood context plays in program engagement and outcomes may inform these strategies. This study includes 170 mother-adolescent pairs (M = 13.44, SD = .52) in Bangkok, Thailand in a prevention program for adolescent substance use and sexual risk. Neighborhoods were related to engagement, which was critical to outcomes. Neighborhood disorganization was related to confidence in program effects and program completion. Completion was related to increased ATOD communication. Neighborhood cohesion was related to less program enjoyment, while neighborhood social control was related to more enjoyment. Enjoyment was related to increased ATOD communication and formation and monitoring of alcohol rules. Prevention strategies should focus on neighborhood contexts and enhancing engagement.     

The conceptualization and measurement of quality of life: implications for program planning and evaluation in the field of intellectual disabilities

The concept of quality of life (QOL) is increasingly being used in the field of intellectual disabilities as a conceptual and measurement framework for program planning and evaluation. This article describes the development of a QOL conceptual and measurement framework, and summarizes how this framework is currently being used both nationally and internationally to assess and report personal QOL-related outcomes, to guide quality improvement strategies, and to evaluate the effectiveness of those strategies. Implications of such use are discussed, including those related to understanding mental models, developing internal data systems, supporting organization change, and building on current public policies. The article concludes with reference to the evolving nature of the QOL concept and the impact of this on model development and transdisciplinary research.