An evaluation model for school psychological services

An evaluation model is presented which provides a structure and a process for the evaluation of school psychological services. The structure incorporates psychological service areas along required and desired service dimensions; information to evaluate a school psychological service program is gathered from school files and interviews with students, parents, educators, and psychological services personnel. The method involves a five-stage process which may lead toward program modification efforts and subsequent re-evaluation. Further, the need for evaluation models in school psychology and for psychological service evaluation is presented and discussed.     

Evaluation approaches in mental health: Implications of the new federalism

Since the introduction of federal mental health legislation in 1963, there has been a changing emphasis on evaluation and accountability. With direct federal funding of community mental health services, accountability demands were met through expectations for local agency evaluation activities which were overseen by federal authorities. The advent of the New Federalism and the shift to block grant funding of mental health services to state mental health authorities have shifted responsibility for evaluation to the states and local programs. This paper reviews federal mental health statutes to trace the extent and locus of required evaluation activities and discusses two approaches to carrying out program evaluation: "top-down" where the evaluation topic, method, and data collection are mandated by an administering or funding body; and "bottom-up" approaches where the subject, method of study, and data to be collected are developed in response to a felt need at the local agency level. A case study of each approach as used at the state level in mental health is examined. Based on the literature and the case studies, conclusions are presented on the pro's and con's of each method in meeting accountability demands and the barriers which must be overcome for either method to be successful.     

Challenges related to the analytical process in realist evaluation and latest developments on the use of NVivo from a realist perspective

ABSTRACT

Realist evaluation (RE) is a research design increasingly used in program evaluation, that aims to explore and understand the influence of context and underlying mechanisms on intervention or program outcomes. Several methodological challenges, however, are associated with this approach. This article summarizes RE key principles and examines some documented challenges and solutions when analyzing RE data, including the development of Context-Mechanism-Outcome configurations. An analytic method using NVivo features is also presented. This method makes it possible to respond to certain analytic difficulties associated with RE by facilitating the identification of patterns and ensuring transparency in the analytical process.     

Assessing local capacity for health intervention

Because of their location within the practice realm, participatory, community-based public health coalitions offer many lessons about implementing and sustaining local interventions. This paper presents a case study of capacity assessment at the local level. Capacity evaluation methods are presented, with emphasis on the theoretical framework used to guide the evaluation. The capacity evaluation framework presented herein was theoretically based and designed to generate practical information to facilitate the adoption of a locally tailored youth obesity prevention program, VERB Summer Scorecard (VSS). Using multiple methods, four aspects of community capacity were assessed, including community, knowledge and skills, resources, and power. Within each category, factors that facilitated or impeded program implementation were distinguished. The evaluation protocol was designed to generate information increasing community capacity to sustain a community-based obesity prevention program. Capacity tables were used as a program-planning tool and as a system for sharing implementation and sustainability requirements with potential adopters. This case study also explores how to use capacity assessment results to empower coalitions to serve as catalysts for development of local programs in other communities.     

The use of evaluation models for decision making: Application to the community anti-crime program

Evaluations are typically designed to provide useful information, including measures of program effectiveness. Although this information is intended to be useful, the general literature on evaluations suggests that instances in which evaluations are explicitly used for decision making are rare. This article discusses the use of decision-theoretic evaluation models, and their applicability in practice as well as in principle. A case study is presented from the evaluation of LEAA's Community Anti-Crime Program, in which a decision analysis based on a decision-theoretic evaluation model influenced a programmatic decision which arose during the conduct of the evaluation.     

An innovative medical and dental hygiene clinic for street youth: Results of a process evaluation

Canada has a noteworthy reputation for high quality health care. Nonetheless, street youth are one of our most vulnerable yet underserved populations. Consequently, a medical and dental clinic was created in downtown Ottawa, Ontario to respond to their needs. The purpose of this study is to describe a process evaluation of the clinic during its first year of operation with a focus on program fidelity, dose, reach, and satisfaction. A mixed methods approach was used involving interviews with providers, focus groups with street youth, analysis of Electronic Medical Record (EMR) data, and supplemental information such as document reviews. The evaluation identified areas that were working well along with challenges to program implementation. Areas of concerns and possible solutions were presented to the management team that then helped to plan and make improvements to the clinic. Our evaluation design and working relationship with clinic management promoted the integration of real-time evidence into program improvements.     

Quality of life in the evaluation of community support systems

As a result of the authors' experience in conducting an evaluation of the community support system (CSS) program in New York State, they have identified five reasons for focusing on quality of life (QOL) as a desired outcome for programs for the chronically ill. These reasons are presented and problems in QOL evaluative research are discussed. Psychological indicators are distinguished from social indicators of QOL, and two methods of operationalizing perceived QOL, the psychological well-being and life domains methods, are examined. A conceptual model is presented and the results of an exploratory study of the QOL of 118 chronic psychiatric patients receiving CSS services are presented.     

Non-formal educator use of evaluation results

Increasing demands for accountability in educational programming have resulted in increasing calls for program evaluation in educational organizations. Many organizations include conducting program evaluations as part of the job responsibilities of program staff. Cooperative Extension is a complex organization offering non-formal educational programs through land grant universities. Many Extension services require non-formal educational program evaluations be conducted by field-based Extension educators. Evaluation research has focused primarily on the efforts of professional, external evaluators. The work of program staff with many responsibilities including program evaluation has received little attention. This study examined how field based Extension educators (i.e. program staff) in four Extension services use the results of evaluations of programs that they have conducted themselves. Four types of evaluation use are measured and explored; instrumental use, conceptual use, persuasive use and process use.Results indicate that there are few programmatic changes as a result of evaluation findings among the non-formal educators surveyed in this study. Extension educators tend to use evaluation results to persuade others about the value of their programs and learn from the evaluation process. Evaluation use is driven by accountability measures with very little program improvement use as measured in this study.Practical implications include delineating accountability and program improvement tasks within complex organizations in order to align evaluation efforts and to improve the results of both. There is some evidence that evaluation capacity building efforts may be increasing instrumental use by educators evaluating their own programs.     

Evaluation of independent-living programs

The program report that states must submit to the federal government under the federal Title IV-E Independent-Living Initiative is, in essence, a program evaluation report and will require comprehensive program and evaluation data to complete. Because the program report will be a key factor in determining the future of this federal initiative, it is important that states improve their information system capability to generate the range of necessary data. This article offers detailed suggestions for ways in which states could establish their independent-living program information systems to produce program and evaluation data; and presented two examples of approaches to evaluating the Title IV-E independent-living program, one in Pennsylvania and another in Texas. These states were selected for the purpose of illustrating two different approaches and not necessarily for the presentation of exemplary evaluation practices.     

Skill acquisition and client movement indices. Implementing cost-effective analysis in rehabilitation programs.

This article outlines the development and implementation of a cost-effective approach to client and program evaluation. The indices presented summarize a client's acquisition of behavioral skills (Skill Acquisition Index) and progression along a continuum of becoming less dependent and more productive (Client Movement Index). Data are presented summarizing how the two non-monetary outcome measures have been used in formative program evaluation. The paper also discusses some of the problems involved in implementing cost-effective analyses in rehabilitation programs and suggests specific strategies to overcome the problems.     

Evaluation of the RAINN National Sexual Assault Online Hotline

ABSTRACT

This article describes the evaluation of rainn National Sexual Assault Online Hotline, a new model for delivery of rape and sexual assault crisis services through a secure, confidential chat-based online hotline. Outcome data from 623 visitors and 54 volunteers for the first 11 months of the program are presented. The results indicate that the model is viable and useful for approximately 80 percent of visitors, and volunteers are satisfied with the program. Recommendations for program development and further research are presented.     

New Curriculum Policy Statement: The Problem of Professional Cohesion–3

This paper presents the Community Learning Center Project, a multicultural training program, as a successful model for the acquisition of culturally relevant social work practice competencies. Cultural pluralism is presented as the underlying framework for the project. The design and model of an evaluation study of the project is described and mechanisms for the implementation of such a culturally relevant effort are suggested.     

Pre- and post-planned evaluation: Which is preferable?

This paper compares the advantages and disadvantages of the “pre-planned” and “post-planned” approaches to evaluating program effectiveness. These evaluative approaches are compared along a number of dimensions which include: (a) Reliability of data and the cost of collecting it; (b) Internal validity; (c) External validity; (d) Evaluation obtrusiveness and threat; and (e) Program goal displacement and program direction. A model designed to help program managers decide when and under what conditions either of these two evaluative approaches should be employed is presented. One major theme throughout this discussion is that despite the growing interest in and use of pre-planned evaluation, the post-planned method has many advantages which often go unnoticed. This paper will help program administrators, planners and evaluators in selecting and implementing these two methods in light of their respective strengths and limitations.     

Using multiple lines of evidence and independent teams to enhance the credibility of evaluations

The utilization of evaluation findings is an ongoing issue among evaluators. This article presents an approach to conducting evaluations that can be used to enhance the credibility of findings and to facilitate the production of adequate documentation in support of evaluation findings. The concepts of multiple lines of evidence and the use of independent teams are discussed. Use of this approach is expected to improve the utilization of evaluation findings.Two program evaluations that were undertaken at Canada's federal department of Consumer and Corporate Affairs are presented. To undertake the data collection and analysis, these studies were broken into separate “evaluation modules, ” each one undertaken by an independent team. The teams reported to an in-house study director, responsible for weighing the evidence and findings of each of the teams and writing the final evaluation report.     

Post hoc procedures for planning and evaluation.

A model designed to isolate, in post hoc fashion, current program services is presented. Concepts from multidimensional scaling are united with program planning techniques to yield an approach which allows for the specification of the five basic dimensions of program (Clients, Problems, Services, Professionals, and Facilities). These five dimensions are also employed in priority setting procedures. Discussion of the application of the model in the field of program planning and evaluation is presented.     

Making bullying prevention a priority in Finnish schools: the KiVa antibullying program

The KiVa antibullying program has been widely implemented in Finnish comprehensive schools since 2009. The program is predicated on the idea that a positive change in the behaviors of classmates can reduce the rewards gained by the perpetrators of bullying and consequently their motivation to bully in the first place. KiVa involves both universal and bullying specific actions to prevent the emergence of new cases of bullying, stop ongoing bullying, and reduce the negative consequences of victimization. The program has been evaluated in a randomized controlled trial involving 234 Finnish schools and during broad dissemination across Finnish schools (the evaluation involving almost one thousand schools) with positive findings. The program content and the implementation model are presented in this article, and the findings from the evaluation studies are summarized.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Conflict of interest in the evaluation and dissemination of "model" school-based drug and violence prevention programs

Conflict of interest refers to a set of conditions in which professional judgment concerning the validity of research might be influenced by a secondary competing interest. The competing interest that has received most attention in the literature addressing the prevalence and effects of such conflicts on the practice of empirical research has been that of financial relationships between investigators and research sponsors. The potential for conflicts of interest to arise in the evaluation of drug prevention programs was raised by Moskowitz in this journal in 1993, but to date there has been no attempt made to estimate the scope of this problem. The present study addressed this issue using a sample of “model” school-based drug and violence prevention interventions by first, identifying the types or relationships that exist between program developers and program distributors, and, second, by assessing how many of the evaluations of these programs published in peer-reviewed journals had been conducted by the developers of the programs compared to independent evaluation teams. The data presented indicate that there are relatively few published evaluations that do not involve program developers and that there are few instances in which there is complete separation between the program developer and program distributor. Using the open systems model of the Institute of Medicine Committee on Research Integrity as a framework, it is argued that the culture and norms of the program developer and those of the program evaluator are fundamentally distinct and therefore failure to separate these roles produces high potential for conflict of interest to arise.     

Ethical problems in program evaluation: Advice for trapped evaluators

This paper argues that ethical problems in program evaluation are increased when conflicting or incompatible evaluation models are applied concurrently. Three models are illustrated: an Amelioration Model, generating better information for a program's own decision makers; an Accountability Model, focusing on public data disclosure and citizen participation in evaluation; and an Advocacy Model, in which the evaluation is designed to advance the program's interests in the competition for resources. Ethical problems inherent in each model, and arising from combining models are presented, along with case examples and discussion. The authors suggest several directions that program evaluators can consider to reduce their vulnerability to ethical problems.