A Computer-Integrated Approach to Program Evaluation

Abstract

Outcome evaluation of group work intervention in residential agencies. Development of a management/clinical information system and related computer applications.

Program evaluation as discussed in this article describes a service delivery-focused approach to program evaluation which is actually designed and implemented in a residential treatment agency for delinquent youth, the evaluation approach represents a practical strategy which applies much of the current thinking and research in the area of user focused evaluation. The approach has been refined by incorporating the product of the authors' experiential learning and pragmatic adaptations which evolved out of field testing the model over several years.

The authors present the overall approach to evaluation and the development of the evaluation model within an agency context. The author's then describe the development of a computer-based system which is designed to serve as both a management information system and a clinical information system. Basic aspects of the computer-based evaluation system are presented. Throughout the article, the authors discuss process-related issues pertaining to the introduction of evaluation to both management and staff and movement toward the implementation of an automated evaluation system.     

Moving Toward Collaboration

Summary

Traditionally, organizations serving children and families have focused service delivery by available funding stream criteria. Federal funding streams supported fragmented services by tightly channeling monies to specific programs for specific needs. The intensity of providing and improving the delivery of services has overshadowed building connectedness across organizations and systems. Service has been the major goal and intense effort has gone into maximizing opportunities and measuring effects through service frequency. Over the past several years, new funding incentives have provided the opportunity for new collaboratives. This article describes an innovative collaboration between the Idaho Department of Health and Welfare (IDHW) and Eastern Washington University (EWU) and the unique directions and support that a university/agency partnership can provide for both organizations. Key features of this collaboration include shifts in funding and staffing strategies that contributed to more flexible services and increased levels of collaboration between IDHW, EWU and other community and state organizations and institutions. This article describes how funding can be viewed as a tool to increase the level of collaboration between systems, thus potentially leading to a breakdown of the traditional service delivery system. Finally, this article describes how an agency/higher education partnership played a key role in documenting the success of a school based program in meeting the emergency assistance needs of children and families, and how program evaluation, like funding requirements, can provide a supportive role in building collaborative relationships.     

A Tracking and Notification System to Support the North Carolina JOBS Evaluation

Abstract

An integral part of the North Carolina Job Opportunities and basic skills (JOBS) Program evaluation is the longitudinal tracking study. The JOBS Traking and Notification System (JTNS) is a computer program developed to automate administration and tracking in the 12 evaluation counties. The JTNS notifies researchers of survey administration dates, tracks surveys, updates participant status, reports county performance, supports data entry from survey, and supports numerous other administrative tasks. The notification program has helped the counties keep abreast of their progress without the burden of the administrative overhead necessary for this evaluation. The system has performed well and has heightened theinterest of many counties concerning automation.     

An Evaluation of a Community Collaboration Approach to Psychosocial Rehabilitation

ABSTRACT

This article reports on the results of a multiagency collaboration demonstration project that was designed to replicate the salient features of an exemplary practice, single-agency program of psychosocial rehabilitation for persons with serious mental illness (SMI). The community collaboration extended over two phases for 2 years and involved over 20 mental health service providers. The demonstration project itself was designed to be a replication of the Village Integrated Services Agency (Village ISA) of Long Beach, California. Additionally, this article reports the results of a rigorous evaluation of the project. The treatment group outcomes were substantially superior to all control group measures at or beyond the 0.05 level of significance. With due regard for limitations imposed by a small sample size (n = 50), the project demonstrates that the replicated model of intervention can be effectively implemented through a collaborative organizational structure involving a coalition of agency service providers.     

The Making of COBRS 2.0

Abstract

Recently introduced advances in microcomputer technology offer opportunities for improving child welfare agencies' access to powerful case assessment and program evaluation tools. This paper discusses the process of fitting a child behavior outcome measurement system to a menu driven microcomputer program. Traditional strategies for measuring outcome effectiveness, managerial priorities and operational constraints are examined in terms of their bearing on the introduction of new microcomputer technology. The developmental process is traced to show how these factors influenced the addition of a number of user options to the basic program to enhance its adaptability to specific agency needs. Initial user feedback indicated general satisfaction with the basic program and the applicability of its output, little or no experimentation, as yet, in adapting the system, and, a minor need to adjust the sequencing of operator activities to better fit the realities of agency work rhythms.     

Aligning research and practice: The role of academic-community partnerships for improving measurement and process

ObjectiveCommunity programs addressing social determinants of health are growing in prominence and are increasingly expected to provide metrics of success. Our objective is to assess the role of an academic-community partnership for a community health worker program targeting social and medical needs, and determine factors impacting its effectiveness.MethodsWe draw on a 4.5-year partnership that includes both quantitative and qualitative data collection and analysis. Quantitative data collection mechanisms evolved as a result of the partnership. Qualitative interviews were conducted with community health workers and leadership.ResultsTo align medical and social support services in a sustainable and measurable manner, our academic-community partnership found that creating and maintaining a mutually beneficial space through small wins enabled us to then address larger problems and needs. Ongoing self-study and process evaluation allowed quick adjustments. Unique partnership elements such as having consistent funding and flexible timelines and objectives were essential.ConclusionsWhen integrating health and social services, academic-community partnerships create pathways for bidirectional learning than can quickly turn research into practice and support sustainability, especially when based on incrementally built trust and a history of small wins.     

Multimedia Interactive Training Development–Journey: Discovering Social Services CD-ROM

SUMMARY

This article describes a multimedia 9-module training program which uses text, graphics, animation, narration, and video as well as a planned approach to the use of color, pace, tone, setting, and instructional activities to achieve its goal of orienting new workers to the complexity of a public social service agency.     

The Federal GPRA Evaluation Mandate

Abstract

The Government Performance and Results Act (GPRA) mandates inclusion of evaluation in federal agency strategic plans, impacting human service programs receiving federal support. Recommended evaluation methods include flexible ongoing and participatory approaches. Yet social work may not be prepared to appropriately respond to the government's demands for accountability. This paper reviews GPRA and its implications for social work practice, evaluation, and research, offering response strategies. The paper suggests that the creation of a National Center for Social Work Research (S. 70/H.R. 663) could help bridge evaluation gaps, offering a centralized mechanism for monitoring evaluation activities and maximizing federal research resources.     

A Joint University—Field Agency: Toward the Integration of Classroom and Practicum

Abstract

The authors describe an innovative program designed to create a joint university and local municipality agency in Israel. The agency was structured to be a teaching laboratory that would offer innovative services for social work clients and a unique field placement for students. Four major goals involved joint outcomes for both university and municipality. These included (a) demonstrating evaluation research as a means of accountability and as a tool for practice; (b) using generic social work methods for work with families; (c) reaching out to difficult and high-risk clients with innovative programs; and (d) using the laboratory as a training center for welfare workers, agency supervisors, and students from all levels of the social work programs. The authors describe and evaluate each of the goals, discussing the implications for teaching and for social work practice.     

Evaluation approaches in mental health: Implications of the new federalism

Since the introduction of federal mental health legislation in 1963, there has been a changing emphasis on evaluation and accountability. With direct federal funding of community mental health services, accountability demands were met through expectations for local agency evaluation activities which were overseen by federal authorities. The advent of the New Federalism and the shift to block grant funding of mental health services to state mental health authorities have shifted responsibility for evaluation to the states and local programs. This paper reviews federal mental health statutes to trace the extent and locus of required evaluation activities and discusses two approaches to carrying out program evaluation: "top-down" where the evaluation topic, method, and data collection are mandated by an administering or funding body; and "bottom-up" approaches where the subject, method of study, and data to be collected are developed in response to a felt need at the local agency level. A case study of each approach as used at the state level in mental health is examined. Based on the literature and the case studies, conclusions are presented on the pro's and con's of each method in meeting accountability demands and the barriers which must be overcome for either method to be successful.     

Interpreting the effectiveness of a summer reading program: The eye of the beholder

In applying a methods-oriented approach to evaluation, this study interpreted the effectiveness of a summer reading program from three different stakeholder perspectives: practitioners from the school district, the funding agency supporting the program, and the policymakers considering mandating summer school. Archival data were obtained on 2330 students reading below benchmark in Grades 2–5. After propensity score matching participants to peers who did not attend the summer program, the final sample consisted of 630 students. Pre-to-posttest growth models revealed positive effects in Grades 2–4 (standardized slopes of .40–.54), but fifth graders demonstrated negligible improvement (standardized slope of .15). The standardized mean differences of propensity score matched treatment and control group students indicated null effects in all grade levels (d = −.13 to .05). Achieving proficient reading performance also was not attributable to summer school participation. Findings underscore the importance of operationalizing effectiveness in summative evaluation.     

Matching AIDS Service Organizations' Philosophy of Service Provision with a Compatible Style of Program Evaluation

Abstract

The purpose of this paper is to outline the unique features of AIDS Service Organizations (ASOs) that should be considered when developing an evaluation strategy. It describes current evaluation approaches used by ASOs which are incongruent with the service philosophy of these organizations and the drawbacks these current evaluation efforts present. It then describes an alternative evaluation approach that better matches ASOs' philosophy of service delivery with the evaluation process. An illustrative case example is used to show how a more participatory approach can be used in an ASO and why such an approach is more likely to produce meaningful data that will be utilized for program development and refinement.     

Evaluation of Adult Protective Services: Perspectives of Community Professionals

ABSTRACT

Due to challenges associated with client characteristics, funding, and the expectation of poor outcomes, few studies have evaluated the services provided by Adult Protective Services (APS). The article reports community-based professionals' satisfaction with APS when investigating cases of suspected elder abuse. Results from a 12-item quantitative survey indicate respondents were most satisfied with the intake process, followed by assessment, and case management. Satisfaction was higher than expected, though areas for improvement are identified. This survey and its results should serve as a starting point for more extensive evaluation methodologies needed to enhance the delivery of services provided by APS.     

Evaluation of the RAINN National Sexual Assault Online Hotline

ABSTRACT

This article describes the evaluation of rainn National Sexual Assault Online Hotline, a new model for delivery of rape and sexual assault crisis services through a secure, confidential chat-based online hotline. Outcome data from 623 visitors and 54 volunteers for the first 11 months of the program are presented. The results indicate that the model is viable and useful for approximately 80 percent of visitors, and volunteers are satisfied with the program. Recommendations for program development and further research are presented.     

Meeting the Challenge of Preparing Social Workers for Integrated Health Practice: Evidence From Two MSW Cohorts

ABSTRACT

Changes in health care policy have led to an expansion of integrated care models that rely on collaboration among interprofessional health teams. Recent federal funding has encouraged the development of innovative training models to prepare social workers for integrated health practice. This article presents evidence from the first two MSW cohorts of the Integrated Mental and Behavioral Health Training Program funded by the Health Resources and Services Administration. Twelve students completing the program in 2014 (n = 5) and 2015 (n = 7) participated in evaluation activities at the time of program completion and at 1-year postgraduation. Findings highlight key components of the training program instrumental in student attainment of program goals and competencies. Implications for MSW education and training for practice in integrated care environments are discussed.     

Towards Program Evaluation for Practitioner Learning: Human Service Practitioners' Perceptions of Evaluation

Abstract

While human service practice improvement is acknowledged as an important outcome of program evaluation, much evaluation is narrowly focused on accountability requirements. This type of evaluation often has limited use and relevance to human service practitioners. In exploring forms of evaluation that do have practice relevance, this article presents the experiences and perceptions of human service practitioners subject to evaluations that set out to foster learning. Across the cases participants expressed concerns about data quality and credibility, the relevance of evaluation data to the program context, the invasiveness of the evaluation, management of the process (particularly limited consultation with practitioners), and the lack of effective dissemination. These cases illustrate the challenges for evaluation to be useful to practice, and to foster meaningful improvements to services.     

Understanding innovation in health program planning and development

IntroductionA need for innovative public health programs is evident as the field adapts to address changes in health priorities and target populations. The Innovative Teen Pregnancy Prevention Program (iTP₃) was created to support and enable innovation in teenage pregnancy prevention, developing programs to reach the most at risk youth.MethodsA formative evaluation was conducted to understand what innovation means in the context of program development, and examine the process of innovation. Qualitative data was collected through baseline interviews with program development teams, referred to as Innovators, prior to the start of the project period and follow-up interviews conducted at the end of a 12-month funding period. Additional open-ended written responses were collected in the middle of the funding cycle. A thematic analysis with an open-coding scheme was used to identify emergent themes.ResultsInnovators considered programs innovative because of the target population of focus, program delivery mechanism, and/or program development approach. They specifically identified that a “culture” of innovation must be present if new programs are to be developed. Over time, Innovators began to shift their definition of innovation toward unique design processes and ecological approaches.DiscussionThrough creating a culture of innovation and utilizing systems thinking, this project provides important insights in how to develop innovations in public health.     

Process evaluation of the ‘Singapore Physical Activity and Nutrition Study’

IntroductionThe Singapore Physical Activity and Nutrition Study (SPANS) aimed to improve the physical activity (PA) and nutrition behaviours of Singaporean women aged 50 years and over. The SPANS program consisted of PA classes, nutrition workshops, telephone dietary counselling, health booklets, a health calendar and program ambassadors. This study aimed to assess and understand the implementation of the program strategies and gain insight into process evaluation components to inform future programs.MethodsThe evaluation was guided by a process evaluation framework and collected data via questionnaires (n = 209), program ambassador documentation and exit interviews with program completers (n = 13) and non-completers (n = 12).ResultsIn total, 295 participants completed the program (response rate = 84 %). Participants reported high levels of satisfaction with the overall program (99.5 %) and program activities (96.7 %), and also rated program ambassadors highly. Participation rates were highest for telephone dietary counselling sessions. The main reason for not attending program activities was having a ‘busy schedule’ (n = 158). Participants cited a need for improved recreational centre facilities and increased flexibility around program delivery.ConclusionsThe process evaluation showed that the program strategies were implemented as planned and were deemed suitable for supporting behaviour change among Singaporean women aged 50 years and over. The program reached and involved the majority of participants throughout the six months. The combination of practical educational resources and supportive program ambassadors were key strategies that facilitated positive PA and dietary behaviours. However, there needs to be some flexibility in the delivery of programs. The findings of this research may inform other programs in the region.     

Breastfeeding interventions in Kansas: A qualitative process evaluation of program goals and objectives

Across the state of Kansas, eighteen public health departments received funding through the 2011 Breastfeeding Grant Initiative to start a breastfeeding intervention. The main objective of this study was to evaluate the progress toward program goals and objectives. This study was a process evaluation. Qualitative data were collected from recipient health departments at two time-points during the program year. Structured, open-ended questions were asked through telephone interviews. This study examined: (1) progress toward program goals and objectives, (2) problems encountered during implementation, and (3) evaluation measures employed to assess program impact. All health departments reported making significant progress toward program goals and objectives and reported successful collaboration with other healthcare providers. The use of breast pumps, educational classes, and professional training of staff were reported as providing the best outcome in the promotion of breastfeeding. The majority of respondents did not measure program impact. From a public health perspective, it is important that infants receive breast milk for the first six months of life. It appears that goals and objectives set a priori guided health departments with the administration of their breastfeeding program. Results may be used to enhance and sustain delivery of breastfeeding support programs in Kansas communities.     

Testing Faith

Abstract

The U.S. federal government, through Charitable Choice, has opened public funding for the delivery of social services to faith-based organizations (FBOs) more than ever before. This increased access to governmental funding at all levels has led to a closer examination of the evidence base on the effectiveness of the services provided by FBOs, and the capacity of FBOs to respond to data needs for accountability and program improvement efforts. This paper discusses the current status of evaluation research on FBO services and the emerging data needs among faith-based providers. Promising avenues for enhancing the current understanding of outcomes of FBO services are explored such as (1) adopting outcome measurement practices in use within the current nonprofit sector, and (2) developing more rigorous research designs tailored to the special contexts of faith-based services. The paper suggests an agenda in which FBO operators, funders, and evaluators work together to move forward in improving the evidence base on faith-based services.