“Dirty” research in “real” places: A practitioner's guide to program evaluation in the human services

Traditional models of evaluation have often fallen short of their potential for improving services to clients. Using illustrations from a study of client turnover in group homes for troubled youth, this paper discusses practical ways to increase the impact of program evaluation in human service settings.     

Applicability of standards for evaluations of educational programs, projects and materials in an international setting: Qualitative research

Qualitative studies are much more the practice in Europe and increasingly elsewhere than in the United States. For an international audience, therefore, there needs to be much greater emphasis on qualitative techniques than there is in the standards report. The most difficult ethical problem arises from the conflict between the right to privacy of research subjects and the right to know of audiences. By its nature, qualitative research focuses on the particular, the detailed analysis of the specific incident. It is therefore often difficult to conceal the identity of the individual organisations and people concerned. Even a confidential report might work to the detriment of the subjects studied. Published reports pose the same problem. Have subjects the right to have material they believe to their disadvantage excised from a report even if that renders the report valueless? There are two technical points in the standards document that should be amplified. The first concerns the verification of data. We need to specify procedures to ensure credibility for qualitative studies. The second concern is the specification of quantitative techniques for the analysis of qualitative data. The statement in the standards document is too weak. We need to define much more clearly the steps that should be taken to ensure that qualitative data can be tested rigorously. Finally, there is the problem of collusion, the danger that the qualitative evaluators will select from their material the data which are in conformity with their own predilictions or presuppositions, the expectations of their audience, or the requirements of their sponsors. This problem may be acute in some parts of the world. To be of maximum international value, the standards report needs to be extended and to pay much greater attention to problems arising with qualitative evaluation.     

Evaluation of the exercise intervention with the "Back School" education program in a Brazilian company of cigars: a case study

The purpose of this study was to assess the benefits of an exercise program associated to the educational phase of the "Back School Program" in the quality of life of workers from a cigar factory. This is a case study conducted over four months with three activities a week, two times a day, with the third day devoted to educational activities, focusing on healthy lifestyle, correct posture, postural orientations, daily living and work activities, work motivation, and other issues.     

A case study of action research in probation : The improved correctional field services project

This case study of the Improved Correctional Field Services Project describes the pre-implementation history of an action research project—in this case a project to improve probation services. That history encompasses a period of about five years (1974–1978). ICFS represented a major initiative in probation programming by the now defunct Law Enforcement Assistance Administration. It was intended to be a test of some of the principle ideas for making probation more effective. Its history illustrates the pitfalls that may confront a program developer and evaluator. Further, it illustrates how concepts and objectives can change within the life-time of a given project.     

Addressing self-selection effects in evaluations of mutual help groups and professional mental health services: An introduction to two-stage sample selection models

Because random assignment to conditions is often neither possible nor desirable in longitudinal evaluations of mutual help organizations, the influence of self-selection effects must be assessed in order to accurately interpret outcome data. One approach to adjusting for self-selection effects is to control for covariates that predict outcome using statistical procedures such as analysis of covariance (ANCOVA), partial correlations, and hierarchical regression. This approach has considerable power, but is less useful when an evaluator is interested in directly modeling the process of entry into a program and incorporating information on the factors affecting self-selection into estimation of program effects. Two-stage sample selection models are designed to address such situations. These models rely on regression procedures in which program participation is modeled in an initial equation, which yields a sample selection correction factor. The correction factor is included with participation in a second equation that predicts outcome. This two-stage procedure allows the evaluator to interpret the observed effects of a professional service or a self-help group in the context of the magnitude and direction of selection effects. We compare and contrast the covariate control and sample selection models in a longitudinal study of the effects of participation in Alcoholics Anonymous on drinking behavior.     

A program of research with Hispanic and African American families: three decades of intervention development and testing influenced by the changing cultural context of Miami

In this article we summarize work with poor, inner-city Hispanic and African American families conducted at the University of Miami Center for Family Studies. We elucidate ways in which this research program has paralleled the treatment development paradigm and has been responsive to changes in local demographics. Specific cultural issues pertaining to Hispanics and African Americans are discussed in light of treatment development and implementation. Future directions and challenges for working with poor, inner-city minority families are addressed.     

Elder mistreatment in underserved populations: Opportunities and challenges to developing a contemporary program of research

ABSTRACT

This article provides an overview of the status of research on elder mistreatment among underserved populations in the United States, including gaps in our current knowledge base and scientific and structural barriers to growing research on the exploitation, neglect, and abuse of older people from diverse and disadvantaged ethnic/racial, geographic, sexual identity, and socioeconomic groups. High-priority areas in need of new elder mistreatment research with underserved populations are identified, and suggestions are given for how this research can be facilitated by researchers, university institutional review boards, and funding agencies.     

A large-scale initiative to disseminate an evidence-based drug abuse prevention program in Italy: Lessons learned for practitioners and researchers

Across developed countries, experimentation with alcohol, tobacco, and other drugs often begins in the early adolescent years. Several evidence-based programs have been developed to prevent adolescent substance use. Many of the most rigorously tested and empirically supported prevention programs were initially developed and tested in the United States. Increasingly, these interventions are being adopted for use in Europe and throughout the world. This paper reports on a large-scale comprehensive initiative designed to select, adapt, implement, and sustain an evidence-based drug abuse prevention program in Italy. As part of a large-scale regionally funded collaboration in the Lombardy region of Italy, we report on processes through which a team of stakeholders selected, translated and culturally adapted, planned, implemented and evaluated the Life Skills Training (LST) school-based drug abuse prevention program, an evidence-based intervention developed in the United States. We discuss several challenges and lessons learned and implications for prevention practitioners and researchers attempting to undertake similar international dissemination projects. We review several published conceptual models designed to promote the replication and widespread dissemination of effective programs, and discuss their strengths and limitations in the context of planning and implementing a complex, large-scale real-world dissemination effort.     

Validity in an evaluation of Healthy Families Florida—A program to prevent child abuse and neglect

Evidence-based continuums have become a guide for identifying the level of evidence in evaluations of home visiting programs conducted to determine their effectiveness in preventing child abuse and neglect. While randomized controlled trials are required for the highest levels of evidence, quasi-experimental designs have also been specified as an appropriate alternative. Using a quasi-experimental evaluation of a home visiting program to prevent child abuse and neglect that adheres to the Healthy Families America model, this paper describes and illustrates how types of validity can be improved. More specifically, we address how threats to internal validity can be identified and reduced through statistical techniques; how construct validity may be strengthened using state records to measure the outcomes; and how external validity is affected by including or excluding study participants. After applying a variety of statistical adjustments to reduce selection bias, we found that the outcomes favored the home visiting program and increased after accounting for covariates that contributed to child abuse and neglect. This was true across the statistical techniques (traditional covariate and propensity score adjustment) used. For evaluations using quasi-experimental designs, recommendations relevant to the illustrations in the paper are also presented.     

Hospital response to the legalization of abortion in New York State: an analysis of program innovation

The reorientation of hospital services in the state of New York to accommodate women's constitutional right to elective abortion was investigated. Market and resource constraints, the social orientations of the organization, and the values of physicians were examined in the effort to evaluate hospital response between 1971 and 1973. Analysis indicates that program innovation in obstetrical and gynecological services to include elective abortion was inhibited by economic factors that generally determined the feasibility of diverting finite resources to a new service and social orientations and values that determined the compatibility of elective abortions with the dominant values underlying hospital operations. The reform of New York abortion statutes and the subsequent ruling by the Supreme Court reiterating the right of women to terminate pregnancy failed to standardize the delivery of health care so that individual rights to service could be obtained everywhere in the state. The social changes ultimately realized through legislative and judicial action were essentially conditional upon the responsiveness of local health care providers. Legal action that failed to specifically address the administrative role of hospitals in social change qualified local access and could not be completely effective in legitimizing the redefinition of abortion in society.     

Application of mixed-methods design in community-engaged research: Lessons learned from an evidence-based intervention for Latinos with chronic illness and minor depression

The purpose of this paper is to share lessons learned from a collaborative, community-informed mixed-methods approach to adapting an evidence-based intervention to meet the needs of Latinos with chronic disease and minor depression and their family members. Mixed-methods informed by community-based participatory research (CBPR) were employed to triangulate multiple stakeholders’ perceptions of facilitators and barriers of implementing the adapted intervention in community settings. Community partners provided an insider perspective to overcome methodological challenges. The study’s community informed mixed-methods: research approach offered advantages to a single research methodology by expanding or confirming research findings and engaging multiple stakeholders in data collection. This approach also allowed community partners to collaborate with academic partners in key research decisions.     

Bringing experiential educational groups to the United States: an analysis of group development in an international travel and study program

International exchanges and travel and study projects have been gaining popularity in many fields including social work. Embedded in a group context, the success of these initiatives depends heavily on the group process of all members involved. This article analyzes the impact group dynamics have on an international travel and study project that brought 11 doctoral students and their dean from Ethiopia for a one-month experiential education program. A model of group development, which emphasizes behavioral outcomes in groups (forming, storming, norming, performing, and adjourning), is presented as an organizing framework. The discussion draws on a qualitative analytical study, which revisits Tuckman's model, and proposes a revised model that defines group development stages by individual, group, purpose, and work concerns that drive conflict throughout all group stages. Implications for social work education include de-emphasizing stages of group development, being mindful about what is known about small groups, and improving empirical research and training for leading international groups.     

The African Evaluation Guidelines: 2002. A checklist to assist in planning evaluations, negotiating clear contracts, reviewing progress and ensuring adequate completion of an evaluation

A review of the US ‘program evaluation standards’ (PES), undertaken in a series of workshops and meetings of networks of evaluators in Africa, resulted in modifications to those standards. The result was presented to a plenary session of the Inaugural Conference of the African Evaluation Association in September 1999, attended by over 300 evaluators from 35 countries. The AfrEA Conference decided that a systematic effort should be made to produce a list of African evaluation guidelines, similar to the PES, and that this checklist should be reviewed by national evaluation associations and networks in Africa and field tested in several countries. Ten national and regional networks and associations suggested modifications to the text and endorsed the final version of the guidelines.     

Changes in service provision in rural areas. Part 1: The use of GIS in analysing accessibility to services in rural deprivation research

Previous work has highlighted the importance of public service provision in rural areas particularly for potentially vulnerable groups such as the elderly, unemployed and single parents especially where this coincides with a lack of access to either private or public transport. Disadvantage can often be compounded by limited access to services for such groups with the superior resources of more affluent social groups enabling them to respond and adapt to the increasing centralisation and reduction of services. This paper reviews previous investigations of the implications of changing accessibility to services in rural areas and draws attention to the need for a new research agenda which uses spatial analytical techniques to gauge the current levels of (in)accessibility to key services at the community level. These techniques are illustrated in Part 2 of the paper with reference to changes in post office provision in mid Wales, and the policy significance of incorporating geographical information systems-based measures of provision into traditional area-based indicators of disadvantage is discussed.     

A theory based evaluation of an intervention to promote positive health behaviors and reduce social isolation in people experiencing homelessness

ABSTRACT

Homelessness adversely affects an individual's ability to access healthcare, opportunities for social interaction and recreational activities such as physical activity. This study aimed to evaluate the impact of a community-based physical activity and peer support intervention on the health and wellbeing of homeless participants. This study employed semi-structured interviews to investigate the perceived impact of the Street Fit Scotland intervention on the health and wellbeing of 10 homeless adults. Interviews were audio recorded and transcribed verbatim. A thematic analysis was conducted. Participants reported that their health and wellbeing had improved since attending the intervention. This was attributed to improvements in self-esteem, social interaction and mental wellbeing. Participants generally felt that their physical activity had increased since attending Street Fit Scotland, and a number of individuals reported that they were making healthier choices with regards to health behaviors. A theory of change logic model was developed that demonstrated how each component of the intervention influences the observed and intended outcomes. Attendance of Street Fit Scotland had positive effects on participant's health and wellbeing, particularly concerning self-esteem, health behaviors, social interaction, and physical activity. More efforts should be made to evaluate small-scale interventions that are reaching vulnerable population groups.     

Knowledge translation and implementation of housing first in Canada: A qualitative assessment of capacity building needs for an evidence-based program

We examined communities’ expressed needs for capacity building in the implementation of Housing First (HF) for persons experiencing homelessness. The findings are based on thematic analyses of qualitative data obtained from participants (n = 77) in 11 focus groups conducted in seven Canadian cities. We identified capacity building needs in the areas of training (e.g., HF principles, clinical services, landlord engagement) and technical assistance (e.g., intake coordination, client prioritization, fidelity assessment). These findings were used to tailor training and technical assessment (TTA) to the stages of HF implementation in these cities. Limitations and implications for future theory, research, and practice are discussed.