Towards Program Evaluation for Practitioner Learning: Human Service Practitioners' Perceptions of Evaluation

Abstract

While human service practice improvement is acknowledged as an important outcome of program evaluation, much evaluation is narrowly focused on accountability requirements. This type of evaluation often has limited use and relevance to human service practitioners. In exploring forms of evaluation that do have practice relevance, this article presents the experiences and perceptions of human service practitioners subject to evaluations that set out to foster learning. Across the cases participants expressed concerns about data quality and credibility, the relevance of evaluation data to the program context, the invasiveness of the evaluation, management of the process (particularly limited consultation with practitioners), and the lack of effective dissemination. These cases illustrate the challenges for evaluation to be useful to practice, and to foster meaningful improvements to services.     

Can a workbook work? Examining whether a practitioner evaluation toolkit can promote instrumental use

In large-scale, multi-site contexts, developing and disseminating practitioner-oriented evaluation toolkits are an increasingly common strategy for building evaluation capacity. Toolkits explain the evaluation process, present evaluation design choices, and offer step-by-step guidance to practitioners. To date, there has been limited research on whether such resources truly foster the successful design, implementation, and use of evaluation findings. In this paper, we describe a multi-site project in which we developed a practitioner evaluation toolkit and then studied the extent to which the toolkit and accompanying technical assistance was effective in promoting successful completion of local-level evaluations and fostering instrumental use of the findings (i.e., whether programs directly used their findings to improve practice, see Patton, 2008). Forensic nurse practitioners from six geographically dispersed service programs completed methodologically rigorous evaluations; furthermore, all six programs used the findings to create programmatic and community-level changes to improve local practice. Implications for evaluation capacity building are discussed.     

Technology as a Weapon in Domestic Violence: Responding to Digital Coercive Control

ABSTRACT

Technology-facilitated domestic violence is an emerging issue for social workers and other service providers. The concept of Digital Coercive Control (DCC) is introduced to highlight the particular nature and impacts of technology-facilitated abuse in the context of domestic violence. While practitioners have become more adept at working with women experiencing DCC, there is still little known about its dynamics and whether this violence requires a change in current service responses. This article explores findings from survey research conducted with 546 Australian domestic violence practitioners about the ways perpetrators use technology as part of their abuse tactics. The findings demonstrate that DV practitioners believe perpetrator use of technology is extensive and has significant impacts on the safety of clients. A major dilemma faced by practitioners is how to promote and facilitate client safety from DCC while still enabling safe use of technology so clients can remain connected to family, friends, and community.

IMPLICATIONS

• The use of digital technology in domestic violence creates a significant practice issue for Australian domestic violence practitioners.

• The development of a practice framework for responding to digital coercive control may assist practitioners to highlight the risks posed by this abuse, while still enabling women and children the freedom to participate in the digital realm.

     

Community Partnerships

Abstract

Community challenges force human service agencies to collaborate in providing services. Such collaborations require practitioners to have skills not found in mainstream social work curricula. This paper explores how a new MSW program evolved through dialog with community leaders and resulted in a curriculum with a sole concentration of community partnerships.     

Experience With a Drug Screening Program at a School of Pharmacy

Abstract

Substance use and abuse among pharmacy students is a concern of pharmacy schools, boards of pharmacy, and training sites alike. Pharmacy students must complete approximately 30% of their academic coursework in experiential settings such as community pharmacies, hospitals, and other health systems as part of any accredited pharmacy school's curriculum, and these training sites are starting to require drug testing of pharmacy students as part of their contractual agreements with schools of pharmacy. The authors describe the implementation of a mandatory random urine drug screening program at their school as well as the changes that occurred owing to assessment of the program. The authors report the basic results to date of the drug screening program. The authors also speculate on secondary benefits of the drug screening program. Finally, the authors describe current and future evaluations that they are undertaking regarding this program.     

Adolescents' Self-Reported Substance Risks and Need to Talk About Them in Mental Health Counseling

Summary

Although adolescents seeking mental health services may not consider them related, the prevalent co-morbidity of substance use and mental health problems makes it imperative that practitioners find ways of engaging troubled youth about substance issues. Based on their responses to a clinical self-assessment questionnaire (Adquest), this study shows that mental health service applicants are willing to disclose their substance use behaviors and are open to talking about them to intake workers. Age and gender differences in substance use patterns and willingness to talk are explored.     

Using network analysis to conduct a system-wide program evaluation within a university

ObjectiveTo conduct a system-wide assessment using social network analysis (SNA) to examine how 14 important issues (e.g., consent; racism) are addressed through education, training, and programming at a university.MethodsEvaluators conducted interviews with campus departments responsible for educating/training on the 14 issues. Interviews revealed which programs (n = 52) were offered that addressed the 14 issues, and data on audience characteristics, date of delivery, and which issues were covered in each program were collected. SNA was used to calculate degree and create visualization graphs illustrating patterns of content-coverage across all 52 programs.ResultsThe average degree was 19.38 (SD = 9.70), meaning programs overlapped in topic area with nearly 20 other programs, on average. Most programs (n = 36; 69.2 %) were attended by audiences of 500 people or less. “Diversity and inclusion” represented the topic area with the most programs (n = 23), whereas “suicide” and “bullying/hazing” had the least number of programs (n = 3). Degree was negatively correlated with attendance numbers (r=-.310, p < .001), indicating the more a program overlapped in content with other programs, the smaller the audience.ConclusionsThis study supports the use of network analysis in conducting systemic evaluations of programs offered at a university, complementing the work of ongoing, local-level program evaluations.     

IMPACT–“Increasing the Impact of Assistive Technology”

Abstract

Human service professionals are increasingly pressured to use sophisticated data analysis tools to support service decisions. However, the application of these tools often involves assumptions and nuances that are difficult for the practitioner to evaluate without specialized information. This article helps the practitioner evaluate two different quantitative methods, a logistic regression and a neural network. Both were used on the same data set to develop a model for predicting employee turnover in a regional child protective services agency. The different steps of building and enhancing the model were discussed. Ultimately, the neural network was able to predict turnover more accurately than a logistic regression by only 1%. The article provides advice to practitioners on comparing, evaluating, and interpreting logistic and neural network tools.     

Combining Primary Prevention and Risk Reduction Approaches in Sexual Assault Protection Programming

Abstract

Objective: The object of this study is to extend prior evaluations of Elemental, a sexual assault protection program that combines primary prevention and risk reduction strategies within a single program. Participants and Methods: During 2012 and 2013, program group and control group students completed pretest, posttest, and 6-week and 6-month follow-up surveys assessing sexual attitudes and knowledge as well as experiences with assault. Results: The results reinforce previous findings that Elemental is effective in reducing sexual assault risk. Program effects were both direct, in that participation was associated with lower risk of assault, and mediated, in that participation impacted attitudes and beliefs that are empirically linked to risk of later assault. Conclusions: By combining both primary prevention and risk reduction approaches, Elemental is not only effective at reducing incidences of assault, it is also consistent with a number of recent recommendations for directions in sexual assault prevention programming.     

Newly Educated MSW Social Workers’ Use of Evidence-Based Practice and Evidence-Supported Interventions: Results From an Online Survey

ABSTRACT

Social work programs have increased evidence-based practice (EBP)-related curricular content; yet, it is unclear whether newer practitioners are successful in incorporating it into their practice. This study examines factors that influence the use of EBP among newly graduated social workers. We developed and implemented a Web-based survey that assessed individual, school, and agency factors that predict implementation of EBP among a sample of master’s of social work practitioners post-graduation. Multiple regression analysis resulted in five factors that were significantly associated with use of EBP: confidence with EBP; feeling that EBP is essential to practice; organizational culture emphasizes EBP; field placement/internship emphasizes EBP; and faculty emphasizes EBP. Findings suggest that individual and organizational factors may be more important in determining use of EBP than educational program factors alone.     

Service-Learning in Parenting Education

Abstract

The field of family studies is particularly suited to service-learning, but one area that has not received much attention in this regard is that of parenting education. This article highlights a series of parenting education workshops that were incorporated into a parent-child relationships course, utilizing a developmental assets framework. Student reflections and parent evaluations are used to consider the extent to which the service-learning activity enabled students to apply course content to real-world experiences, view themselves as emerging professionals, and have a positive impact on the community. Implications for continuing or replicating similar service learning experiences are included.     

Teaching Practice Methods Using Interactive Television: A Partial Replication Study

ABSTRACT

This study compares 52 on-campus and 26 distance education (DE) MSW students enrolled in two practice methods courses. This is a partial replication study, describing the results of the evaluation of the second cohort of a large DE program. The results from the first cohort are reported elsewhere (Kleinpeter & Potts, 2000). The DE students were located at four universities linked through interactive television (ITV). The comparison group was located at an urban university, taught in traditional classrooms. Comparisons were made on student grades, faculty evaluations, and field instructors' evaluations. Results indicate that no significant differences were found between on-campus and DE students which supports the findings of the first cohort, despite a reduction in faculty visits to off-campus locations, a greater use of ITV instruction, and an increased reliance on the use of local site coordinators as assistant instructors in the classroom.     

Cross-sectoral collaboration working with perinatal women who use substances: outcomes and lessons from HerWay Home

ABSTRACT

HerWay Home is a multi-service drop-in and outreach program women for with substance use issues and who also may be affected by mental illness, trauma, and/or violence. Philosophical foundations of the program include harm reduction and being relationship-based and trauma-informed. This article presents the program’s evaluation findings related to cross-sectoral service collaborations and outcomes for service partners as well as for women and families, including prevention of children going into care. For service partners, HerWay Home provided peace of mind, knowing that vulnerable women’s needs would be met and also led to increased understanding about substance-using women and best practice approaches.     

Use of Collateral Contacts in Child Custody Evaluations

ABSTRACT

Four hundred nineteen members of the Association of Family and Conciliation Courts (AFCC) identify themselves as custody evaluators in the 1999–2000 membership directory. One hundred of these custody evaluators were randomly selected for inclusion in this survey of the use of collateral contact interviews in child custody evaluations. Fifty-three of those selected responded to a brief questionnaire concerning their use and views of collaterals. All evaluators reported use of the collateral interview. Most practitioners reported using both personal and telephone interviews. Respondents indicated that they interview family and friends, but routinely place greater trust in more objective sources that have no obvious, vested interest in outcome. More experienced evaluators reported evolving toward use of written questionnaires for use in interviewing collateral sources for purposes of risk management as well as increasing relevance and convergent validity.     

Information Use by Levels in Human Service Organizations

Summary

Theories on organizational structure and the value of information informed observations of patterns of information use by workers in a human service organization. Differences were found in the type and frequency of information used by workers at three different levels-executive, manager and direct service. Findings suggest a gap in the education and training of human service practitioners; suggest effectiveness and efficiency improvements through an organization based design of information systems, To facilitate this, schools of social work must teach the decision-making purposes and processes at each level of the organization. Schools also must expose students to the technologies available to support practice decisions.     

The State of Our Art

Abstract

In this two-part paper, the author examines inter-generational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, conceptual foundations for intergenerational programming are considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, appearing in the next issue of this journal, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

The State of Our Art

Abstract

In the second part of this two-part paper, the author examines intergenerational program research and evaluation based on a framework derived from a collaborative UNESCO (2000)-sponsored review of the intergenerational program field. In Part One, which appeared in the previous issue, conceptual foundations for intergenerational programming were considered, taking into account theories that focus on individuals and groups within interactive contexts, those that focus primarily on individual development, and conceptually based program evaluations. In Part Two, effects of intergenerational program participation are described, with emphasis on program activities and various program contexts. Challenges and questions emerging from the literature are presented, identifying the need for a greater use of theory in research, more cross-cultural research, expanded outcomes, and solutions to some of the methodological challenges in intergenerational program research and evaluation.     

Self-Employment as a Social and Economic Development Intervention for Recipients of AFDC

Summary

This paper reports findings from program evaluations of two demonstrations designed to (1) test self-employment as a social and economic development strategy for recipients of AFDC, and (2) identify policy barriers to improving the economic well-being of this population. The authors collected data through interviews with program participants and program operators and used secondary data contained in operators' management information systems and state Department of Human Services electronic files. Program results showed that self-employment is a viable social and economic development strategy for self-selected welfare recipients who receive business training and assistance in locating operating capital. The paper discusses implications for social work practice, education, and research.     

Computer Games for the Frail Elderly

Summary

The benefits of computer use by the elderly are named in this article. Described here is a project instigated to determine feasibility of computer use with frail institutionalized elderly. The four games used in this project are presented, as are participant reactions to these games. Computer use was found to be not only valuable as a learning tool, but a diagnostic one as well The author looks to practitioners in the future to investigate the possibilities of using computer games in their work with the elderly.