Evolutionary Evaluation: Implications for evaluators,researchers, practitioners,funders and the evidence-based program mandate

Evolutionary theory, developmental systems theory, and evolutionary epistemology provide deep theoretical foundations for understanding programs, their development over time, and the role of evaluation. This paper relates core concepts from these powerful bodies of theory to program evaluation. Evolutionary Evaluation is operationalized in terms of program and evaluation evolutionary phases, which are in turn aligned with multiple types of validity. The model of Evolutionary Evaluation incorporates Chen's conceptualization of bottom-up versus top-down program development. The resulting framework has important implications for many program management and evaluation issues. The paper illustrates how an Evolutionary Evaluation perspective can illuminate important controversies in evaluation using the example of the appropriate role of randomized controlled trials that encourages a rethinking of “evidence-based programs”. From an Evolutionary Evaluation perspective, prevailing interpretations of rigor and mandates for evidence-based programs pose significant challenges to program evolution. This perspective also illuminates the consequences of misalignment between program and evaluation phases; the importance of supporting both researcher-derived and practitioner-derived programs; and the need for variation and evolutionary phase diversity within portfolios of programs.     

Let’s Rework Our Approach with “Angry Young People”

This commentary by Social Work with Groups Editorial Advisory Board member Dr. Katrina Skewes McFerran is reprinted here with permission under a Creative Commons Attribution. Dr. McFerran, a music therapist, offers a reflection in the aftermath of a mass murder and suicide by a young person in Munich, Germany in July 2016. She states that “many youth services rely on government funding that demands “evidence” in order to justify spending from limited budgets.” She goes on to ask, “How do you prove that by establishing a meaningful connection with an angry young man you have stopped a mass shooting?”     

A Primer for Social Worker Researchers on How to Conduct a Multinomial Logistic Regression

ABSTRACT

The purpose of this article is to explain how to conduct a multinomial logistic regression (MLR) to increase its usage among social work researchers. A challenge for social work researchers carrying out practice-based research is to incorporate statistical analyses that are comprehensible to clinicians. Another challenge is that categorical variables, as opposed to continuous, commonly occur in clinical settings, yet their usage is seldom taught in social work education. This article will discuss MLR, a categorical data analysis used when there are three or more unordered categories in the outcome variable. Multinomial logistic regression determines differences across client groups, and can be useful in assessment, case planning, and examination of outcomes.     

Programs to Improve the Social and Emotional Wellbeing of Aboriginal and Torres Strait Islander Communities

Attending to the social and emotional wellbeing of those from Aboriginal and Torres Strait Islander cultural backgrounds is widely recognised to play a key role in effective social work practice. However, relatively little is currently known about what effective practice comprises, and significant challenges exist in any effort to demonstrate that programs do achieve significant change both for individual participants and the communities in which they live. This paper considers one program, Seasons for Healing, to illustrate the type of intervention that holds promise when working with adults and discusses the difficulties that arise in both defining and assessing program outcomes.     

Usability testing,initial implementation,and formative evaluation of an evidence-based intervention: Lessons from a demonstration project to reduce long-term foster care

The field of child welfare faces an undersupply of evidence-based interventions to address long-term foster care. The Permanency Innovations Initiative is a five-year federal demonstration project intended to generate evidence to reduce long stays in foster care for those youth who encounter the most substantial barriers to permanency. This article describes a systematic and staged approach to implementation and evaluation of a PII project that included usability testing as one of its key activities. Usability testing is an industry-derived practice which analyzes early implementation processes and evaluation procedures before they are finalized. This article describes the iterative selection, testing, and analysis of nine usability metrics that were designed to assess three important constructs of the project's initial implementation and evaluation: intervening early, obtaining consent, and engaging parents. Results showed that seven of nine metrics met a predetermined target. This study demonstrates how findings from usability testing influenced the initial implementation and formative evaluation of an evidence-supported intervention. Implications are discussed for usability testing as a quality improvement cycle that may contribute to better operationalized interventions and more reliable, valid, and replicable evidence.     

How Effect Size (Practical Significance) Misleads Clinical Practice: The Case for Switching to Practical Benefit to Assess Applied Research Findings

ABSTRACT

Relying on effect size as a measure of practical significance is turning out to be just as misleading as using p-values to determine the effectiveness of interventions for improving clinical practice in complex organizations such as schools. This article explains how effect sizes have misdirected practice in education and other disciplines. Even when effect size is incorporated into RCT research the recommendations of whether interventions are effective are misleading and generally useless to practitioners. As a result, a new criterion of practical benefit is recommended for evaluating research findings about the effectiveness of interventions in complex organizations where benchmarks of existing performance exist. Practical benefit exists when the unadjusted performance of an experimental group provides a noticeable advantage over an existing benchmark. Some basic principles for determining practical benefit are provided. Practical benefit is more intuitive and is expected to enable leaders to make more accurate assessments as to whether published research findings are likely to produce noticeable improvements in their organizations. In addition, practical benefit is used routinely as the research criterion for the alternative scientific methodology of improvement science that has an established track record of being a more efficient way to develop new interventions that improve practice dramatically than RCT research. Finally, the problems with practical significance suggest that the research community should seek different inferential methods for research designed to improve clinical performance in complex organizations, as compared to methods for testing theories and medicines.     

Concept mapping as a method to enhance evidence-based public health

In this paper we explore the suitability of concept mapping as a method for integrating knowledge from science, practice, and policy. In earlier research we described and analysed five cases of concept mapping procedures in the Netherlands, serving different purposes and fields in public health. In the current paper, seven new concept mapping studies of co-produced work are added to extend this analysis.For each of these twelve studies we analysed: (1) how the method was able to integrate knowledge from practice with scientific knowledge by facilitating dialogue and collaboration between different stakeholders in the field of public health, such as academic researchers, practitioners, policy-makers and the public; (2) how the method was able to bring theory development a step further (scientific relevance); and (3) how the method was able to act as a sound basis for practical decision-making (practical relevance).Based on the answers to these research questions, all but one study was considered useful for building more evidence-based public health, even though the extent to which they underpinned actual decision-making varied. The chance of actually being implemented in practice seems strongly related to the extent to which the responsible decision-makers are involved in the way the concept map is prepared and executed.     

Evidence-based practice in social work: perceptions and attitudes among Norwegian social workers

Although the concept of evidence-based practice (EBP) eludes precise definition, it is used as a trademark label to ensure that practice is based on science and thereby effective and of high quality. Proponents of EBP have proposed that both education and practice in social work should be transformed according to the principle of EBP. They attribute critical opposition to EBP to a lack of knowledge of what EBP is. In this article, we endeavour to clarify the concept and present results from a survey among social workers (N?=?2060) from Norway. The survey results show that although many have heard of the concept, few have precise knowledge about it. However, the more educated social workers are (master’s level), the clearer their opinions, both critical and non-critical. The results are discussed in relation to neo-liberal governance and development of social work.     

Social Work Research in the Field of Disability in Australia: A Scoping Review*

Although in Australia disability is receiving unprecedented attention with the rollout of major reforms, the body of research on disability to inform policy and practice has been found “not fit for purpose”. This scoping review of empirical research papers published by Australian social work authors between 2007 and 2015 investigated the quantity, nature, and scope of social work research on disability in Australia. We found a steady growth, an annual average of 13.8 papers, and a total of 124. Social work disability research makes a distinctive contribution; it is contextualised in service systems or policy, has a greater focus on community and civic participation and social relationships, and concentrates on adults, with either intellectual disability or traumatic brain injury. These research strengths provide foundations for building the profession’s research capacity and informing its practice and contribution to the multidisciplinary field of disability.

IMPLICATIONS

• Service system reform is increasing engagement of social workers with people with disabilities, yet disability has a low profile in the profession.

• Social work disability research has strengths in understanding service systems, social relationships, and strategies for social and civic participation, particularly with adults with cognitive disabilities.

• Increased attention to disability in qualifying and higher degree programs will better prepare students for complex practice and develop the distinctive contribution of social work research in this interdisciplinary field.

     

Science as a Strategy for Social Work

For more than one hundred years, various forms of science have been promoted to boost social work’s status as a profession. Evidence-based practice and talk about the science of social work are only the most recent manifestations of this pattern. Counter to this trend, this article argues that the latest upsurge of interest in scientific social work is no more likely than any of its predecessors to address the problems of the profession. Beginning with a linguistic analysis of the words science and scientific as positive signifiers, the article traces social work’s long history of science talk as a means of reassuring funding agencies and bolstering the profession’s status. Drawing on a critique of this discourse as the latest example of abstracted empiricism, it then contends that while scientific social work can only rarely capture what occurs in the field, its attempt to do so actually undercuts practitioners’ professional judgment and discretion. Last, after rejecting science talk as counterproductive, the article concludes with recommendations for an alternative strategy that might better position social work to reconcile its advocacy of social justice with its concerns about its professional status.