Narrative review of strategies by organizations for building evaluation capacity

Program evaluation is an important source of information to assist organizations to make “evidence-informed” decisions about program planning and development. The objectives of this study were to identify evaluated strategies used by organizations and program developers to build the program evaluation capacity of their workforce, and to describe success factors and lessons learned. Common elements for successful evaluation capacity building (ECB) include: a tailored strategy based on needs assessment, an organizational commitment to evaluation and ECB, experiential learning, training with a practical element, and some form of ongoing technical support within the workplace. ECB is a relatively new field of endeavor, and, while existing studies in ECB are characterized by lower levels of evidence, they suggest the most successful approaches to ECB are likely to be multifaceted. To build the level of evidence in this field, more rigorous study designs need to be implemented in the future.     

Results from a psychometric assessment of a new tool for measuring evidence-based decision making in public health organizations

BackgroundIn order to better understand how to improve evidence-based decision making (EBDM) in state health departments, measurement tools are needed to evaluate changes in EBDM. The purpose of this study was to test the psychometric properties of a new measurement tool to assess EBDM in public health practice settings.MethodsA questionnaire was developed, pilot-tested and refined in an iterative process with the input of public health practitioners with the aim of identifying a set of specific measures representing different components of EBDM. Data were collected in a national survey of state health department chronic disease practitioners. The final dataset (n = 879) for psychometric testing was comprised of 19 EBDM items that were first examined using exploratory factor analysis, and then confirmatory factor analysis.ResultsThe final model from confirmatory factor analysis includes five latent factors representing components of EBDM: capacity for evaluation, expectations and incentives for EBDM, access to evidence and resources for EBDM, participatory decision making, and leadership support and commitment.ConclusionsThis study addresses the need for empirically tested and theory-aligned measures that may be used to assess the extent to which EBDM is currently implemented, and further, to gauge the success of strategies to improve EBDM, in public health settings. This EBDM measurement tool may help identify needed supports for enhanced capacity and implementation of effective strategies.     

Programs to Improve the Social and Emotional Wellbeing of Aboriginal and Torres Strait Islander Communities

Attending to the social and emotional wellbeing of those from Aboriginal and Torres Strait Islander cultural backgrounds is widely recognised to play a key role in effective social work practice. However, relatively little is currently known about what effective practice comprises, and significant challenges exist in any effort to demonstrate that programs do achieve significant change both for individual participants and the communities in which they live. This paper considers one program, Seasons for Healing, to illustrate the type of intervention that holds promise when working with adults and discusses the difficulties that arise in both defining and assessing program outcomes.     

Official statistics, public policy and public trust

Summary.  The question of public confidence in official statistics has been central to government statistical policy for the last 10 years. This year the Statistics and Registration Service Act 2007 was passed. The paper suggests key characteristics of the new arrangements that will be needed if public confidence in the official statistics outputs is to be strengthened. It is argued that this will depend on public confidence in the statistical system as a whole rather than just the new Board. The organizational structure of the UK statistical system is described and this is linked to the issue of public confidence. Finally the wider questions of evidence-based policy and the use of statistics and statistical thinking throughout government are discussed.     

How Effect Size (Practical Significance) Misleads Clinical Practice: The Case for Switching to Practical Benefit to Assess Applied Research Findings

ABSTRACT

Relying on effect size as a measure of practical significance is turning out to be just as misleading as using p-values to determine the effectiveness of interventions for improving clinical practice in complex organizations such as schools. This article explains how effect sizes have misdirected practice in education and other disciplines. Even when effect size is incorporated into RCT research the recommendations of whether interventions are effective are misleading and generally useless to practitioners. As a result, a new criterion of practical benefit is recommended for evaluating research findings about the effectiveness of interventions in complex organizations where benchmarks of existing performance exist. Practical benefit exists when the unadjusted performance of an experimental group provides a noticeable advantage over an existing benchmark. Some basic principles for determining practical benefit are provided. Practical benefit is more intuitive and is expected to enable leaders to make more accurate assessments as to whether published research findings are likely to produce noticeable improvements in their organizations. In addition, practical benefit is used routinely as the research criterion for the alternative scientific methodology of improvement science that has an established track record of being a more efficient way to develop new interventions that improve practice dramatically than RCT research. Finally, the problems with practical significance suggest that the research community should seek different inferential methods for research designed to improve clinical performance in complex organizations, as compared to methods for testing theories and medicines.     

Concept mapping as a method to enhance evidence-based public health

In this paper we explore the suitability of concept mapping as a method for integrating knowledge from science, practice, and policy. In earlier research we described and analysed five cases of concept mapping procedures in the Netherlands, serving different purposes and fields in public health. In the current paper, seven new concept mapping studies of co-produced work are added to extend this analysis.For each of these twelve studies we analysed: (1) how the method was able to integrate knowledge from practice with scientific knowledge by facilitating dialogue and collaboration between different stakeholders in the field of public health, such as academic researchers, practitioners, policy-makers and the public; (2) how the method was able to bring theory development a step further (scientific relevance); and (3) how the method was able to act as a sound basis for practical decision-making (practical relevance).Based on the answers to these research questions, all but one study was considered useful for building more evidence-based public health, even though the extent to which they underpinned actual decision-making varied. The chance of actually being implemented in practice seems strongly related to the extent to which the responsible decision-makers are involved in the way the concept map is prepared and executed.     

Evidence-based practice in social work: perceptions and attitudes among Norwegian social workers

Although the concept of evidence-based practice (EBP) eludes precise definition, it is used as a trademark label to ensure that practice is based on science and thereby effective and of high quality. Proponents of EBP have proposed that both education and practice in social work should be transformed according to the principle of EBP. They attribute critical opposition to EBP to a lack of knowledge of what EBP is. In this article, we endeavour to clarify the concept and present results from a survey among social workers (N?=?2060) from Norway. The survey results show that although many have heard of the concept, few have precise knowledge about it. However, the more educated social workers are (master’s level), the clearer their opinions, both critical and non-critical. The results are discussed in relation to neo-liberal governance and development of social work.     

Social Work Research in the Field of Disability in Australia: A Scoping Review*

Although in Australia disability is receiving unprecedented attention with the rollout of major reforms, the body of research on disability to inform policy and practice has been found “not fit for purpose”. This scoping review of empirical research papers published by Australian social work authors between 2007 and 2015 investigated the quantity, nature, and scope of social work research on disability in Australia. We found a steady growth, an annual average of 13.8 papers, and a total of 124. Social work disability research makes a distinctive contribution; it is contextualised in service systems or policy, has a greater focus on community and civic participation and social relationships, and concentrates on adults, with either intellectual disability or traumatic brain injury. These research strengths provide foundations for building the profession’s research capacity and informing its practice and contribution to the multidisciplinary field of disability.

IMPLICATIONS

• Service system reform is increasing engagement of social workers with people with disabilities, yet disability has a low profile in the profession.

• Social work disability research has strengths in understanding service systems, social relationships, and strategies for social and civic participation, particularly with adults with cognitive disabilities.

• Increased attention to disability in qualifying and higher degree programs will better prepare students for complex practice and develop the distinctive contribution of social work research in this interdisciplinary field.

     

Science as a Strategy for Social Work

For more than one hundred years, various forms of science have been promoted to boost social work’s status as a profession. Evidence-based practice and talk about the science of social work are only the most recent manifestations of this pattern. Counter to this trend, this article argues that the latest upsurge of interest in scientific social work is no more likely than any of its predecessors to address the problems of the profession. Beginning with a linguistic analysis of the words science and scientific as positive signifiers, the article traces social work’s long history of science talk as a means of reassuring funding agencies and bolstering the profession’s status. Drawing on a critique of this discourse as the latest example of abstracted empiricism, it then contends that while scientific social work can only rarely capture what occurs in the field, its attempt to do so actually undercuts practitioners’ professional judgment and discretion. Last, after rejecting science talk as counterproductive, the article concludes with recommendations for an alternative strategy that might better position social work to reconcile its advocacy of social justice with its concerns about its professional status.     

Unmet needs in addressing child neglect: Should we go back to the drawing board?

Intervention in child neglect faces many challenges to effectiveness, including: (a) The lack of a cohesive, agreed-upon conceptual/theoretical framework and inconsistent definition of the problem; (b) disjointed intervention in various components of the problem; and (c) the social justice issues of inequity based on class, race, and gender. These conditions have led to a situation in which the role of poverty, potentially one of the most important contextual factors in neglect, can be overlooked. This article presents an argument for the need to pay more explicit attention to the definition of neglect, with particular focus on the role of poverty, in order to improve intervention in this pervasive problem. It concludes with recommendations for research, policy, and practice in child welfare.