Risk assessment: the emperor's new clothes?

This article reviews the risk assessment procedures and instruments that are being used by child protection agencies. Although supportive of the concept of risk assessment, the authors argue that all current instruments have major methodological deficiencies that limit the utility of such instruments as a means of predicting future abusive or neglectful behavior. The authors suggest how risk assessment decisions should be made at each stage of a child protection investigation.     

Concept guided risk assessment: Promoting prediction and understanding

This article presents a concept-guided approach to the assessment of risk in the life situations of children alleged to be maltreatment victims. This concept-guided risk assessment is theoretically based and is the first to use a nomological hierarchy which can yield itself to systematic testing for construct validity. Data were collected on 1199 cases referred for investigations of child maltreatment. Reliability was examined by applying Cronbach's alpha coefficient to the 25 risk features and the 18 risk category constructs within the seven risk areas. Convergent validity coefficients for seven risk area constructs and eighteen risk category constructs were computed and structural validity was examined using an analysis technique adapted from the multitrait-multimethod technique of Campbell and Fiske (1959). Predictive validity was assessed using a multivariate measure assessing rated risk, seriousness, and concern. In general, the results support construct validity. Improvements are needed in assessments of child vulnerability and caregiver stress. The overall psychometric evidence, combined with the model's hierarchical and theoretical nature, suggests a promising direction for future research on understanding and preventing child maltreatment.     

The relative validity of actuarial- and consensus-based risk assessment systems

In an effort to improve decision-making in child protective services (CPS), most states have, over the last two decades, implemented risk assessment systems to guide staff faced with making critical decisions in limited time frames. Generally, these systems are characterized as consensus-based or actuarial models. This study is the first to directly compare the relative validity of these two approaches. Three risk assessment instruments, two consensus-based and one actuarial, were completed on cohorts of cases from four different jurisdictions and outcome information was collected over an 18-month follow-up period. Rates of subsequent investigations, substantiations, and placements were computed for cases classified at low, moderate, and high risk levels in each model. Results clearly demonstrate that the actuarial approach more accurately classifies cases to different risk levels. These actuarial models, therefore, have the greatest potential to improve CPS decision making and better protect America's at risk children.     

Risk assessment in context

This article provides an overview of the context in which decisions about risk are made in child welfare including personal, task, and environmental factors that may contribute to uncertainty and less-than-optimal decision making, as well as some of the methodological challenges posed by the use of current risk assessment instruments. Actuarial, consensus-based, and clinical instruments are discussed and the more successful track record of actuarial decision-making in child welfare and related fields is highlighted. Methodological challenges to assessing risk are also presented including lack of reliability and validity of measures, definitional dilemmas, temporal issues such as changes in risk over time, absence of base rate data, predicting for individuals and sensitivity and specificity of measures. Implications for the design and implementation of risk assessment tools are considered in light of contextual influences and methodological limitations. Lastly, an overview of the contents of Part One of this special issue on risk assessment is provided.     

An examination of relationships between children's protective services social worker assessment of risk and independent LONGSCAN measures of risk constructs

Risk assessment as a Child Protective Services (CPS) decision-making model has been implemented in the field since the late 1980s. Research activities on risk assessment have included comparative analyses, prioritization, classification studies, implementation, and cultural sensitivity analyses. In the last decade, the major research focus has been on studies related to predictive validity. Little research on risk assessment has focused on other types of construct validity, including convergent validity. Using data from the Longitudinal Studies of Child Abuse and Neglect (LONGSCAN) and CPS investigation records, this study examined the correlations between CPS workers' ratings of risk on nine risk factors and independent measures of the same risk constructs collected by research interviewers. The data indicate significant correlations with caregiver risk factors such as physical/mental/emotional impairment, however there is little correlation with child risk factors associated with developmental or behavioral issues, or socio-economic factors such as stress and social support. Implications of the findings for policy and practice are discussed.     

Relationship of risk assessment to placement characteristics in a statewide child welfare population

Risk assessments allow child and youth services to identify children who are at risk for maltreatment (e.g., abuse, neglect) and help determine the restrictiveness of placements or need for services among youth entering a child welfare system. Despite the use of instruments by many agencies within the U.S. to determine the appropriate placements for youth, research has shown that placement decisions are often influenced by factors such as gender, age, and severity of social–emotional and behavior problems. This study examined ratings of risk across multiple domains using a structured assessment tool used by caseworkers in the Rhode Island child welfare system. The relationship between ratings of risk and placement restrictiveness was also examined. Risk levels varied across placement settings. Multivariate analyses revealed that lower caseworker ratings of parent risk and higher ratings of youth risk were associated with more restrictive placements for youth. Implications for the child welfare system are discussed.     

Looks can be deceiving: Using a risk assessment instrument to evaluate the out-comes of child protection services

This study explores the use of a risk assessment instrument based on the Illinois CANTS-17B in the child protection services division of a large urban public social services agency. It addresses the usefulness of the instrument as an outcome measure tool, that is, as means of measuring successful case outcomes based on reductions in maltreatment risk between case opening and case closing. Two separate studies were conducted in which the initial and closing risk levels on the 14-item scale were compared. Results were highly statistically significant in both studies, with the difference being in the expected and desired direction: a decrease in risk scores at case closing. However, a more detailed exploration of the data and of the practice issues involved in the agency setting strongly suggests that these differences are largely due to the artificial inflation of initial risk scores by caseworkers in order to ensure children's acceptance for ongoing child protection services. These factors are discussed in detail, along with the policy decisions that ensued from the study.     

Systemic differences in views on risk: A comparative case vignette study of risk assessment in England,Norway and the United States (California)

This article examines how child welfare workers from three countries assess risk to a child in the context of different risk assessment tools, child welfare systems and welfare regimes. Previous research suggests that there are distinct differences between child protection-oriented child welfare systems such as England and the U.S., and family service-oriented child welfare systems such as Norway (Gilbert et al., 2011). We use a case vignette method to analyze how 299 child welfare workers from England, Norway and California (U.S.) assess risk. The case vignette describes the case of ‘Beatrice’, a nine year-old girl of Black African descent who was born with an organic heart disease and a cleft palate. We found that respondents from California assessed the risk to be the lowest, followed by respondents from England and Norway. The risk factors that respondents highlighted as important for their assessment also varied significantly between countries, displaying different perceptions of elements in a case constituting risk. Respondents from Norway, who, comparatively, practice within the context of the least regulated assessment platform, identified the most homogenous assessments and types of reasoning, whereas both the assessment of risk levels and identifications of risk factors were more heterogeneous among workers in England and California. We argue that the different risk assessment tools only partly influence what workers identify as risk factors in a case, and that type of welfare states and child welfare systems is also an influence. This study thus supports existing scholarship on the distinctions between child welfare systems. However, we also found significant differences in perceptions of risk factors between England and the United States.     

Actuarial risk assessment in child protective services: Construction methodology and performance criteria

Purpose

To prevent the recurrence of child maltreatment, actuarial risk assessment can help child protective services (CPS) workers make more accurate and consistent decisions. However, there are few published articles describing construction methodologies and performance criteria to evaluate how well actuarial risk assessments perform in CPS. This article describes methodology to construct and revise an actuarial risk assessment, reviews criteria to evaluate the performance of actuarial tools, and applies a methodology and performance criteria in one state.

Methods

The sample included 6832 families who were followed for two years to determine whether they were re-reported and re-substantiated for maltreatment.

Results

Both the adopted and the revised tools had adequate separation and good predictive accuracy for all families and for the state's three largest ethnic/racial groups (White, Latino, and African American). The adopted tool classified relatively few families in the low-risk category; the revised tool distributed families across risk categories.

Conclusions

The revised tool classified more families as low-risk, allowing CPS to allocate more resources to higher-risk families, but at the cost of more false negatives.     

Prenatal child abuse risk assessment: a preliminary validation study

Workers need an efficient prenatal screener that can identify mothers at greatest risk of child abuse. Existing risk assessment methods are often invasive and difficult to administer. This study assessed child abuse risk in a sample of 49 expectant mothers using the Brigid Collins Risk Screener (BCRS). At three months postpartum, high-risk mothers scored significantly lower on the quality of infants' physical, social, and emotional environments than moderate or low-risk mothers. BCRS appears to offer a noninvasive, efficient approach to assessing risk of child abuse.     

Computer learning and risk assessment in child protection

The volume and complexity of information in child protection cases means that there can be an overwhelming number of factors which seem pertinent to decision-making but which obscure any pattern within it. This paper examines the applicability of a technique known as computer learning to the area of risk assessment in order to extract any underlying patterns. The paper proposes first that there are a few key interrelated, broad-level concepts used to assess and thereby classify risk. These can be used as the basis for producing a set of rules under which a social work team operates. The classification of risk made by one social work team on 20 child protection cases was analysed to find underlying patterns of their decision-making. These patterns are presented in the form of ‘decision trees’, as a way of illustrating the group's past experience in assessing risk. The results are evaluated in terms of the complexity and plausibility of the decision tree produced. © 1998 John Wiley & Sons, Ltd.     

Risk assessment in child sexual abuse cases

Despite continuing improvements in risk assessment for child protective services (CPS) and movement toward actuarial prediction of child maltreatment, current models have not adequately addressed child sexual abuse. Sexual abuse cases present unique and ambiguous indicators to the investigating professional, and risk factors differ from those related to physical abuse and neglect. Incorporation of research on risk factors specifically related to sexual offender recidivism into existing CPS risk assessment models may improve the ability to assess the risk of future sexual maltreatment to children. This article reviews the literature on risk factors for sexual offense recidivism and discusses their relevance and application to CPS assessment models. An evidence-based model for assessing risk in child sexual abuse cases is proposed.     

Children at risk: rationales for risk assessments and interventions

In this study, we examine and compare the rationales that social workers and laypersons give for their judgments, assessments and decisions in cases of alleged child abuse and neglect. We have used Toulmin's framework to explore the structure of the arguments/rationales of social workers and laypersons. We analyzed separately basic and complementary level arguments. We used Rosen's work to analyze the content of the backing provided by respondents for their claims.We presented two groups of respondents—52 social workers and 50 laypersons—with a case vignette that was a referral of a child and family made by a family physician. Respondents were asked to read the case, provide their assessment of risk to the child, recommend an intervention, and give written and detailed rationales for their assessment and recommended intervention.The findings indicate that social workers and laypersons differ in the structure and in the content of their rationales. Social workers formulated significantly more complete arguments, both basic and complementary levels of arguments, than laypersons. The content of the backing for the judgments was also significantly different: Social workers used more theory, experience and policy than the laypersons who used more general knowledge and values.In the discussion, we address the training implications of the fact that social workers pay very little attention to rebuttal of arguments, and do not refer to research evidence as a basis for the arguments they make. We recommend that more research focus on arguments and rationales given by professionals in this area.     

Screening for pediatric bipolar disorder in primary care

A child or adolescent with pediatric bipolar disorder (PBD) is at higher risk for suicide, violence, and impaired psychosocial functioning. The prevalence of diagnosed PBD has increased 40-fold in less than 2 decades, leading some to believe that PBD is inaccurately diagnosed. Complicating this issue, disagreements exist among clinicians as to the utility of current screening methods. The assessment picture is further muddied by the high rate of comorbid psychiatric conditions. A literature review was performed to describe the present understanding of PBD and identify current practices of screening for the disorder. Although screening tools are available, the literature suggests they lack validity. Awareness, a thorough interview, and expedient referral may assist clinicians in making a correct diagnosis. Accurate assessment will help ease the psychological and economic burden of PBD.     

Content validity and utility of the Assessment of Work Performance (AWP)

In the area of work rehabilitation, many decisions about future interventions for the client are based on the results of various kinds of assessments. Therefore, it is important that the assessment instruments used are adequate, useful, and reliable. The purpose of this study was to investigate the content validity and utility of the instrument Assessment of Work Performance (AWP) which is used to assess an individual's observable (working) skills during work performance, i.e. how efficient and appropriate a client performs a work activity. A questionnaire was answered by 67 respondents who used the AWP in various work rehabilitation settings in Sweden. The result indicates content validity and utility for the AWP that supports further testing of the instrument.     

Child protection decision-making

The effects of interagency collaboration on risk assessment in child protection practice were analysed in respect of the decision-making processes in child death inquiries. Universal features in the assessment of risk as identified in psychological literature were applied; in particular, the concepts of ‘groupthink’, the ‘certainty effect’ and ‘group polarization’. It is suggested that case conference decisions are inherently more risky than those taken by professionals with individual responsibility because of the way in which cases are framed in terms of losses.     

Hospital social workers' assessment processes for children at risk: positions in and contributions to inter-professional teams

This article explores hospital social workers' assessment processes for children at risk within the context of inter-professional teams working in child hospital care in Sweden. Based on qualitative interviews, the study discusses how assessment processes for children at risk can be understood from a professionalism perspective. Three institutionalized norms—juridical, therapeutic, and medical, building on different knowledge systems—are analyzed in relation to the three positions taken by hospital social workers as team members—active, reflective, or passive—leading to different kinds of actions in the assessment processes. The outlined norms and the positions taken involve consequences for the children and their families, but also pose questions for the future professionalization efforts of hospital social work.     

The shadow of the past: Financial risk taking and negative life events

Based on data from the four 2004–2010 waves of the US Health and Retirement Study (HRS), we show that financial risk taking is significantly related to life-history negative events out of an individual’s control. Using observed portfolio decisions to proxy for risk taking, we find correlation with two of such individual-specific events: having been victim of a physical attack and (especially) the loss of a child are associated with lower and less frequent investments in risky assets, with an intensity similar to that of the beginning, in 2008, of a collectively experienced event such as the recent financial crisis. We also find evidence that the correlation of risk taking with a child loss is long-lasting, as opposed to the correlation with a physical attack that disappears after few years. Our analysis is more in favor of a preference-based – rather than a belief-based – explanation of the observed change in risk taking. Overall our findings indicate that the past, especially through the loss of a child, casts a long shadow that extends over individuals’ current decisions also within unrelated domains.     

Child abuse/neglect risk assessment under field practice conditions: Tests of external and temporal validity and comparison with heart disease prediction

Objectives(1) Identify validation design and accuracy assessment standards for medical prognostic models applicable to evaluation of child abuse/neglect (CA/N) risk assessment models. (2) Assess the accuracy of the California Family Risk Assessment (CFRA) in predicting CA/N using the foregoing standards. (3) Compare the prediction accuracy of the CFRA with the prediction accuracy of coronary heart disease (CHD) prediction models.Questions addressed(1) What validation design and accuracy assessment standards are used to evaluate medical prognostic models? (2) What is the evidence for the accuracy of the CFRA using those standards? (3) How does the accuracy of the CFRA in predicting CA/N compare with the accuracy of CHD prediction models, which are a reasonable exemplar for the CA/N prediction effort?MethodAn external validation sample of 236 California reports of CA/N from San Luis Obispo and Sutter counties, and a larger temporal validation sample of 6307 California reports from Orange, Los Angeles, and Humboldt counties were investigated and assessed with the CFRA by line child welfare staff and were followed prospectively statewide for two years to discover reported, substantiated CA/N in any California county. CFRA accuracy in predicting substantiated CA/N was assessed by calibration and discrimination. Calibration was measured as the ratio of predicted to observed cases of CA/N seen during follow-up, with a ratio of 1.0 registering perfect calibration. Discrimination was measured by the area under the receiver operating characteristic (ROC) curve (AUC), with values from .60 to .85 found typical for medical prognostic models. CHD prediction literature was reviewed to acquire values of these accuracy measures for CHD prediction models. CFRA CA/N prediction accuracy and CHD prediction accuracy were then compared.ResultsFindings from external and temporal validation samples support the accuracy of CFRA prediction of CA/N. CFRA accuracy in predicting CA/N compared well with CHD prediction accuracy: (1) in the external validation sample, 43.42 CA/N cases were predicted during follow-up and 47 were observed, with consequent 7.6% deviation from perfect calibration. (2) In the temporal validation sample 857.49 CA/N cases were predicted and 801 were observed, with 7.1% deviation from perfect calibration. (3) The best performing of 20 Framingham CHD prediction models identified by systematic literature review predicted 222 CHD cases and 206 were observed, with 7.8% deviation from perfect calibration. (3) The CFRA external and temporal validation sample AUCs were .74 and .64, respectively. (4) For 26 CHD prediction cohorts found by literature review, the AUC mean and median values were .72 and .71, respectively, with a range from .60 to .84.Conclusions/practice implications(1) External and temporal validation results support the accuracy of the CFRA. (2) CFRA CA/N prediction accuracy parity with that for CHD prediction is encouraging, suggesting that wide use of the CFRA, properly implemented, could improve risk assessment accuracy in child protection. (3) Findings underline the importance of ensuring that no risk assessment model or method, including actuarial and consensus models and clinical judgment, is used in the field unless it has passed a test of external, or at least temporal validation.     

Improving the quality of child welfare services: needs assessment for staff training

Competency-based needs assessment surveys can be useful in assessing worker training needs in child welfare. Worker and supervisor surveys, consisting largely of task-based ability statements, constitute the core of this approach. Additional information can also be obtained from supervisors and allied professionals. Data from a study of 276 frontline child welfare workers in Alaska and Oregon are used to illustrate the utility of this approach.