What's Wrong with Risk Matrices?

Risk matrices—tables mapping "frequency" and "severity" ratings to corresponding risk priority levels—are popular in applications as diverse as terrorism risk analysis, highway construction project management, office building risk analysis, climate change risk management, and enterprise risk management (ERM). National and international standards (e.g., Military Standard 882C and AS/NZS 4360:1999) have stimulated adoption of risk matrices by many organizations and risk consultants. However, little research rigorously validates their performance in actually improving risk management decisions. This article examines some mathematical properties of risk matrices and shows that they have the following limitations. (a) Poor Resolution . Typical risk matrices can correctly and unambiguously compare only a small fraction (e.g., less than 10%) of randomly selected pairs of hazards. They can assign identical ratings to quantitatively very different risks ("range compression"). (b) Errors . Risk matrices can mistakenly assign higher qualitative ratings to quantitatively smaller risks. For risks with negatively correlated frequencies and severities, they can be "worse than useless," leading to worse-than-random decisions. (c) Suboptimal Resource Allocation . Effective allocation of resources to risk-reducing countermeasures cannot be based on the categories provided by risk matrices. (d) Ambiguous Inputs and Outputs . Categorizations of severity cannot be made objectively for uncertain consequences. Inputs to risk matrices (e.g., frequency and severity categorizations) and resulting outputs (i.e., risk ratings) require subjective interpretation, and different users may obtain opposite ratings of the same quantitative risks. These limitations suggest that risk matrices should be used with caution, and only with careful explanations of embedded judgments.     

Risk Assessment Can Be a Game‐Changing Information Technology—But Too Often It Isn't

The printing press was a game‐changing information technology. Risk assessment could be also. At present, risk assessments are commonly used as one‐time decision aids: they provide justification for a particular decision, and afterwards usually sit on a shelf. However, when viewed as information technologies, their potential uses are much broader. Risk assessments: (1) are repositories of structured information and a medium for communication; (2) embody evaluative structures for setting priorities; (3) can preserve information over time and permit asynchronous communication, thus encouraging learning and adaptation; and (4) explicitly address uncertain futures. Moreover, because of their “what‐if” capabilities, risk assessments can serve as a platform for constructive discussion among parties that hold different values. The evolution of risk assessment in the nuclear industry shows how such attributes have been used to lower core‐melt risks substantially through improved templates for maintenance and more effective coordination with regulators (although risk assessment has been less commonly used in improving emergency‐response capabilities). The end result of this evolution in the nuclear industry has been the development of “living” risk assessments that are updated more or less in real time to answer even routine operational questions. Similar but untapped opportunities abound for the use of living risk assessments to reduce risks in small operational decisions as well as large policy decisions in other areas of hazard management. They can also help improve understanding of and communication about risks, and future risk assessment and management. Realization of these opportunities will require significant changes in incentives and active promotion by the risk analytic community.     

Times Are Tough—Brother,Can You Paradigm?

Ten years ago, the National Academy of Science released its risk assessment/risk management (RA/RM) “paradigm” that served to crystallize much of the early thinking about these concepts. By defining RA as a four-step process, operationally independent from RM, the paradigm has presented society with a scheme, or a conceptually common framework, for addressing many risky situations (e.g., carcinogens, noncarcinogens, and chemical mixtures). The procedure has facilitated decision-making in a wide variety of situations and has identified the most important research needs. The past decade, however, has revealed that additional progress is needed. These areas include addressing the appropriate interaction (not isolation) between RA and RM, improving the methods for assessing risks from mixtures, dealing with “adversity of effect,” deciding whether “hazard” should imply an exposure to environmental conditions or to laboratory conditions, and evolving the concept to include both health and ecological risk. Interest in and expectations of risk assessment are increasing rapidly. The emerging concept of “comparative risk” (i.e., distinguishing between large risks and smaller risks that may be qualitatively different) is at a level comparable to that held by the concept of “risk” just 10 years ago. Comparative risk stands in need of a paradigm of its own, especially given the current economic limitations. “Times are tough; Brother, can you paradigm?”     

Rethinking Risk Assessment for Public Utility Safety Regulation

To aid in their safety oversight of large‐scale, potentially dangerous energy and water infrastructure and transportation systems, public utility regulatory agencies increasingly seek to use formal risk assessment models. Yet some of the approaches to risk assessment used by utilities and their regulators may be less useful for this purpose than is supposed. These approaches often do not reflect the current state of the art in risk assessment strategy and methodology. This essay explores why utilities and regulatory agencies might embrace risk assessment techniques that do not sufficiently assess organizational and managerial factors as drivers of risk, nor that adequately represent important uncertainties surrounding risk calculations. Further, it describes why, in the special legal, political, and administrative world of the typical public utility regulator, strategies to identify and mitigate formally specified risks might actually diverge from the regulatory promotion of “safety.” Some improvements are suggested that can be made in risk assessment approaches to support more fully the safety oversight objectives of public regulatory agencies, with examples from “high‐reliability organizations” (HROs) that have successfully merged the management of safety with the management of risk. Finally, given the limitations of their current risk assessments and the lessons from HROs, four specific assurances are suggested that regulatory agencies should seek for themselves and the public as objectives in their safety oversight of public utilities.     

Ecological Risk Assessment,Prediction, and Assessing Risk Predictions

Ecological risk assessment embodied in an adaptive management framework is becoming the global standard approach for formally assessing and managing the ecological risks of technology and development. Ensuring the continual improvement of ecological risk assessment approaches is partly achieved through the dissemination of not only the types of risk assessment approaches used, but also their efficacy. While there is an increasing body of literature describing the results of general comparisons between alternate risk assessment methods and models, there is a paucity of literature that post hoc assesses the performance of specific predictions based on an assessment of risk and the effectiveness of the particular model used to predict the risk. This is especially the case where risk assessments have been used to grant consent or approval for the construction of major infrastructure projects. While postconstruction environmental monitoring is increasingly commonplace, it is not common for a postconstruction assessment of the accuracy and performance of the ecological risk assessment and underpinning model to be undertaken. Without this “assessment of the assessment,” it is difficult for other practitioners to gain insight into the performance of the approach and models used and therefore, as argued here, this limits the rate of improvement of risk assessment approaches.     

Human and Organizational Factors in Reliability Assessment and Management of Offshore Structures

Today, there is a worldwide infrastructure of offshore structure systems that include fixed, floating, and mobile platforms, pipelines, and ships. Background on current and future trends in development of comprehensive programs to help improve the quality and reliability of offshore structure systems are discussed. A combination of proactive, reactive, and interactive risk assessment and management approaches have been developed and applied. Two risk assessment and management instruments are detailed in this article: a qualitative Quality Management Assessment System (QMAS), and a quantitative System Risk Analysis System (SYRAS). Application of QMAS to produce human and organizational performance shaping factors that are used as input to SYRAS is discussed.     

When Is a Risk Assessment Deficient According to an Uncertainty‐Based Risk Perspective?

The Petroleum Safety Authority Norway (PSA‐N) has recently adopted a new definition of risk: “the consequences of an activity with the associated uncertainty.” The PSA‐N has also been using “deficient risk assessment” for some time as a basis for assigning nonconformities in audit reports. This creates an opportunity to study the link between risk perspective and risk assessment quality in a regulatory context, and, in the present article, we take a hard look at the term “deficient risk assessment” both normatively and empirically. First, we perform a conceptual analysis of how a risk assessment can be deficient in light of a particular risk perspective consistent with the new PSA‐N risk definition. Then, we examine the usages of the term “deficient” in relation to risk assessments in PSA‐N audit reports and classify these into a set of categories obtained from the conceptual analysis. At an overall level, we were able to identify on what aspects of the risk assessment the PSA‐N is focusing and where deficiencies are being identified in regulatory practice. A key observation is that there is a diversity in how the agency officials approach the risk assessments in audits. Hence, we argue that improving the conceptual clarity of what the authorities characterize as “deficient” in relation to the uncertainty‐based risk perspective may contribute to the development of supervisory practices and, eventually, potentially strengthen the learning outcome of the audit reports.     

How to Design Rating Schemes of Risk Matrices: A Sequential Updating Approach

Risk matrices have been widely used as a risk evaluation tool in many fields due to their simplicity and intuitive nature. Designing a rating scheme, i.e., determining the number of ratings used in a risk matrix and assigning different ratings to different cells, is an essential part of risk matrix construction. However, most of the related literature has focused on applying a risk matrix to various fields, instead of researching how to design risk matrices. Based on the analysis of several current rules, we propose a new approach, namely, the sequential updating approach (SUA), to design the rating scheme of a risk matrix in a reliable way. In this article, we propose three principles and a rating algorithm based on these principles. The three principles, namely, adjusted weak consistency, consistent internality, and continuous screening, characterize a good rating scheme. The resulting rating scheme has been proven to be unique. A global rating algorithm is then proposed to create the design that satisfies the three principles. We then explore the performance of the SUA. An illustrative application is first given to explain the feasibility of our approach. The sensitivity analysis shows that our method captures a resolution‐reliability tradeoff for decisionmakers in choosing an appropriate rating scheme for a risk matrix. Finally, we compare the designs based on the SUA and Cox's axioms, highlighting the advantages of the SUA.     

Cyber risk assessment in small and medium-sized enterprises: A multilevel decision-making approach for small e-tailors

The role played by information and communication technologies in today's businesses cannot be underestimated. While such technological advancements provide numerous advantages and opportunities, they are known to thread organizations with new challenges such as cyberattacks. This is particularly important for small and medium-sized enterprises (SMEs) that are deemed to be the least mature and highly vulnerable to cybersecurity risks. Thus, this research is set to assess the cyber risks in online retailing SMEs (e-tailing SMEs). Therefore, this article employs a sample of 124 small e-tailers in the United Kingdom and takes advantage of a multi-criteria decision analysis (MCDA) method. Indeed, we identified a total number of 28 identified cyber-oriented risks in five exhaustive themes of “security,” “dependency,” “employee,” “strategic,” and “legal” risks. Subsequently, an integrated approach using step-wise weight assessment ratio analysis (SWARA) and best–worst method (BWM) has been employed to develop a pathway of risk assessment. As such, the current study outlines a novel approach toward cybersecurity risk management for e-tailing SMEs and discusses its effectiveness and contributions to the cyber risk management literature.     

Quantitative Uncertainty Analysis for a Weed Risk Assessment System

Weed risk assessments (WRA) are used to identify plant invaders before introduction. Unfortunately, very few incorporate uncertainty ratings or evaluate the effects of uncertainty, a fundamental risk component. We developed a probabilistic model to quantitatively evaluate the effects of uncertainty on the outcomes of a question‐based WRA tool for the United States. In our tool, the uncertainty of each response is rated as Negligible, Low, Moderate, or High. We developed the model by specifying the likelihood of a response changing for each uncertainty rating. The simulations determine if responses change, select new responses, and sum the scores to determine the risk rating. The simulated scores reveal potential variation in WRA risk ratings. In testing with 204 species assessments, the ranges of simulated risk scores increased with greater uncertainty, and analyses for most species produced simulated risk ratings that differed from the baseline WRA rating. Still, the most frequent simulated rating matched the baseline rating for every High Risk species, and for 87% of all tested species. The remaining 13% primarily involved ambiguous Low Risk results. Changing final ratings based on the uncertainty analysis results was not justified here because accuracy (match between WRA tool and known risk rating) did not improve. Detailed analyses of three species assessments indicate that assessment uncertainty may be best reduced by obtaining evidence for unanswered questions, rather than obtaining additional evidence for questions with responses. This analysis represents an advance in interpreting WRA results, and has enhanced our regulation and management of potential weed species.     

Public Perceptions of the Risks and Benefits of Technology1

This study attempted to verify and extend previous research on people's perceptions of the risks and benefits of technology and their judgments concerning the acceptability of technology safety regulations. The study addressed several limitations of prior work, in that: (1) it was the first “expressed-preference” study to collect data from large, representative samples of Americans; (2) the research design made “person,” rather than “technology,” the unit of statistical analysis; and (3) the study employed an expanded set of independent variables, including three qualitative benefit characteristics. The results confirmed several major conclusions of prior expressed-preference research, the most important being that members of the public tend to define “risks,”“benefits,” and “acceptability” in a complex, multidimensional manner; and that their definitions differ significantly from those used by professional risk-managers and other technical experts in quantitative assessments of risk and acceptability. The results also indicated that people's stances toward technology regulation tend to cut across traditional sociodemographic lines.     

Improving Risk Management: From Lame Excuses to Principled Practice

The three classic pillars of risk analysis are risk assessment (how big is the risk and how sure can we be?), risk management (what shall we do about it?), and risk communication (what shall we say about it, to whom, when, and how?). We propose two complements as important parts of these three bases: risk attribution (who or what addressable conditions actually caused an accident or loss?) and learning from experience about risk reduction (what works, and how well?). Failures in complex systems usually evoke blame, often with insufficient attention to root causes of failure, including some aspects of the situation, design decisions, or social norms and culture. Focusing on blame, however, can inhibit effective learning, instead eliciting excuses to deflect attention and perceived culpability. Productive understanding of what went wrong, and how to do better, thus requires moving past recrimination and excuses. This article identifies common blame‐shifting “lame excuses” for poor risk management. These generally contribute little to effective improvements and may leave real risks and preventable causes unaddressed. We propose principles from risk and decision sciences and organizational design to improve results. These start with organizational leadership. More specifically, they include: deliberate testing and learning—especially from near‐misses and accident precursors; careful causal analysis of accidents; risk quantification; candid expression of uncertainties about costs and benefits of risk‐reduction options; optimization of tradeoffs between gathering additional information and immediate action; promotion of safety culture; and mindful allocation of people, responsibilities, and resources to reduce risks. We propose that these principles provide sound foundations for improving successful risk management.     

Qualitative Risk Assessment of the Acquisition of Meticillin‐Resistant Staphylococcus aureus in Pet Dogs

This article presents a qualitative risk assessment of the acquisition of meticillin‐resistant Staphylococcus aureus (MRSA) in pet dogs, representing an important first step in the exploration of risk of bidirectional MRSA transfer between dogs and humans. A conceptual model of the seven potential pathways for MRSA acquisition in a dog in any given 24‐hour period was developed and the data available to populate that model were considered qualitatively. Humans were found to represent the most important source of MRSA for dogs in both community and veterinary hospital settings. The environment was found to be secondary to humans in terms of importance and other dogs less still. This study highlights some important methodological limitations of a technique that is heavily relied upon for qualitative risk assessments and applies a novel process, the use of relative risk ranking, to enable the generation of a defensible output using a matrix combination approach. Given the limitations of the prescribed methods as applied to the problem under consideration, further validation, or repudiation, of the findings contained herein is called for using a subsequent quantitative assessment.     

Improving the Regulation of Carcinogens by Expediting Cancer Potency Estimation1

The statutory language of the Safe Drinking Water and Toxic Enforcement Act of 1986 (Proposition 65; California Health and Safety Code 25249.5 et seq.) encourages rapid adoption of “no significant risk levels” (NSRLs), intakes associated with estimated cancer risks of no more than 1 in 100,000. Derivation of an NSRL for a carcinogen listed under Proposition 65 requires the development of a cancer potency value. This paper discusses the methodology for the derivation of cancer potencies using an expedited procedure, and provides potency estimates for a number of agents listed as carcinogens under Proposition 65. To derive expedited potency values, default risk assessment methods are applied to data sets selected from an extensive tabulation of animal cancer bioassays according to criteria used by regulatory agencies. A subset of these expedited values is compared to values previously developed by regulatory agencies using conventional quantitative risk assessment and found to be in good agreement. Specific regulatory activities which could be facilitated by adopting similar expedited procedures are identified.     

Variability and Uncertainty Meet Risk Management and Risk Communication

In the past decade, the use of probabilistic risk analysis techniques to quantitatively address variability and uncertainty in risks increased in popularity as recommended by the 1994 National Research Council that wrote Science and Judgment in Risk Assessment. Under the 1996 Food Quality Protection Act, for example, the U.S. EPA supported the development of tools that produce distributions of risk demonstrating the variability and/or uncertainty in the results. This paradigm shift away from the use of point estimates creates new challenges for risk managers, who now struggle with decisions about how to use distributions in decision making. The challenges for risk communication, however, have only been minimally explored. This presentation uses the case studies of variability in the risks of dying on the ground from a crashing airplane and from the deployment of motor vehicle airbags to demonstrate how better characterization of variability and uncertainty in the risk assessment lead to better risk communication. Analogies to food safety and environmental risks are also discussed. This presentation demonstrates that probabilistic risk assessment has an impact on both risk management and risk communication, and highlights remaining research issues associated with using improved sensitivity and uncertainty analyses in risk assessment.     

One Agency's Use of Risk Assessment and Risk Communication1

Few organizations have the courage to evaluate their own use of risk assessment (identifying hazards and estimating their probability and magnitude) and risk communication (interacting with internal and external stakeholder groups about risks). The USDA Animal and Plant Health Inspection Service (APHIS) wants to enhance its overall risk analysis process for managing a wide range of risks to animals, plants, and human health. We gathered survey data for a baseline of APHIS professionals’ understanding and use of risk assessment and risk communication. APHIS professionals spend a surprisingly large share of their time communicating about risks. They perceive that risk estimates influence decisions, but that risk estimates should have more influence. Respondents reported little opposition to APHIS risk management decisions, and little use of channels such as USDA Extension Service for disseminating risk messages. Substantial variance across responses is explained mostly by differences in the roles of the 11 work units (now 10) within the agency. Location also contributes to the variance. Demographic variables seem less important.     

Quantifying Potential Human Health Impacts of Animal Antibiotic Use: Enrofloxacin and Macrolides in Chickens

Use of similar or identical antibiotics in both human and veterinary medicine has come under increasing scrutiny by regulators concerned that bacteria resistant to animal antibiotics will infect people and resist treatment with similar human antibiotics, leading to excess illnesses and deaths. Scientists, regulators, and interest groups in the United States and Europe have urged bans on nontherapeutic and some therapeutic uses of animal antibiotics to protect human health. Many regulators and public health experts have also expressed dissatisfaction with the perceived limitations of quantitative risk assessment and have proposed alternative qualitative and judgmental approaches ranging from "attributable fraction" estimates to risk management recommendations based on the precautionary principle or on expert judgments about the importance of classes of compounds in human medicine. This article presents a more traditional quantitative risk assessment of the likely human health impacts of continuing versus withdrawing use of fluoroquinolones and macrolides in production of broiler chickens in the United States. An analytic framework is developed and applied to available data. It indicates that withdrawing animal antibiotics can cause far more human illness-days than it would prevent: the estimated human BENEFIT:RISK health ratio for human health impacts of continued animal antibiotic use exceeds 1,000:1 in many cases. This conclusion is driven by a hypothesized causal sequence in which withdrawing animal antibiotic use increases illnesses rates in animals, microbial loads in servings from the affected animals, and hence human health risks. This potentially important aspect of human health risk assessment for animal antibiotics has not previously been quantified.     

Peak Exposures in Epidemiologic Studies and Cancer Risks: Considerations for Regulatory Risk Assessment

We review approaches for characterizing “peak” exposures in epidemiologic studies and methods for incorporating peak exposure metrics in dose–response assessments that contribute to risk assessment. The focus was on potential etiologic relations between environmental chemical exposures and cancer risks. We searched the epidemiologic literature on environmental chemicals classified as carcinogens in which cancer risks were described in relation to “peak” exposures. These articles were evaluated to identify some of the challenges associated with defining and describing cancer risks in relation to peak exposures. We found that definitions of peak exposure varied considerably across studies. Of nine chemical agents included in our review of peak exposure, six had epidemiologic data used by the U.S. Environmental Protection Agency (US EPA) in dose–response assessments to derive inhalation unit risk values. These were benzene, formaldehyde, styrene, trichloroethylene, acrylonitrile, and ethylene oxide. All derived unit risks relied on cumulative exposure for dose–response estimation and none, to our knowledge, considered peak exposure metrics. This is not surprising, given the historical linear no‐threshold default model (generally based on cumulative exposure) used in regulatory risk assessments. With newly proposed US EPA rule language, fuller consideration of alternative exposure and dose–response metrics will be supported. “Peak” exposure has not been consistently defined and rarely has been evaluated in epidemiologic studies of cancer risks. We recommend developing uniform definitions of “peak” exposure to facilitate fuller evaluation of dose response for environmental chemicals and cancer risks, especially where mechanistic understanding indicates that the dose response is unlikely linear and that short‐term high‐intensity exposures increase risk.