A Deliberative Method for Ranking Risks (II): Evaluation of Validity and Agreement among Risk Managers

A deliberative method for ranking risks was evaluated in a study involving 218 risk managers. Both holistic and multiattribute procedures were used to assess individual and group rankings of health and safety risks facing students at a fictitious middle school. Consistency between the rankings that emerged from these two procedures was reasonably high for individuals and for groups, suggesting that these procedures capture an underlying construct of riskiness. Participants reported high levels of satisfaction with their groups' decision-making processes and the resulting rankings, and these reports were corroborated by regression analyses. Risk rankings were similar across individuals and groups, even though individuals and groups did not always agree on the relative importance of risk attributes. Lower consistency between the risk rankings from the holistic and multiattribute procedures and lower agreement among individuals and groups regarding these rankings were observed for a set of high-variance risks. Nonetheless, the generally high levels of consistency, satisfaction, and agreement suggest that this deliberative method is capable of producing risk rankings that can serve as informative inputs to public risk-management decision making.     

A Deliberative Method for Ranking Risks (I): Overview and Test Bed Development

Risk ranking offers a potentially powerful means for gathering public input to help set risk-management priorities. In most rankings conducted to date, the categories and attributes used to describe the risks have varied widely, the materials and procedures have not been designed to facilitate comparisons among risks on all important attributes, and the validity and reproducibility of the resulting rankings have not been assessed. To address these needs, a risk-ranking method was developed in which risk experts define and categorize the risks to be ranked, identify the relevant risk attributes, and characterize the risks in a set of standardized risk summary sheets, which are then used by lay or other groups in structured ranking exercises. To evaluate this method, a test bed involving 22 health and safety risks in a fictitious middle school was created. This article provides an overview of the risk-ranking method and describes the challenges faced in designing the middle school test bed. A companion article in this issue reports on the validity of the ranking procedures and the level of agreement among risk managers regarding ranking of risks and attributes.     

Aggregate, Disaggregate, and Hybrid Analyses of Ecological Risk Perceptions

Laypeople's perceptions of health and safety risks have been widely studied, but only a few studies have addressed perceptions of ecological hazards. We assembled a list of 39 attributes of ecological hazards from the literatures on comparative risk assessment, ecological health, environmental conservation and management, environmental psychology, and risk perception. In Study 1, 125 laypeople evaluated 83 hazards on subsets of this attribute set. Factor analysis of attribute ratings (averaged over participants) revealed six oblique factors: ecological impacts, human impacts, human benefits, aesthetic impacts, scientific understanding, and controllability. These factors predicted mean judgments of overall riskiness, ecological riskiness, acceptability, and regulatory strictness. In Study 2, 30 laypeople each evaluated 34 hazards on 17 attributes and 3 dependent variables. Aggregate-level factor analysis of these data replicated the appropriate portion of the factor solution and yielded similar regression results. Parallel analyses at the individual-participant level yielded factors that explained less variance in judgments of overall riskiness, ecological riskiness, and acceptability. However, the decrease in explanatory power was much less than is often reported for disaggregate-level analyses of psychometric data. This discrepancy illustrates the importance of distinguishing between the level of analysis (aggregate versus disaggregate) and the focus of analysis (distinctions among hazards versus distinctions among participants). In a hybrid analysis, aggregate-level factor scores predicted individual participants' riskiness judgments reasonably well. Psychometric studies such as these provide a sound empirical basis for selecting attributes of ecological hazards for use in comparative risk assessment.     

Categorizing Risks for Risk Ranking

Any practical process of risk ranking must group hazards into a manageable number of categories. Defining such categories requires value choices that can have important implications for the rankings that result. Most risk-management organizations will find it useful to begin defining categories in terms of environmental loadings or initiating events. However, the resulting categories typically need to be modified in light of other considerations. Risk-ranking projects can benefit from considering several alternative categorization strategies and drawing upon elements of each in developing their final categorization of risks. In principle, conducting multiple ranking exercises by using different categorizations could be interesting and useful. In practice, agencies are unlikely to have either the resources or patience to do this, but other groups in society might. Done well, such additional independent rankings could add valuable inputs to democratic risk-management decision making.     

The Roles of Group Membership, Beliefs, and Norms in Ecological Risk Perception

Variability in ecological risk perceptions was investigated by surveying members of four stakeholder groups commonly involved in environmental policy debates. Fifty-six individuals from government, industry, environmental, and general-public groups completed a risk-perception survey in which they evaluated 34 environmental hazards on 17 attributes and also evaluated the riskiness and acceptability of each hazard. In addition, participants reported their environmental beliefs and norms using Dunlap et al.'s revised New Ecological Paradigm Scale and modified versions of Schwartz's Awareness of Consequences and Personal Norms Scales. Group membership was predictive of participants' scores on the belief and norm scales. Factor analysis of attribute ratings (averaged across participants) revealed the anticipated three oblique factors: ecological impacts, scientific understanding, and aesthetic impacts. Factor patterns were very similar for the four stakeholder groups. Factors from the aggregate analysis were predictive of individuals' riskiness judgments, but these relationships were moderated by participants' group membership, beliefs, and norms. Compared to members of other groups, members of the general public placed less emphasis on ecological impacts and more emphasis on the other two factors when judging the ecological riskiness of hazards. To our knowledge, these results represent the first formal tests of interactions between hazard characteristics and participant characteristics in determining riskiness judgments, and illustrate how traditional psychometric analyses can be successfully coupled with individual-difference measures to improve the understanding of risk perception.     

Augmenting the Deliberative Method for Ranking Risks

The Department of Homeland Security (DHS) characterized and prioritized the physical cross‐border threats and hazards to the nation stemming from terrorism, market‐driven illicit flows of people and goods (illegal immigration, narcotics, funds, counterfeits, and weaponry), and other nonmarket concerns (movement of diseases, pests, and invasive species). These threats and hazards pose a wide diversity of consequences with very different combinations of magnitudes and likelihoods, making it very challenging to prioritize them. This article presents the approach that was used at DHS to arrive at a consensus regarding the threats and hazards that stand out from the rest based on the overall risk they pose. Due to time constraints for the decision analysis, it was not feasible to apply multiattribute methodologies like multiattribute utility theory or the analytic hierarchy process. Using a holistic approach was considered, such as the deliberative method for ranking risks first published in this journal. However, an ordinal ranking alone does not indicate relative or absolute magnitude differences among the risks. Therefore, the use of the deliberative method for ranking risks is not sufficient for deciding whether there is a material difference between the top‐ranked and bottom‐ranked risks, let alone deciding what the stand‐out risks are. To address this limitation of ordinal rankings, the deliberative method for ranking risks was augmented by adding an additional step to transform the ordinal ranking into a ratio scale ranking. This additional step enabled the selection of stand‐out risks to help prioritize further analysis.     

Engaging with Comparative Risk Appraisals: Public Views on Policy Priorities for Environmental Risk Governance

Communicating the rationale for allocating resources to manage policy priorities and their risks is challenging. Here, we demonstrate that environmental risks have diverse attributes and locales in their effects that may drive disproportionate responses among citizens. When 2,065 survey participants deployed summary information and their own understanding to assess 12 policy‐level environmental risks singularly, their assessment differed from a prior expert assessment. However, participants provided rankings similar to those of experts when these same 12 risks were considered as a group, allowing comparison between the different risks. Following this, when individuals were shown the prior expert assessment of this portfolio, they expressed a moderate level of confidence with the combined expert analysis. These are important findings for the comprehension of policy risks that may be subject to augmentation by climate change, their representation alongside other threats within national risk assessments, and interpretations of agency for public risk management by citizens and others.     

A New Multicriteria Risk Mapping Approach Based on a Multiattribute Frontier Concept

Invasive species risk maps provide broad guidance on where to allocate resources for pest monitoring and regulation, but they often present individual risk components (such as climatic suitability, host abundance, or introduction potential) as independent entities. These independent risk components are integrated using various multicriteria analysis techniques that typically require prior knowledge of the risk components’ importance. Such information is often nonexistent for many invasive pests. This study proposes a new approach for building integrated risk maps using the principle of a multiattribute efficient frontier and analyzing the partial order of elements of a risk map as distributed in multidimensional criteria space. The integrated risks are estimated as subsequent multiattribute frontiers in dimensions of individual risk criteria. We demonstrate the approach with the example of Agrilus biguttatus Fabricius, a high‐risk pest that may threaten North American oak forests in the near future. Drawing on U.S. and Canadian data, we compare the performance of the multiattribute ranking against a multicriteria linear weighted averaging technique in the presence of uncertainties, using the concept of robustness from info‐gap decision theory. The results show major geographic hotspots where the consideration of tradeoffs between multiple risk components changes integrated risk rankings. Both methods delineate similar geographical regions of high and low risks. Overall, aggregation based on a delineation of multiattribute efficient frontiers can be a useful tool to prioritize risks for anticipated invasive pests, which usually have an extremely poor prior knowledge base.     

Evaluating Consumer Preferences for Existing Multiattribute Products: A Non-Metric Approach

A model evaluating consumer preferences for multiattribute products is derived. The model possesses the following features: (1) the method works for existing products; (2) the input data require only overall product rankings and attribute rankings; (3) the distribution of part worths (utility) for particular attributes is derived and used to judge the trade-offs among different attributes; (4) the procedure can be implemented with existing software; and (5) the attributes do not need to be quantifiable.     

Prioritizing Environmental Health Risks in the UAE

This article presents the results of a comparative environmental risk‐ranking exercise that was conducted in the United Arab Emirates (UAE) to inform a strategic planning process led by the Environment Agency‐Abu Dhabi (EAD). It represents the first national‐level application of a deliberative method for comparative risk ranking first published in this journal. The deliberative method involves a five‐stage process that includes quantitative risk assessment by experts and deliberations by groups of stakeholders. The project reported in this article considered 14 categories of environmental risks to health identified through discussions with EAD staff: ambient and indoor air pollution; drinking water contamination; coastal water pollution; soil and groundwater contamination; contamination of fruits, vegetables, and seafood; ambient noise; stratospheric ozone depletion; electromagnetic fields from power lines; health impacts from climate change; and exposure to hazardous substances in industrial, construction, and agricultural work environments. Results from workshops involving 73 stakeholders who met in five separate groups to rank these risks individually and collaboratively indicated strong consensus that outdoor and indoor air pollution are the highest priorities in the UAE. Each of the five groups rated these as being among the highest risks. All groups rated soil and groundwater contamination as being among the lowest risks. In surveys administered after the ranking exercises, participants indicated that the results of the process represented their concerns and approved of using the ranking results to inform policy decisions. The results ultimately shaped a strategic plan that is now being implemented.     

Taxonomic Analysis of Perceived Risk: Modeling Individual and Group Perceptions Within Homogeneous Hazard Domains

Previous studies of risk perception have typically focused on the mean judgments of a group of people regarding the riskiness (or safety) of a diverse set of hazardous activities, substances, and technologies. This paper reports the results of two studies that take a different path. Study 1 investigated whether models within a single technological domain were similar to previous models based on group means and diverse hazards. Study 2 created a group taxonomy of perceived risk for only one technological domain, railroads, and examined whether the structure of that taxonomy corresponded with taxonomies derived from prior studies of diverse hazards. Results from Study 1 indicated that the importance of various risk characteristics in determining perceived risk differed across individuals and across hazards, but not so much as to invalidate the results of earlier studies based on group means and diverse hazards. In Study 2, the detailed analysis of railroad hazards produced a structure that had both important similarities to, and dissimilarities from, the structure obtained in prior research with diverse hazard domains. The data also indicated that railroad hazards are really quite diverse, with some approaching nuclear reactors in their perceived seriousness. These results suggest that information about the diversity of perceptions within a single domain of hazards could provide valuable input to risk-management decisions.     

Evaluation of Multicriteria Decision Analysis Algorithms in Food Safety: A Case Study on Emerging Zoonoses Prioritization

Decision making in food safety is a complex process that involves several criteria of different nature like the expected reduction in the number of illnesses, the potential economic or health-related cost, or even the environmental impact of a given policy or intervention. Several multicriteria decision analysis (MCDA) algorithms are currently used, mostly individually, in food safety to rank different options in a multifactorial environment. However, the selection of the MCDA algorithm is a decision problem on its own because different methods calculate different rankings. The aim of this study was to compare the impact of different uncertainty sources on the rankings of MCDA problems in the context of food safety. For that purpose, a previously published data set on emerging zoonoses in the Netherlands was used to compare different MCDA algorithms: MMOORA, TOPSIS, VIKOR, WASPAS, and ELECTRE III. The rankings were calculated with and without considering uncertainty (using fuzzy sets), to assess the importance of this factor. The rankings obtained differed between algorithms, emphasizing that the selection of the MCDA method had a relevant impact in the rankings. Furthermore, considering uncertainty in the ranking had a high influence on the results. Both factors were more relevant than the weights associated with each criterion in this case study. A hierarchical clustering method was suggested to aggregate results obtained by the different algorithms. This complementary step seems to be a promising way to decrease extreme difference among algorithms and could provide a strong added value in the decision-making process.     

A Score Comparison Method as an Aid to Integrating Separate Comparative Risk Rankings into a Single, Comparative Risk Ranking

A Score Comparison Method (SCM), for use in comparative risk projects, is described. It provides a degree of analytical guidance for those undertaking to integrate environmental issues which have been placed into separate, qualitative rankings according to different types of risk into a single, qualitative, integrated risk ranking. Its use in an actual case is shown.     

Does Communicating (Flood) Risk Affect (Flood) Risk Perceptions? Results of a Quasi‐Experimental Study

People's risk perceptions are generally regarded as an important determinant of their decisions to adjust to natural hazards. However, few studies have evaluated how risk communication programs affect these risk perceptions. This study evaluates the effects of a small-scale flood risk communication program in the Netherlands, consisting of workshops and focus group discussions. The effects on the workshop participants' ( n  = 24) and focus group participants' ( n  = 16) flood risk perceptions were evaluated in a pretest-posttest control group ( n  = 40) design that focused on two mechanisms of attitude change—direct personal experience and attitude polarization. We expected that (H1) workshop participants would show greater shifts in their flood risk perceptions compared with control group participants and that (H2) focus groups would rather produce the conditions for attitude polarization (shifts toward more extreme attitudinal positions after group discussion). However, the results provide only modest support for these hypotheses, perhaps because of a mismatch between the sessions' contents and the risk perception measures. An important contribution of this study is that it examined risk perception data by both conventional tests of the mean differences and tests for attitude polarization. Moreover, the possibility that attitude polarization could cause people to confirm their preexisting (hazard) beliefs could have important implications for risk communication.     

Multiattribute Risk Analysis in Nuclear Emergency Management

Radiation protection authorities have seen a potential for applying multiattribute risk analysis in nuclear emergency management and planning to deal with conflicting objectives, different parties involved, and uncertainties. This type of approach is expected to help in the following areas: to ensure that all relevant attributes are considered in decision making; to enhance communication between the concerned parties, including the public; and to provide a method for explicitly including risk analysis in the process. A multiattribute utility theory analysis was used to select a strategy for protecting the population after a simulated nuclear accident. The value-focused approach and the use of a neutral facilitator were identified as being useful.     

What's Wrong with Hazard‐Ranking Systems? An Expository Note

Two commonly recommended principles for allocating risk management resources to remediate uncertain hazards are: (1) select a subset to maximize risk-reduction benefits (e.g., maximize the von Neumann-Morgenstern expected utility of the selected risk-reducing activities), and (2) assign priorities to risk-reducing opportunities and then select activities from the top of the priority list down until no more can be afforded. When different activities create uncertain but correlated risk reductions, as is often the case in practice, then these principles are inconsistent: priority scoring and ranking fails to maximize risk-reduction benefits. Real-world risk priority scoring systems used in homeland security and terrorism risk assessment, environmental risk management, information system vulnerability rating, business risk matrices, and many other important applications do not exploit correlations among risk-reducing opportunities or optimally diversify risk-reducing investments. As a result, they generally make suboptimal risk management recommendations. Applying portfolio optimization methods instead of risk prioritization ranking, rating, or scoring methods can achieve greater risk-reduction value for resources spent.     

Perceived Risks of Conventional and Organic Produce: Pesticides, Pathogens, and Natural Toxins

Public risk perceptions and demand for safer food are important factors shaping agricultural production practices in the United States. Despite documented food safety concerns, little attempt has been made to elicit consumers' subjective risk judgments for a range of food safety hazards or to identify factors most predictive of perceived food safety risks. In this study, over 700 conventional and organic fresh produce buyers in the Boston area were surveyed for their perceived food safety risks. Survey results showed that consumers perceived relatively high risks associated with the consumption and production of conventionally grown produce compared with other public health hazards. For example, conventional and organic food buyers estimated the median annual fatality rate due to pesticide residues on conventionally grown food to be about 50 per million and 200 per million, respectively, which is similar in magnitude to the annual mortality risk from motor vehicle accidents in the United States. Over 90% of survey respondents also perceived a reduction in pesticide residue risk associated with substituting organically grown produce for conventionally grown produce, and nearly 50% perceived a risk reduction due to natural toxins and microbial pathogens. Multiple regression analyses indicate that only a few factors are consistently predictive of higher risk perceptions, including feelings of distrust toward regulatory agencies and the safety of the food supply. A variety of factors were found to be significant predictors of specific categories of food hazards, suggesting that consumers may view food safety risks as dissimilar from one another. Based on study findings, it is recommended that future agricultural policies and risk communication efforts utilize a comparative risk approach that targets a range of food safety hazards.     

Issues in Ecological Risk Assessment: The CRAM Perspective

In 1989, a Committee on Risk Assessment Methodology (CRAM) was convened by the National Research Council (NRC) to identify and investigate important scientific issues in risk assessment. One of the first issues considered by the committee was the development of a conceptual framework for ecological risk assessment, defined as "the characterization of the adverse ecological effects of environmental exposures to hazards imposed by human activities." Adverse ecological effects include all biological and nonbiological environmental changes that society perceives as undesirable. The committee's opinion was that a general framework is needed to define the relationship of ecological risk assessment to environmental management and to facilitate the development of uniform technical guidelines. The framework for human health risk assessment proposed by the NRC in 1983 was adopted as a starting point for discussion. CRAM concluded that, although ecological risk assessment and human health risk assessment differ substantially in terms of scientific disciplines and technical problems, the underlying decision process is the same for both. Therefore, CRAM recommended that the 1983 risk assessment framework be modified to accommodate both human health and ecological risk assessment. CRAM defined an integrated health/ ecological risk assessment framework consisting of the four components: Hazard Identification, Exposure Assessment, Exposure-Response Assessment, and Risk Characterization. CRAM further provided recommendations on the scope of issues to be addressed in ecological risk assessment, critical research needs, and mechanisms for providing more detailed guidance on the scientific content of ecological risk assessments.     

Workers’ Risk Tolerance and Occupational Injuries

This study explores the relationship between individuals’ risk tolerance and occupational injuries. We analyze data from a national representative survey of U.S. workers that includes information about injuries, risk tolerance, cognitive and noncognitive attributes, and risky behaviors. We measure risk tolerance through questions regarding individuals’ willingness to gamble on their lifetime income. We estimate zero‐inflated count models to assess the role played by such measures on workers’ recurrent injuries. We discuss some implications of our results for future research and occupational safety policies. Our results highlight the concurrent and changing role played by individual, work, and environmental factors in explaining recurrent incidents. They show that risk tolerance affects recurrent injuries, although not in the direction that proponents of the concept of proneness would expect. Our measure of risk aversion shows that individuals who are somewhat more risk tolerant have fewer recurrent injuries than those who are risk averse. But the estimated relationship is U‐shaped, not monotonic and, therefore, not easy to predict. At the same time, we find that individuals’ “revealed risk preferences”—specific risky behaviors—are related to higher injury probabilities. Demanding working conditions, measures of socioeconomic status, health, and safety problems experienced by workers during their youth remain among the most important factors explaining the phenomena of recurrent injuries. So our results contribute also to the important debate about the relationship between health and socioeconomic status.