An Empirical Test of the Relative Validity of Expert and Lay Judgments of Risk

This article investigates how accurately experts (underwriters) and lay persons (university students) judge the risks posed by life-threatening events Only one prior study (Slovic, Fischhoff, & Lichtenstein, 1985) has previously investigated the veracity of expert versus lay judgments of the magnitude of risk. In that study, a heterogeneous grouping of 15 experts was found to judge, using marginal estimations, a variety of risks as closer to the true annual frequencies of death than convenience samples of the lay population. In this study, we use a larger, homogenous sample of experts performing an ecologically valid task. We also ask our respondents to assess frequencies and relative frequencies directly, rather than ask for a "risk" estimate--a response mode subject to possible qualitative attributions-as was done in the Slovic et al. study. Although we find that the experts outperformed lay persons on a number of measures, the differences are small, and both groups showed similar global biases in terms of: (1) overestimating the likelihood of dying from a condition (marginal probability) and of dying from a condition given that it happens to you (conditional probability), and (2) underestimating the ratios of marginal and conditional likelihoods between pairs of potentially lethal events. In spite of these scaling problems, both groups showed quite good performance in ordering the lethal events in terms of marginal and conditional likelihoods. We discuss the nature of expertise using a framework developed by Bolger and Wright (1994), and consider whether the commonsense assumption of the superiority of expert risk assessors in making magnitude judgments of risk is, in fact, sensible.     

Determining Differences in Expert Judgment: Implications for Knowledge Acquisition and Validation*

In knowledge acquisition, it is often desirable to aggregate the judgments of multiple experts into a single system. In some cases this takes the form of averaging the judgments of those experts. In these situations it is desirable to determine if the experts have different views of the world before their individual judgments are aggregated. In validation, multiple experts often are employed to compare the performance of expert systems and other human actors. Often those judgments are then averaged to establish performance quality of the expert system. An important part of the comparison process should be determining if the experts have a similar view of the world. If the experts do not have similar views, their evaluations of performance may differ, resulting in a meaningless average performance measure. Alternatively, if all the validating experts do have similar views of the world then the validation process may result in paradigm myopia.     

What Number is “Fifty‐Fifty”?: Redistributing Excessive 50% Responses in Elicited Probabilities

Studies using open-ended response modes to elicit probabilistic beliefs have sometimes found an elevated frequency (or blip) at 50 in their response distributions. Our previous research suggests that this is caused by intrusion of the phrase "fifty-fifty," which represents epistemic uncertainty, rather than a true numeric probability of 50%. Such inappropriate responses pose a problem for decision analysts and others relying on probabilistic judgments. Using an explicit numeric probability scale (ranging from 0-100%) reduces thinking about uncertain events in verbal terms like "fifty-fifty," and, with it, exaggerated use of the 50 response. Here, we present two procedures for adjusting response distributions for data already collected with open-ended response modes and hence vulnerable to an exaggerated presence of 50%. Each procedure infers the prevalence of 50s had a numeric probability scale been used, then redistributes the excess. The two procedures are validated on some of our own existing data and then applied to judgments elicited from experts in groundwater pollution and bioremediation.     

Use of Technical Expert Panels: Applications to Probabilistic Seismic Hazard Analysis*

Probabilistic Seismic Hazard Analysis (PSHA) is a methodology that estimates the likelihood that various levels of earthquake-caused ground motions will be exceeded at a given location in a given future time period. Due to large uncertainties in all of the geosciences data and in their modeling, multiple model interpretations are often possible. This leads to disagreements among the experts, which in the past has led to disagreement on the selection of a ground motion for design at a given site. This paper reports on a project, co-sponsored by the U.S. Nuclear Regulatory Commission, the U.S. Department of Energy, and the Electric Power Research Institute, that was undertaken to review the state-of-the-art and improve on the overall stability of the PSHA process, by providing methodological guidance on how to perform a PSHA. The project reviewed past studies and examined ways to improve on the present state-of-the-art. In analyzing past PSHA studies, the most important conclusion is that differences in PSHA results are commonly due to process rather than technical differences. Thus, the project concentrated heavily on developing process recommendations, especially on the use of multiple experts, and this paper reports on those process recommendations. The problem of facilitating and integrating the judgments of a diverse group of experts is analyzed in detail. The authors believe that the concepts and process principles apply just as well to non-earthquake fields such as volcanic hazard, flood risk, nuclear-plant safety, and climate change.     

Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject‐matter experts about the value of a decision‐relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal‐weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal‐weight, best‐expert, performance, frequentist, and copula) using simulated expert‐judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well‐calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best‐expert approaches perform better and the equal‐weight combination method performs worse than the alternative approaches.     

Differences in Expert and Lay Judgments of Risk: Myth or Reality?

This article evaluates the nine empirical studies that have been conducted on expert versus lay judgments of risk. Contrary to received wisdom, this study finds that there is little empirical evidence for the propositions (1) that experts judge risk differently from members of the public or (2) that experts are more veridical in their risk assessments. Methodological weaknesses in the early research are documented, and it is shown that the results of more recent studies are confounded by social and demographic factors that have been found to correlate with judgments of risk. Using a task-analysis taxonomy, a template is provided for the documentation of future studies of expert-lay differences/similarities that will facilitate analytic comparison.     

Modeling Risk‐Related Knowledge in Tunneling Projects

Knowledge on failure events and their associated factors, gained from past construction projects, is regarded as potentially extremely useful in risk management. However, a number of circumstances are constraining its wider use. Such knowledge is usually scarce, seldom documented, and even unavailable when it is required. Further, there exists a lack of proven methods to integrate and analyze it in a cost‐effective way. This article addresses possible options to overcome these difficulties. Focusing on limited but critical potential failure events, the article demonstrates how knowledge on a number of important potential failure events in tunnel works can be integrated. The problem of unavailable or incomplete information was addressed by gathering judgments from a group of experts. The elicited expert knowledge consisted of failure scenarios and associated probabilistic information. This information was integrated using Bayesian belief‐networks‐based models that were first customized in order to deal with the expected divergence in judgments caused by epistemic uncertainty of risks. The work described in the article shows that the developed models that integrate risk‐related knowledge provide guidance as to the use of specific remedial measures.     

Predicting Emergency Evacuation and Sheltering Behavior: A Structured Analytical Approach

We offer a general approach to predicting public compliance with emergency recommendations. It begins with a formal risk assessment of an anticipated emergency, whose parameters include factors potentially affecting and affected by behavior, as identified by social science research. Standard procedures are used to elicit scientific experts' judgments regarding these behaviors and dependencies, in the context of an emergency scenario. Their judgments are used to refine the model and scenario, enabling local emergency coordinators to predict the behavior of citizens in their area. The approach is illustrated with a case study involving a radiological dispersion device (RDD) exploded in downtown Pittsburgh, PA. Both groups of experts (national and local) predicted approximately 80-90% compliance with an order to evacuate workplaces and 60-70% compliance with an order to shelter in place at home. They predicted 10% lower compliance for people asked to shelter at the office or to evacuate their homes. They predicted 10% lower compliance should the media be skeptical, rather than supportive. They also identified preparatory policies that could improve public compliance by 20-30%. We consider the implications of these results for improving emergency risk assessment models and for anticipating and improving preparedness for disasters, using Hurricane Katrina as a further case in point.     

Cognitive and Motivational Biases in Decision and Risk Analysis

Behavioral decision research has demonstrated that judgments and decisions of ordinary people and experts are subject to numerous biases. Decision and risk analysis were designed to improve judgments and decisions and to overcome many of these biases. However, when eliciting model components and parameters from decisionmakers or experts, analysts often face the very biases they are trying to help overcome. When these inputs are biased they can seriously reduce the quality of the model and resulting analysis. Some of these biases are due to faulty cognitive processes; some are due to motivations for preferred analysis outcomes. This article identifies the cognitive and motivational biases that are relevant for decision and risk analysis because they can distort analysis inputs and are difficult to correct. We also review and provide guidance about the existing debiasing techniques to overcome these biases. In addition, we describe some biases that are less relevant because they can be corrected by using logic or decomposing the elicitation task. We conclude the article with an agenda for future research.     

Shared Values, Social Trust, and the Perception of Geographic Cancer Clusters

Extensive scientific investigations often fail to identify specific carcinogens that have caused geographic clusters of cancer cases. In many such examples, public health officials and other experts have concluded that the cluster is not the result of a particular local environmental condition. Despite this conclusion by experts, concerned members of local communities often persist in believing that the cancer cluster was not random. The present study accounts for the persistence of this belief on the basis of two factors: (a) the tendency of the human mind to identify patterns (and causes), rather than randomness; and (b) a lack of social trust in public health experts. It was expected that perceived shared values evoke social trust. Individuals who conclude that public health experts share their values should be more likely to accept the experts' conclusion that a cancer cluster reflects randomness, not a particular local cause. Individuals who trust authorities should be more inclined than individuals not having trust to accept that a geographic cluster of cancer cases is a coincidence. Data from Swiss students (N = 334) supported these expectations. Additionally, significant gender differences were observed. Females had less trust in authorities and perceived the cancer cluster as less likely to be a result of pure chance than did males. Practical implications of the results are discussed.     

Public Perceptions of the Risks and Benefits of Technology1

This study attempted to verify and extend previous research on people's perceptions of the risks and benefits of technology and their judgments concerning the acceptability of technology safety regulations. The study addressed several limitations of prior work, in that: (1) it was the first “expressed-preference” study to collect data from large, representative samples of Americans; (2) the research design made “person,” rather than “technology,” the unit of statistical analysis; and (3) the study employed an expanded set of independent variables, including three qualitative benefit characteristics. The results confirmed several major conclusions of prior expressed-preference research, the most important being that members of the public tend to define “risks,”“benefits,” and “acceptability” in a complex, multidimensional manner; and that their definitions differ significantly from those used by professional risk-managers and other technical experts in quantitative assessments of risk and acceptability. The results also indicated that people's stances toward technology regulation tend to cut across traditional sociodemographic lines.     

Capturing and Integrating Knowledge for Managing Risks in Tunnel Works

Risk‐related knowledge gained from past construction projects is regarded as potentially extremely useful in risk management. This article describes a proposed approach to capture and integrate risk‐related knowledge to support decision making in construction projects. To ameliorate the problem related to the scarcity of risks information often encountered in construction projects, Bayesian Belief Networks are used and expert judgment is elicited to augment available information. Particularly, the article provides an overview of judgment‐based biases that can appear in the elicitation of judgments for constructing Bayesian Networks and the provisos that can be made in this respect to minimize these types of bias. The proposed approach is successfully applied to develop six models for top risks in tunnel works. More than 30 tunneling experts in the Netherlands and Germany were involved in the investigation to provide information on identifying relevant scenarios than can lead to failure events associated with tunneling risks. The article has provided an illustration of the applicability of the developed approach for the case of “face instability in soft soils using slurry shields.”     

Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in Their Judgments About Future Energy Technologies

Expert elicitations are now frequently used to characterize uncertain future technology outcomes. However, their usefulness is limited, in part because: estimates across studies are not easily comparable; choices in survey design and expert selection may bias results; and overconfidence is a persistent problem. We provide quantitative evidence of how these choices affect experts’ estimates. We standardize data from 16 elicitations, involving 169 experts, on the 2030 costs of five energy technologies: nuclear, biofuels, bioelectricity, solar, and carbon capture. We estimate determinants of experts’ confidence using survey design, expert characteristics, and public R&D investment levels on which the elicited values are conditional. Our central finding is that when experts respond to elicitations in person (vs. online or mail) they ascribe lower confidence (larger uncertainty) to their estimates, but more optimistic assessments of best‐case (10th percentile) outcomes. The effects of expert affiliation and country of residence vary by technology, but in general: academics and public‐sector experts express lower confidence than private‐sector experts; and E.U. experts are more confident than U.S. experts. Finally, extending previous technology‐specific work, higher R&D spending increases experts’ uncertainty rather than resolves it. We discuss ways in which these findings should be seriously considered in interpreting the results of existing elicitations and in designing new ones.     

Representations as Decision Aids: The Asymmetric Effects of Words and Numbers on Auditors' Inherent Risk Judgments*

Decision aids often change the representations used to express risk cue information and communicate risk judgments. In this study, we hypothesize that numbers and words may have asymmetric effects when used to represent risk cue information as opposed to when used to communicate risk judgments. Specifically, decision makers' agreement on cue weights and judgment consensus may be higher with cue information stated in words rather than numbers, since appropriately chosen words can more directly convey the risk implications of information cues. In contrast, decision makers' consistency in applying their judgment policies and judgment consensus may be higher with risk judgments communicated in numbers instead of words, because the greater precision available with numbers allows finer discriminations of risk. To test our hypotheses, we conducted an experiment in which experienced financial auditors made inherent risk judgments for a hypothetical manufacturing client. Cue and response representations were manipulated between numbers and words. The results generally support our arguments. In addition, representing cue information in words increased the time required to acquire cue information relative to numeric representation. In contrast, requiring auditors to express their risk judgments in numbers increased the time required to express risk assessments relative to     

Rating Threat Mitigators: Faith in Experts,Governments, and Individuals Themselves to Create a Safer World

This research explores public judgments about the threat-reducing potential of experts, individual behavior, and government spending. The data are responses of a national sample of 1225 to mail surveys that include measures of several dimensions of public judgments about violent crime, automobile accidents, hazardous chemical waste, air pollution, water pollution, global warming, AIDS, heart disease, and cancer. Beliefs about who can best mitigate threats are specific to classes of threats. In general, there is little faith that experts can do much about violent crime and automobile accidents, moderate faith in their ability to address problems of global warming, and greater expectations for expert solutions to the remaining threats. People judge individual behavior as effective in reducing the threats of violent crime, AIDS, heart disease, and automobile accidents but less so for the remaining threats. Faith in more government spending is highest for AIDS and the other two health items, lowest for the trio of violent crime, automobile accidents, and global warming, and moderate for the remaining threats. For most threats, people are not distributed at the extremes in judging mitigators. Strong attitudinal and demographic cleavages are also lacking, although some interesting relationships occur. This relative lack of sharp cleavages and the generally moderate opinion indicate ample opportunity for public education and risk communication.     

A Fairness Justification of Utilitarianism

Differences in preferences are important to explain variation in individuals' behavior. There is, however, no consensus on how to take these differences into account when evaluating policies. While prominent in the economic literature, the standard utilitarian criterion is controversial. According to some, interpersonal comparability of utilities involves value judgments with little objective basis. Others argue that social justice is primarily about the distribution of commodities assigned to individuals, rather than their subjective satisfaction or happiness. In this paper, we propose and axiomatically characterize a criterion, named opportunity‐equivalent utilitarian, that addresses these claims. First, our criterion ranks social alternatives on the basis of individuals' ordinal preferences. Second, it compares individuals based on the fairness of their assignments. Opportunity‐equivalent utilitarianism requires society to maximize the sum of specific indices of well‐being that are cardinal, interpersonally comparable, and represent each individual's preferences.     

会计估计的准确性、行业共识信息与个体投资者的决策

会计信息在投资者的投资决策过程中发挥着十分重要的作用,其质量直接影响着投资者的判断和决策.会计估计在提高会计信息相关性的同时,也为公司管理层提供了盈余管理的空间.本文实验检验了对会计估计的准确性进行事后披露这一机制是否有助于个体投资者做出正确的判断和决策,以及行业共识信息对这一机制发挥作用的影响.研究发现:(1)事后披露会计估计准确性的信息只有通过行业共识信息的辅助才能有效发挥作用,即只有在两者的共同作用下,个体投资者才能够正确识别会计估计准确的原因,也才能够进行正确的归因,并做出正确的判断和决策;(2)投资者对会计估计的准确性会产生不同的归因,投资者的归因进一步影响了其对管理层评价,而对管理层的评价影响了其对公司市盈率的评价,对公司市盈率的评价进一步影响了投资者对其投资可能性的判断.     

Trust in Risk Regulation: Cause or Consequence of the Acceptability of GM Food?

Although there is ample empirical evidence that trust in risk regulation is strongly related to the perception and acceptability of risk, it is less clear what the direction of this relationship is. This article explores the nature of the relationship, using three separate data sets on perceptions of genetically modified (GM) food among the British public. The article has two discrete but closely interrelated objectives. First, it compares two models of trust. More specifically, it investigates whether trust is the cause (causal chain account) or the consequence (associationist view) of the acceptability of GM food. Second, this study explores whether the affect heuristic can be applied to a wider number of risk-relevant concepts than just perceived risk and benefit. The results suggest that, rather than a determinant, trust is an expression or indicator of the acceptability of GM food. In addition, and as predicted, "affect" accounts for a large portion of the variance between perceived risk, perceived benefit, trust in risk regulation, and acceptability. Overall, the results support the associationist view that specific risk judgments are driven by more general evaluative judgments The implications of these results for risk communication and policy are discussed.     

The Asbestos Case: A Comment on the Appointment and Use of Nonpartisan Experts in World Trade Organization Dispute Resolution Involving Health Risk

The World Trade Organization is currently evolving its approach to incorporating scientific and technological evidence into its dispute-resolution process. In European Communities-Measures Affecting Asbestos and Asbestos-Containing Products, the Panel was faced with a large amount of complex and conflicting scientific evidence presented by the partisan experts. The Asbestos Panel's solution was to appoint independent, nonpartisan experts to help it understand and evaluate the scientific evidence. While this was far better than trying to unravel the conflicting scientific evidence on its own, two aspects of the Panel's adopted procedure merit scrutiny. First, the expert-selection process used by the Panel may not assure that the collective expertise of the appointed experts is broad enough when the dispute involves multidisciplinary scientific issues. Second, the process adopted by the Panel for consulting the appointed experts-which involved individual consultation rather than a consensus process-may leave a panel with a distorted or confused picture of the science. A consensus approach is the best means of obtaining scientific advice from appointed experts; it is most calculated to provide a clear and accurate report of the scientific information needed by a panel to make a fair and informed decision on the dispute before it. The underlying principle of world trade agreements is that it is beneficial to all of us to have free trade. Among other things, this requires an effective means of resolving disputes, and increasingly that includes disputes involving complex scientific and technological issues. This can be achieved only if the parties have confidence that their disputes will be decided in a fair and informed manner, based on the best science available. To achieve this goal, we suggest that future WTO panels depart in certain respects from the procedures utilized by the Asbestos Panel.     

Deciding what's right: The role of external sanctions and embarrassment in shaping moral judgments in the workplace

We assert that previous research has overlooked the pervasive ambiguity in ethical situations in organizations, as well as how people pierce through this ambiguity to realize new distinctions between right and wrong. Focusing on well-intentioned individuals who unknowingly transgress, we present a theory of how they come to recalibrate their moral judgments. We begin by discussing the composition and nature of a moral judgment. Building on this discussion, we then consider how external sanctions can be used to shift moral judgments. Finally, we posit that internal emotional responses to sanctions (namely embarrassment) will facilitate this shift by triggering a sense of moral deficiency. More specifically, we assert that embarrassment will focus the transgressor's attention on what went wrong. This reflection provides an opportunity for the recalibration of the initial moral judgment. We conclude with a discussion of the implications of our theory.