Expert opinion's role in assessment

How many articles start off by extolling the randomized clinical trial as the pinnacle of scientific evidence? The next sentence typically confronts the practical reality that randomized trials are simply not feasible in most cases. In the absence of definitive trials, technology assessment methodologies have been forced to rely on various types of observational studies--prospective cohort, case control, retrospective studies, cross-sectional and case studies--and a certain component of expert opinion/group judgement, as expressed either in the published medical literature or by the assessors themselves.     

Expert Judgment-Based Risk Assessment Using Statistical Scenario Analysis: A Case Study—Running the Bulls in Pamplona (Spain)

In this article, we present a methodology to assess the risk incurred by a participant in an activity involving danger of injury. The lack of high-quality historical data for the case considered prevented us from constructing a sufficiently detailed statistical model. It was therefore decided to generate a risk assessment model based on expert judgment. The methodology is illustrated in a real case context: the assessment of risk to participants in a San Fermin bull-run in Pamplona (Spain). The members of the panel of "experts on the bull-run" represented very different perspectives on the phenomenon: runners, surgeons and other health care personnel, journalists, civil defense workers, security staff, organizers, herdsmen, authors of books on the bull-run, etc. We consulted 55 experts. Our methodology includes the design of a survey instrument to elicit the experts' views and the statistical and mathematical procedures used to aggregate their subjective opinions.     

A Methodology for Assessing the Probability of Occurrence of Undesired Events in the Tietê–Paraná Inland Waterway Based on Expert Opinion

The market share of Tietê–Paraná inland waterway (TPIW) in the transport matrix of the São Paulo state, Brazil, is currently only 0.6%, but it is expected to increase to 6% over the next 20 years. In this scenario, to identify and explore potential undesired events a risk assessment is necessary. Part of this involves assigning the probability of occurrence of events, which usually is accomplished by a frequentist approach. However, in many cases, this approach is not possible due to unavailable or nonrepresentative data. This is the case of the TPIW that even though an expressive accident history is available, a frequentist approach is not suitable due to differences between current operational conditions and those met in the past. Therefore, a subjective assessment is an option as allows for working independently of the historical data, thus delivering more reliable results. In this context, this article proposes a methodology for assessing the probability of occurrence of undesired events based on expert opinion combined with fuzzy analysis. This methodology defines a criterion to weighting the experts and, using the fuzzy logic, evaluates the similarities among the experts’ beliefs to be used in the aggregation process before the defuzzification that quantifies the probability of occurrence of the events based on the experts’ opinion. Moreover, the proposed methodology is applied to the real case of the TPIW and the results obtained from the elicited experts are compared with a frequentist approach evidencing the impact on the results when considering different interpretations of the probability.     

A fuzzy AHP application in government-sponsored R&D project selection

Due to the funding scale and complexity of technology, the selection of government sponsored technology development projects can be viewed as a multiple-attribute decision that is normally made by a review committee with experts from academia, industry, and the government. In this paper, we present a fuzzy analytic hierarchy process method and utilize crisp judgment matrix to evaluate subjective expert judgments made by the technical committee of the Industrial Technology Development Program in Taiwan. Our results indicate that the scientific and technological merit is the most important evaluation criterion considered in overall technical committees. We demonstrate how the relative importance of the evaluation criteria changes under various risk environments via simulation.     

Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.     

An Empirical Test of the Relative Validity of Expert and Lay Judgments of Risk

This article investigates how accurately experts (underwriters) and lay persons (university students) judge the risks posed by life-threatening events Only one prior study (Slovic, Fischhoff, & Lichtenstein, 1985) has previously investigated the veracity of expert versus lay judgments of the magnitude of risk. In that study, a heterogeneous grouping of 15 experts was found to judge, using marginal estimations, a variety of risks as closer to the true annual frequencies of death than convenience samples of the lay population. In this study, we use a larger, homogenous sample of experts performing an ecologically valid task. We also ask our respondents to assess frequencies and relative frequencies directly, rather than ask for a "risk" estimate--a response mode subject to possible qualitative attributions-as was done in the Slovic et al. study. Although we find that the experts outperformed lay persons on a number of measures, the differences are small, and both groups showed similar global biases in terms of: (1) overestimating the likelihood of dying from a condition (marginal probability) and of dying from a condition given that it happens to you (conditional probability), and (2) underestimating the ratios of marginal and conditional likelihoods between pairs of potentially lethal events. In spite of these scaling problems, both groups showed quite good performance in ordering the lethal events in terms of marginal and conditional likelihoods. We discuss the nature of expertise using a framework developed by Bolger and Wright (1994), and consider whether the commonsense assumption of the superiority of expert risk assessors in making magnitude judgments of risk is, in fact, sensible.     

The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?

Good policy making should be based on available scientific knowledge. Sometimes this knowledge is well established through research, but often scientists must simply express their judgment, and this is particularly so in risk scenarios that are characterized by high levels of uncertainty. Usually in such cases, the opinions of several experts will be sought in order to pool knowledge and reduce error, raising the question of whether individual expert judgments should be given different weights. We argue—against the commonly advocated “classical method”—that no significant benefits are likely to accrue from unequal weighting in mathematical aggregation. Our argument hinges on the difficulty of constructing reliable and valid measures of substantive expertise upon which to base weights. Practical problems associated with attempts to evaluate experts are also addressed. While our discussion focuses on one specific weighting scheme that is currently gaining in popularity for expert knowledge elicitation, our general thesis applies to externally imposed unequal weighting schemes more generally.     

Risk Analysis for Autonomous Underwater Vehicle Operations in Extreme Environments

Autonomous underwater vehicles (AUVs) are used increasingly to explore hazardous marine environments. Risk assessment for such complex systems is based on subjective judgment and expert knowledge as much as on hard statistics. Here, we describe the use of a risk management process tailored to AUV operations, the implementation of which requires the elicitation of expert judgment. We conducted a formal judgment elicitation process where eight world experts in AUV design and operation were asked to assign a probability of AUV loss given the emergence of each fault or incident from the vehicle's life history of 63 faults and incidents. After discussing methods of aggregation and analysis, we show how the aggregated risk estimates obtained from the expert judgments were used to create a risk model. To estimate AUV survival with mission distance, we adopted a statistical survival function based on the nonparametric Kaplan‐Meier estimator. We present theoretical formulations for the estimator, its variance, and confidence limits. We also present a numerical example where the approach is applied to estimate the probability that the Autosub3 AUV would survive a set of missions under Pine Island Glacier, Antarctica in January–March 2009.     

Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject‐matter experts about the value of a decision‐relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal‐weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal‐weight, best‐expert, performance, frequentist, and copula) using simulated expert‐judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well‐calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best‐expert approaches perform better and the equal‐weight combination method performs worse than the alternative approaches.     

Predicting EPA''s Forthcoming CO Standards in Light of New Clinical Evidence

For the past several years, the EPA has been monitoring clinical studies on the adverse health effects of heart patients' exposure to carbon monoxide. This paper examines the impact of three such studies (including the multi-center CO study organized by the Health Effects Institute) on the forthcoming EPA standards for carbon monoxide. Eleven peer-nominated specialists with expertise in environmental science and public policy were interviewed according to a specific protocol. Overall, the experts felt that the HEI study would receive greater weight than the other two studies in the regulatory process for a variety of technical reasons. Weighting each expert opinion equally, we conclude that there is about a 0.69 probability that the CO standards will be retained at their present levels, a 0.28 probability that the standards will be tightened, and a 0.03 probability that the standards will be relaxed. When asked to imagine that the HEI data were never collected and then asked to reassess their probability judgments, the mean probability judgment that the standards will be relaxed rose to 0.11, while the mean probability judgment that the standards will be tightened fell to 0.20. A modest degree of variability in expert opinion is reported in the paper. While the HEI study may receive relatively great weight in the regulatory process, it increases only slightly the probability that EPA's CO standards will be changed.     

Regulating Under Uncertainty: Newsboy for Exposure Limits

Setting action levels or limits for health protection is complicated by uncertainty in the dose-response relation across a range of hazards and exposures. To address this issue, we consider the classic newsboy problem. The principles used to manage uncertainty for that case are applied to two stylized exposure examples, one for high dose and high dose rate radiation and the other for ammonia. Both incorporate expert judgment on uncertainty quantification in the dose-response relationship. The mathematical technique of probabilistic inversion also plays a key role. We propose a coupled approach, whereby scientists quantify the dose-response uncertainty using techniques such as structured expert judgment with performance weights and probabilistic inversion, and stakeholders quantify associated loss rates.     

Testing for Additivity at Select Mixture Groups of Interest Based on Statistical Equivalence Testing Methods

Several assumptions, defined and undefined, are used in the toxicity assessment of chemical mixtures. In scientific practice mixture components in the low-dose region, particularly subthreshold doses, are often assumed to behave additively (i.e., zero interaction) based on heuristic arguments. This assumption has important implications in the practice of risk assessment, but has not been experimentally tested. We have developed methodology to test for additivity in the sense of Berenbaum (Advances in Cancer Research, 1981), based on the statistical equivalence testing literature where the null hypothesis of interaction is rejected for the alternative hypothesis of additivity when data support the claim. The implication of this approach is that conclusions of additivity are made with a false positive rate controlled by the experimenter. The claim of additivity is based on prespecified additivity margins, which are chosen using expert biological judgment such that small deviations from additivity, which are not considered to be biologically important, are not statistically significant. This approach is in contrast to the usual hypothesis-testing framework that assumes additivity in the null hypothesis and rejects when there is significant evidence of interaction. In this scenario, failure to reject may be due to lack of statistical power making the claim of additivity problematic. The proposed method is illustrated in a mixture of five organophosphorus pesticides that were experimentally evaluated alone and at relevant mixing ratios. Motor activity was assessed in adult male rats following acute exposure. Four low-dose mixture groups were evaluated. Evidence of additivity is found in three of the four low-dose mixture groups. The proposed method tests for additivity of the whole mixture and does not take into account subset interactions (e.g., synergistic, antagonistic) that may have occurred and cancelled each other out.     

Application of Geostatistics to Risk Assessment

Geostatistics offers two fundamental contributions to environmental contaminant exposure assessment: (1) a group of methods to quantitatively describe the spatial distribution of a pollutant and (2) the ability to improve estimates of the exposure point concentration by exploiting the geospatial information present in the data. The second contribution is particularly valuable when exposure estimates must be derived from small data sets, which is often the case in environmental risk assessment. This article addresses two topics related to the use of geostatistics in human and ecological risk assessments performed at hazardous waste sites: (1) the importance of assessing model assumptions when using geostatistics and (2) the use of geostatistics to improve estimates of the exposure point concentration (EPC) in the limited data scenario. The latter topic is approached here by comparing design-based estimators that are familiar to environmental risk assessors (e.g., Land's method) with geostatistics, a model-based estimator. In this report, we summarize the basics of spatial weighting of sample data, kriging, and geostatistical simulation. We then explore the two topics identified above in a case study, using soil lead concentration data from a Superfund site (a skeet and trap range). We also describe several areas where research is needed to advance the use of geostatistics in environmental risk assessment.     

Using Expert Models in Human Reliability Analysis—A Dependence Assessment Method Based on Fuzzy Logic

In human reliability analysis (HRA), dependence analysis refers to assessing the influence of the failure of the operators to perform one task on the failure probabilities of subsequent tasks. A commonly used approach is the technique for human error rate prediction (THERP). The assessment of the dependence level in THERP is a highly subjective judgment based on general rules for the influence of five main factors. A frequently used alternative method extends the THERP model with decision trees. Such trees should increase the repeatability of the assessments but they simplify the relationships among the factors and the dependence level. Moreover, the basis for these simplifications and the resulting tree is difficult to trace. The aim of this work is a method for dependence assessment in HRA that captures the rules used by experts to assess dependence levels and incorporates this knowledge into an algorithm and software tool to be used by HRA analysts. A fuzzy expert system (FES) underlies the method. The method and the associated expert elicitation process are demonstrated with a working model. The expert rules are elicited systematically and converted into a traceable, explicit, and computable model. Anchor situations are provided as guidance for the HRA analyst's judgment of the input factors. The expert model and the FES‐based dependence assessment method make the expert rules accessible to the analyst in a usable and repeatable way, with an explicit and traceable basis.     

Issues in Ecological Risk Assessment: The CRAM Perspective

In 1989, a Committee on Risk Assessment Methodology (CRAM) was convened by the National Research Council (NRC) to identify and investigate important scientific issues in risk assessment. One of the first issues considered by the committee was the development of a conceptual framework for ecological risk assessment, defined as "the characterization of the adverse ecological effects of environmental exposures to hazards imposed by human activities." Adverse ecological effects include all biological and nonbiological environmental changes that society perceives as undesirable. The committee's opinion was that a general framework is needed to define the relationship of ecological risk assessment to environmental management and to facilitate the development of uniform technical guidelines. The framework for human health risk assessment proposed by the NRC in 1983 was adopted as a starting point for discussion. CRAM concluded that, although ecological risk assessment and human health risk assessment differ substantially in terms of scientific disciplines and technical problems, the underlying decision process is the same for both. Therefore, CRAM recommended that the 1983 risk assessment framework be modified to accommodate both human health and ecological risk assessment. CRAM defined an integrated health/ ecological risk assessment framework consisting of the four components: Hazard Identification, Exposure Assessment, Exposure-Response Assessment, and Risk Characterization. CRAM further provided recommendations on the scope of issues to be addressed in ecological risk assessment, critical research needs, and mechanisms for providing more detailed guidance on the scientific content of ecological risk assessments.     

Explosion Probability of Unexploded Ordnance: Expert Beliefs

This article reports on a study to quantify expert beliefs about the explosion probability of unexploded ordnance (UXO). Some 1,976 sites at closed military bases in the United States are contaminated with UXO and are slated for cleanup, at an estimated cost of $15–140 billion. Because no available technology can guarantee 100% removal of UXO, information about explosion probability is needed to assess the residual risks of civilian reuse of closed military bases and to make decisions about how much to invest in cleanup. This study elicited probability distributions for the chance of UXO explosion from 25 experts in explosive ordnance disposal, all of whom have had field experience in UXO identification and deactivation. The study considered six different scenarios: three different types of UXO handled in two different ways (one involving children and the other involving construction workers). We also asked the experts to rank by sensitivity to explosion 20 different kinds of UXO found at a case study site at Fort Ord, California. We found that the experts do not agree about the probability of UXO explosion, with significant differences among experts in their mean estimates of explosion probabilities and in the amount of uncertainty that they express in their estimates. In three of the six scenarios, the divergence was so great that the average of all the expert probability distributions was statistically indistinguishable from a uniform (0, 1) distribution—suggesting that the sum of expert opinion provides no information at all about the explosion risk. The experts' opinions on the relative sensitivity to explosion of the 20 UXO items also diverged. The average correlation between rankings of any pair of experts was 0.41, which, statistically, is barely significant (p= 0.049) at the 95% confidence level. Thus, one expert's rankings provide little predictive information about another's rankings. The lack of consensus among experts suggests that empirical studies are needed to better understand the explosion risks of UXO.     

Uncertainties in Pharmacokinetic Modeling for Perchloroethylene. I. Comparison of Model Structure, Parameters, and Predictions for Low-Dose Metabolism Rates for Models Derived by Different Authors

In recent years physiologically based pharmacokinetic models have come to play an increasingly important role in risk assessment for carcinogens. The hope is that they can help open the black box between external exposure and carcinogenic effects to experimental observations, and improve both high-dose to low-dose and interspecies projections of risk. However, to date, there have been only relatively preliminary efforts to assess the uncertainties in current modeling results. In this paper we compare the physiologically based pharmacokinetic models (and model predictions of risk-related overall metabolism) that have been produced by seven different sets of authors for perchloroethylene (tetrachloroethylene). The most striking conclusion from the data is that most of the differences in risk-related model predictions are attributable to the choice of the data sets used for calibrating the metabolic parameters. Second, it is clear that the bottom-line differences among the model predictions are appreciable. Overall, the ratios of low-dose human to bioassay rodent metabolism spanned a 30-fold range for the six available human/rat comparisons, and the seven predicted ratios of low-dose human to bioassay mouse metabolism spanned a 13-fold range. (The greater range for the rat/human comparison is attributable to a structural assumption by one author group of competing linear and saturable pathways, and their conclusion that the dangerous saturable pathway constitutes a minor fraction of metabolism in rats.) It is clear that there are a number of opportunities for modelers to make different choices of model structure, interpretive assumptions, and calibrating data in the process of constructing pharmacokinetic models for use in estimating "delivered" or "biologically effective" dose for carcinogenesis risk assessments. We believe that in presenting the results of such modeling studies, it is important for researchers to explore the results of alternative, reasonably likely approaches for interpreting the available data--and either show that any conclusions they make are relatively insensitive to particular interpretive choices, or to acknowledge the differences in conclusions that would result from plausible alternative views of the world.     

Engaging with Comparative Risk Appraisals: Public Views on Policy Priorities for Environmental Risk Governance

Communicating the rationale for allocating resources to manage policy priorities and their risks is challenging. Here, we demonstrate that environmental risks have diverse attributes and locales in their effects that may drive disproportionate responses among citizens. When 2,065 survey participants deployed summary information and their own understanding to assess 12 policy‐level environmental risks singularly, their assessment differed from a prior expert assessment. However, participants provided rankings similar to those of experts when these same 12 risks were considered as a group, allowing comparison between the different risks. Following this, when individuals were shown the prior expert assessment of this portfolio, they expressed a moderate level of confidence with the combined expert analysis. These are important findings for the comprehension of policy risks that may be subject to augmentation by climate change, their representation alongside other threats within national risk assessments, and interpretations of agency for public risk management by citizens and others.     

Comparison of Scoring Systems for Invasive Pests Using ROC Analysis and Monte Carlo Simulations

Different international plant protection organisations advocate different schemes for conducting pest risk assessments. Most of these schemes use structured questionnaire in which experts are asked to score several items using an ordinal scale. The scores are then combined using a range of procedures, such as simple arithmetic mean, weighted averages, multiplication of scores, and cumulative sums. The most useful schemes will correctly identify harmful pests and identify ones that are not. As the quality of a pest risk assessment can depend on the characteristics of the scoring system used by the risk assessors (i.e., on the number of points of the scale and on the method used for combining the component scores), it is important to assess and compare the performance of different scoring systems. In this article, we proposed a new method for assessing scoring systems. Its principle is to simulate virtual data using a stochastic model and, then, to estimate sensitivity and specificity values from these data for different scoring systems. The interest of our approach was illustrated in a case study where several scoring systems were compared. Data for this analysis were generated using a probabilistic model describing the pest introduction process. The generated data were then used to simulate the outcome of scoring systems and to assess the accuracy of the decisions about positive and negative introduction. The results showed that ordinal scales with at most 5 or 6 points were sufficient and that the multiplication‐based scoring systems performed better than their sum‐based counterparts. The proposed method could be used in the future to assess a great diversity of scoring systems.     

Different Roles and Viewpoints of Scientific Experts in Advising on Environmental Health Risks

Environmental health risks are often complex, largescale, and uncertain. The uncertainties inherent in these problems permit differences among experts in the appraisal of risks. This raises the question of whether different expert roles exist and, if so, how this affects the policy advice that is given. Here, we present a pilot study of the different roles and viewpoints that can be discerned among scientific experts in the Netherlands. Q methodology was used to empirically explore existing theoretical treatises on different expert roles. In total, 26 electromag­netic field (EMF) experts and 21 particulate matter (PM) experts participated. The responses were analyzed separately for EMF and PM respondents using Q factor analysis. In both the EMF and PM domain, three different expert roles were identified. This suggests that particular expert roles depend on the specific environmental health risk. The results indicate that different expert roles exist among scientists who provide policy advice on environmental health risks. This empirical study adds new data and insights to the literature on expert roles. The results of this study are relevant for the selection and composition of expert committees and the interpretation of expert advice.