Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.     

Expert Judgment in Risk Analysis and Management: Process, Context, and Pitfalls

The regulation and management of hazardous industrial activities increasingly rely on formal expert judgment processes to provide wisdom in areas of science and technology where traditional "good science" is, in practice, unable to supply unambiguous "facts." Expert judgment has always played a significant, if often unrecognized, role in analysis; however, recent trends are to make it formal, explicit, and documented so it can be identified and reviewed by others. We propose four categories of expert judgment and present three case studies which illustrate some of the pitfalls commonly encountered in its use. We conclude that there will be an expanding policy role for formal expert judgment and that the openness, transparency, and documentation that it requires have implications for enhanced public involvement in scientific and technical affairs.     

Risk Analysis for Autonomous Underwater Vehicle Operations in Extreme Environments

Autonomous underwater vehicles (AUVs) are used increasingly to explore hazardous marine environments. Risk assessment for such complex systems is based on subjective judgment and expert knowledge as much as on hard statistics. Here, we describe the use of a risk management process tailored to AUV operations, the implementation of which requires the elicitation of expert judgment. We conducted a formal judgment elicitation process where eight world experts in AUV design and operation were asked to assign a probability of AUV loss given the emergence of each fault or incident from the vehicle's life history of 63 faults and incidents. After discussing methods of aggregation and analysis, we show how the aggregated risk estimates obtained from the expert judgments were used to create a risk model. To estimate AUV survival with mission distance, we adopted a statistical survival function based on the nonparametric Kaplan‐Meier estimator. We present theoretical formulations for the estimator, its variance, and confidence limits. We also present a numerical example where the approach is applied to estimate the probability that the Autosub3 AUV would survive a set of missions under Pine Island Glacier, Antarctica in January–March 2009.     

A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler-Chicken Processing

A structured expert judgment study was organized to obtain input data for a microbial risk-assessment model describing the transmission of campylobacter during broiler-chicken processing in the Netherlands. More specially, the expert study was aimed at quantifying the uncertainty on input parameters of this model and focused on the contamination of broiler-chicken carcasses with campylobacter during processing. Following the protocol for structured expert judgment studies, expert assessments were elicited individually through subjective probability distribution functions. The classical model was used to aggregate the individual experts' distributions in order to obtain a single combined distribution per variable. Three different weighting schemes were applied, including equal weighting and performance-based weighting with and without optimalization of the combined distributions. The individual experts' weights were based on their performance on the seed variables. Results of the various weighting schemes are presented in terms of performance, robustness, and combined distributions of the seed variables and some of the query variables. All three weighting schemes had adequate performance, with the optimized combined distributions significantly outperforming both the equal weight and the nonoptimized combined distributions. Hence, this weighting scheme, having adequate robustness, was chosen for further processing of the results.     

Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject‐matter experts about the value of a decision‐relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal‐weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal‐weight, best‐expert, performance, frequentist, and copula) using simulated expert‐judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well‐calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best‐expert approaches perform better and the equal‐weight combination method performs worse than the alternative approaches.     

Explosion Probability of Unexploded Ordnance: Expert Beliefs

This article reports on a study to quantify expert beliefs about the explosion probability of unexploded ordnance (UXO). Some 1,976 sites at closed military bases in the United States are contaminated with UXO and are slated for cleanup, at an estimated cost of $15–140 billion. Because no available technology can guarantee 100% removal of UXO, information about explosion probability is needed to assess the residual risks of civilian reuse of closed military bases and to make decisions about how much to invest in cleanup. This study elicited probability distributions for the chance of UXO explosion from 25 experts in explosive ordnance disposal, all of whom have had field experience in UXO identification and deactivation. The study considered six different scenarios: three different types of UXO handled in two different ways (one involving children and the other involving construction workers). We also asked the experts to rank by sensitivity to explosion 20 different kinds of UXO found at a case study site at Fort Ord, California. We found that the experts do not agree about the probability of UXO explosion, with significant differences among experts in their mean estimates of explosion probabilities and in the amount of uncertainty that they express in their estimates. In three of the six scenarios, the divergence was so great that the average of all the expert probability distributions was statistically indistinguishable from a uniform (0, 1) distribution—suggesting that the sum of expert opinion provides no information at all about the explosion risk. The experts' opinions on the relative sensitivity to explosion of the 20 UXO items also diverged. The average correlation between rankings of any pair of experts was 0.41, which, statistically, is barely significant (p= 0.049) at the 95% confidence level. Thus, one expert's rankings provide little predictive information about another's rankings. The lack of consensus among experts suggests that empirical studies are needed to better understand the explosion risks of UXO.     

The Assessment of Probability Distributions from Expert Opinions with an Application to Seismic Fragility Curves

A method is developed for estimating a probability distribution using estimates of its percentiles provided by experts. The analyst's judgment concerning the credibility of these expert opinions is quantified in the likelihood function of Bayes'Theorem. The model considers explicitly the random variability of each expert estimate, the dependencies among the estimates of each expert, the dependencies among experts, and potential systematic biases. The relation between the results of the formal methods of this paper and methods used in practice is explored. A series of sensitivity studies provides insights into the significance of the parameters of the model. The methodology is applied to the problem of estimation of seismic fragility curves (i.e., the conditional probability of equipment failure given a seismically induced stress).     

Expert Scientific Judgment and Cancer Risk Assessment: A Pilot Study of Pharmacokinetic Data

When high-dose tumor data are extrapolated to low doses, it is typically assumed that the dose of a carcinogen delivered to target cells is proportional to the dose administered to test animals, even at exposure levels below the experimental range. Since pharmacokinetic data are becoming available that in some cases question the validity of this assumption, risk assessors must decide whether to maintain the standard assumption. A pilot study of formaldehyde is reported that was undertaken to demonstrate how expert scientific judgment can help guide a controversial risk assessment where pharmacokinetic data are considered inconclusive. Eight experts on pharmacokinetic data were selected by a formal procedure, and each was interviewed personally using a structured interview protocol. The results suggest that expert scientific opinion is polarized in this case, a situation that risk assessors can respond to with a range of risk characterizations considered biologically plausible by the experts. Convergence of expert opinion is likely in this case of several specific research strategies ar executed in a competent fashion. Elicitation of expert scientific judgment is a promising vehicle for evaluating the quality of pharmacokinetic data, expressing uncertainty in risk assessment, and fashioning a research agenda that offers possible forging of scientific consensus.     

Probabilistic Modeling of the Fate of Listeria Monocytogenes in Diced Bacon During the Manufacturing Process

To assess the impact of the manufacturing process on the fate of Listeria monocytogenes, we built a generic probabilistic model intended to simulate the successive steps in the process. Contamination evolution was modeled in the appropriate units (breasts, dice, and then packaging units through the successive steps in the process). To calibrate the model, parameter values were estimated from industrial data, from the literature, and based on expert opinion. By means of simulations, the model was explored using a baseline calibration and alternative scenarios, in order to assess the impact of changes in the process and of accidental events. The results are reported as contamination distributions and as the probability that the product will be acceptable with regards to the European regulatory safety criterion. Our results are consistent with data provided by industrial partners and highlight that tumbling is a key step for the distribution of the contamination at the end of the process. Process chain models could provide an important added value for risk assessment models that basically consider only the outputs of the process in their risk mitigation strategies. Moreover, a model calibrated to correspond to a specific plant could be used to optimize surveillance.     

Bayesian Reanalysis of the Challenger O-Ring Data

A Bayesian forecasting model is developed to quantify uncertainty about the postflight state of a field-joint primary O-ring (not damaged or damaged), given the O-ring temperature at the time of launch of the space shuttle Challenger in 1986. The crux of this problem is the enormous extrapolation that must be performed: 23 previous shuttle flights were launched at temperatures between 53 °F and 81 °F, but the next launch is planned at 31 °F. The fundamental advantage of the Bayesian model is its theoretic structure, which remains correct over the entire sample space of the predictor and that affords flexibility of implementation. A novel approach to extrapolating the input elements based on expert judgment is presented; it recognizes that extrapolation is equivalent to changing the conditioning of the model elements. The prior probability of O-ring damage can be assessed subjectively by experts following a nominal-interacting process in a group setting. The Bayesian model can output several posterior probabilities of O-ring damage, each conditional on the given temperature and on a different strength of the temperature effect hypothesis. A lower bound on, or a value of, the posterior probability can be selected for decision making consistently with expert judgment, which encapsulates engineering information, knowledge, and experience. The Bayesian forecasting model is posed as a replacement for the logistic regression and the nonparametric approach advocated in earlier analyses of the Challenger O-ring data. A comparison demonstrates the inherent deficiency of the generalized linear models for risk analyses that require (1) forecasting an event conditional on a predictor value outside the sampling interval, and (2) combining empirical evidence with expert judgment.     

Combining Experts' Risk Judgments on Technology Performance of Phytoremediation: Self-Confidence Ratings, Averaging Procedures, and Formative Consensus Building

Expert panels and averaging procedures are common means for coping with the uncertainty of effects of technology application in complex environments. We investigate the connection between confidence and the validity of expert judgment. Moreover, a formative consensus building procedure (FCB) is introduced that generates probability statements on the performance of technologies, and we compare different algorithms for the statistical aggregation of individual judgments. The case study refers to an expert panel of 10 environmental scientists assessing the performance of a soil cleanup technology that uses the capability of certain plants to accumulate heavy metals from the soil in the plant body (phytoremediation). The panel members first provided individual statements on the effectiveness of a phytoremediation. Such statements can support policymakers, answering the questions concerning the expected performance of the new technology in contaminated areas. The present study reviews (1) the steps of the FCB, (2) the constraints of technology application (contaminants, soil structure, etc.), (3) the measurement of expert knowledge, (4) the statistical averaging and the discursive agreement procedures, and (5) the boundaries of application for the FCB method. The quantitative statement oriented part of FCB generates terms such as: "The probability that the concentration of soil contamination will be reduced by at least 50% is 0.8." The data suggest that taking the median of the individual expert estimates provides the most accurate aggregated estimate. The discursive agreement procedure of FCB appears suitable for deriving politically relevant singular statements rather than for obtaining comprehensive information about uncertainties as represented by probability distributions.     

Reducing Overconfidence in the Interval Judgments of Experts

Elicitation of expert opinion is important for risk analysis when only limited data are available. Expert opinion is often elicited in the form of subjective confidence intervals; however, these are prone to substantial overconfidence. We investigated the influence of elicitation question format, in particular the number of steps in the elicitation procedure. In a 3‐point elicitation procedure, an expert is asked for a lower limit, upper limit, and best guess, the two limits creating an interval of some assigned confidence level (e.g., 80%). In our 4‐step interval elicitation procedure, experts were also asked for a realistic lower limit, upper limit, and best guess, but no confidence level was assigned; the fourth step was to rate their anticipated confidence in the interval produced. In our three studies, experts made interval predictions of rates of infectious diseases (Study 1, n = 21 and Study 2, n = 24: epidemiologists and public health experts), or marine invertebrate populations (Study 3, n = 34: ecologists and biologists). We combined the results from our studies using meta‐analysis, which found average overconfidence of 11.9%, 95% CI [3.5, 20.3] (a hit rate of 68.1% for 80% intervals)—a substantial decrease in overconfidence compared with previous studies. Studies 2 and 3 suggest that the 4‐step procedure is more likely to reduce overconfidence than the 3‐point procedure (Cohen's d = 0.61, [0.04, 1.18]).     

Representation,Propagation, and Decision Issues in Risk Analysis Under Incomplete Probabilistic Information

This article tries to clarify the potential role to be played by uncertainty theories such as imprecise probabilities, random sets, and possibility theory in the risk analysis process. Instead of opposing an objective bounding analysis, where only statistically founded probability distributions are taken into account, to the full‐fledged probabilistic approach, exploiting expert subjective judgment, we advocate the idea that both analyses are useful and should be articulated with one another. Moreover, the idea that risk analysis under incomplete information is purely objective is misconceived. The use of uncertainty theories cannot be reduced to a choice between probability distributions and intervals. Indeed, they offer representation tools that are more expressive than each of the latter approaches and can capture expert judgments while being faithful to their limited precision. Consequences of this thesis are examined for uncertainty elicitation, propagation, and at the decision‐making step.     

基于信息熵的群组聚类组合赋权法

在多属性群决策方法的研究中,为了科学地确定专家的权重,提出一种基于信息熵的群组聚类组合赋权法。依据各个专家的判断矩阵归一化得到的排序向量,利用相关系数法构造相关矩阵。通过分析阀值变化率选取最优聚类阀值,对相似程度较高的排序向量给出合理的聚类。运用信息熵为类内专家赋权,综合聚类结果和排序向量的信息熵,确定专家的总权重。算例表明该方法可以对较为相近的专家评价结果进行有效分类,并准确衡量每位专家评价信息量的大小,能够有效提高专家赋权的合理性和群组决策的科学性。     

Bacon, Boole, the EPA, and Scientific Standards

The scientific standards of the Environmental Protection Agency (EPA) include the use of retrospective meta-analysis. This analysis entails a use of the theory of probability that is only a simulation and cannot accurately measure the confidence that should be placed in the results. The uncertainty necessary for probability is, in a retrospective study, simulated rather than real. There are three logical forms for establishing a proposition. In the logic of the syllogism, a proposition is established by deduction from assumed propositions. In the logic of the physical sciences, a proposition is established by its ability to predict the outcomes of future experiments. In the logic of the courtroom, a proposition is established by its ability to explain past events. The logic of the courtroom operates under the handicap of working with nonrepeatable events. It is more subject to the preferences of the judge than the logic of the physical sciences or that of the syllogism. Because the logic of the courtroom is less reliable than either the logic of the physical sciences or that of the syllogism, it is the logic of last resort, i.e., it is used only when the other two are not applicable. Under the EPA scientific standards, the logic of the courtroom is accepted for establishing propositions about the physical world. As the logic of the courtroom is less reliable than that of the physical sciences, this practice increases the likelihood of errors.     

Methods for Assessing Uncertainty in Fundamental Assumptions and Associated Models for Cancer Risk Assessment

The distributional approach for uncertainty analysis in cancer risk assessment is reviewed and extended. The method considers a combination of bioassay study results, targeted experiments, and expert judgment regarding biological mechanisms to predict a probability distribution for uncertain cancer risks. Probabilities are assigned to alternative model components, including the determination of human carcinogenicity, mode of action, the dosimetry measure for exposure, the mathematical form of the dose‐response relationship, the experimental data set(s) used to fit the relationship, and the formula used for interspecies extrapolation. Alternative software platforms for implementing the method are considered, including Bayesian belief networks (BBNs) that facilitate assignment of prior probabilities, specification of relationships among model components, and identification of all output nodes on the probability tree. The method is demonstrated using the application of Evans, Sielken, and co‐workers for predicting cancer risk from formaldehyde inhalation exposure. Uncertainty distributions are derived for maximum likelihood estimate (MLE) and 95th percentile upper confidence limit (UCL) unit cancer risk estimates, and the effects of resolving selected model uncertainties on these distributions are demonstrated, considering both perfect and partial information for these model components. A method for synthesizing the results of multiple mechanistic studies is introduced, considering the assessed sensitivities and selectivities of the studies for their targeted effects. A highly simplified example is presented illustrating assessment of genotoxicity based on studies of DNA damage response caused by naphthalene and its metabolites. The approach can provide a formal mechanism for synthesizing multiple sources of information using a transparent and replicable weight‐of‐evidence procedure.     

“系族企业”内部资本市场有效率吗?——基于鸿仪系的案例研究

本文运用案例分析法,对鸿仪系内部资本市场的效率进行了深入地研究。研究结果表明,鸿仪系通过内部资本市场,频繁地进行着内部资本配置。在配置内部资本时,效率的原则服从于最终控制人的"战略"需要。鸿仪系内部资本配置是无效率的。本文认为,系族企业内部资本市场的运作应该引起人们的高度重视。同时应该加强对系族企业内部配置资本行为的监督,强化控股股东的责任,提高内部资本市场配置效率。本案例研究的发现可以为未来大样本的实证研究提供一定的线索。     

群决策中基于二元语义的主客观权重集成方法

针对一类属性值和属性权重信息均以语言评价信息形式给出的多属性群决策问题,提出了一种基于二元语义信息处理的主客观属性权重集成方法.首先运用二元语义集成算子计算属性的主观权重,然后给出一种基于最小偏差的属性客观权重计算方法,该方法根据决策者给出的语言决策矩阵计算决策群体对属性判断的偏差,偏差越大,表明决策者对该属性评价意见的分歧越大,则应赋给该属性越小的权重值,以提高群体意见的一致性.在此基础上,对属性的主、客观权重信息进行集成,得到属性综合权重.最后通过算例说明了该方法的有效性和实用性.     

Designing Risk Communications: Completing and Correcting Mental Models of Hazardous Processes, Part I

Many risk communications are intended to help the lay public make complex decisions about risk. To guide risk communicators with this objective, a mental models approach to the design and characterization of risk communications is proposed. Building on text comprehension and mental models research, this approach offers an integrated set of methods to help the risk communication designer choose and analyze risk communication content, structure, and organization. An applied example shows that two radon brochures designed with this approach present roughly the same expert facts as a radon brochure widely distributed by the U.S. EPA but meet higher standards on other content, structure, and organization criteria.