Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject‐matter experts about the value of a decision‐relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal‐weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal‐weight, best‐expert, performance, frequentist, and copula) using simulated expert‐judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well‐calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best‐expert approaches perform better and the equal‐weight combination method performs worse than the alternative approaches.     

Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.     

Expert Scientific Judgment and Cancer Risk Assessment: A Pilot Study of Pharmacokinetic Data

When high-dose tumor data are extrapolated to low doses, it is typically assumed that the dose of a carcinogen delivered to target cells is proportional to the dose administered to test animals, even at exposure levels below the experimental range. Since pharmacokinetic data are becoming available that in some cases question the validity of this assumption, risk assessors must decide whether to maintain the standard assumption. A pilot study of formaldehyde is reported that was undertaken to demonstrate how expert scientific judgment can help guide a controversial risk assessment where pharmacokinetic data are considered inconclusive. Eight experts on pharmacokinetic data were selected by a formal procedure, and each was interviewed personally using a structured interview protocol. The results suggest that expert scientific opinion is polarized in this case, a situation that risk assessors can respond to with a range of risk characterizations considered biologically plausible by the experts. Convergence of expert opinion is likely in this case of several specific research strategies ar executed in a competent fashion. Elicitation of expert scientific judgment is a promising vehicle for evaluating the quality of pharmacokinetic data, expressing uncertainty in risk assessment, and fashioning a research agenda that offers possible forging of scientific consensus.     

Predicting EPA''s Forthcoming CO Standards in Light of New Clinical Evidence

For the past several years, the EPA has been monitoring clinical studies on the adverse health effects of heart patients' exposure to carbon monoxide. This paper examines the impact of three such studies (including the multi-center CO study organized by the Health Effects Institute) on the forthcoming EPA standards for carbon monoxide. Eleven peer-nominated specialists with expertise in environmental science and public policy were interviewed according to a specific protocol. Overall, the experts felt that the HEI study would receive greater weight than the other two studies in the regulatory process for a variety of technical reasons. Weighting each expert opinion equally, we conclude that there is about a 0.69 probability that the CO standards will be retained at their present levels, a 0.28 probability that the standards will be tightened, and a 0.03 probability that the standards will be relaxed. When asked to imagine that the HEI data were never collected and then asked to reassess their probability judgments, the mean probability judgment that the standards will be relaxed rose to 0.11, while the mean probability judgment that the standards will be tightened fell to 0.20. A modest degree of variability in expert opinion is reported in the paper. While the HEI study may receive relatively great weight in the regulatory process, it increases only slightly the probability that EPA's CO standards will be changed.     

Regulating Under Uncertainty: Newsboy for Exposure Limits

Setting action levels or limits for health protection is complicated by uncertainty in the dose-response relation across a range of hazards and exposures. To address this issue, we consider the classic newsboy problem. The principles used to manage uncertainty for that case are applied to two stylized exposure examples, one for high dose and high dose rate radiation and the other for ammonia. Both incorporate expert judgment on uncertainty quantification in the dose-response relationship. The mathematical technique of probabilistic inversion also plays a key role. We propose a coupled approach, whereby scientists quantify the dose-response uncertainty using techniques such as structured expert judgment with performance weights and probabilistic inversion, and stakeholders quantify associated loss rates.     

Expert Elicitation of the Value per Statistical Life in an Air Pollution Context

The monetized value of avoided premature mortality typically dominates the calculated benefits of air pollution regulations; therefore, characterization of the uncertainty surrounding these estimates is key to good policymaking. Formal expert judgment elicitation methods are one means of characterizing this uncertainty. They have been applied to characterize uncertainty in the mortality concentration‐response function, but have yet to be used to characterize uncertainty in the economic values placed on avoided mortality. We report the findings of a pilot expert judgment study for Health Canada designed to elicit quantitative probabilistic judgments of uncertainties in Value‐per‐Statistical‐Life (VSL) estimates for use in an air pollution context. The two‐stage elicitation addressed uncertainties in both a base case VSL for a reduction in mortality risk from traumatic accidents and in benefits transfer‐related adjustments to the base case for an air quality application (e.g., adjustments for age, income, and health status). Results for each expert were integrated to develop example quantitative probabilistic uncertainty distributions for VSL that could be incorporated into air quality models.     

Using Expert Models in Human Reliability Analysis—A Dependence Assessment Method Based on Fuzzy Logic

In human reliability analysis (HRA), dependence analysis refers to assessing the influence of the failure of the operators to perform one task on the failure probabilities of subsequent tasks. A commonly used approach is the technique for human error rate prediction (THERP). The assessment of the dependence level in THERP is a highly subjective judgment based on general rules for the influence of five main factors. A frequently used alternative method extends the THERP model with decision trees. Such trees should increase the repeatability of the assessments but they simplify the relationships among the factors and the dependence level. Moreover, the basis for these simplifications and the resulting tree is difficult to trace. The aim of this work is a method for dependence assessment in HRA that captures the rules used by experts to assess dependence levels and incorporates this knowledge into an algorithm and software tool to be used by HRA analysts. A fuzzy expert system (FES) underlies the method. The method and the associated expert elicitation process are demonstrated with a working model. The expert rules are elicited systematically and converted into a traceable, explicit, and computable model. Anchor situations are provided as guidance for the HRA analyst's judgment of the input factors. The expert model and the FES‐based dependence assessment method make the expert rules accessible to the analyst in a usable and repeatable way, with an explicit and traceable basis.     

Updating Parameters of the Chicken Processing Line Model

A mathematical model of chicken processing that quantitatively describes the transmission of Campylobacter on chicken carcasses from slaughter to chicken meat product has been developed in Nauta et al. (2005). This model was quantified with expert judgment. Recent availability of data allows updating parameters of the model to better describe processes observed in slaughterhouses. We propose Bayesian updating as a suitable technique to update expert judgment with microbiological data. Berrang and Dickens's data are used to demonstrate performance of this method in updating parameters of the chicken processing line model.     

Risk Analysis for Autonomous Underwater Vehicle Operations in Extreme Environments

Autonomous underwater vehicles (AUVs) are used increasingly to explore hazardous marine environments. Risk assessment for such complex systems is based on subjective judgment and expert knowledge as much as on hard statistics. Here, we describe the use of a risk management process tailored to AUV operations, the implementation of which requires the elicitation of expert judgment. We conducted a formal judgment elicitation process where eight world experts in AUV design and operation were asked to assign a probability of AUV loss given the emergence of each fault or incident from the vehicle's life history of 63 faults and incidents. After discussing methods of aggregation and analysis, we show how the aggregated risk estimates obtained from the expert judgments were used to create a risk model. To estimate AUV survival with mission distance, we adopted a statistical survival function based on the nonparametric Kaplan‐Meier estimator. We present theoretical formulations for the estimator, its variance, and confidence limits. We also present a numerical example where the approach is applied to estimate the probability that the Autosub3 AUV would survive a set of missions under Pine Island Glacier, Antarctica in January–March 2009.     

Rapid Benefit-Risk Assessments: No Escape from Expert Judgments in Risk Management

The "human health impacts assessment" described by Cox and Popken (this issue) is intended to be a benefit-risk tool that avoids pitfalls of using expert judgments for policy analysis or during strict application of the precautionary principle in risk management. The proposed benefit-risk calculation uses numerous assumptions and suppositions to calculate a ratio of quality-adjusted life years (QALYs) lost for the number of human illness days prevented by the use of a food-animal antimicrobial drug, to the number of human illness days caused by the use of the antimicrobial drug. Assumptions about data--e.g., expert judgments on the representativeness of parameter estimates--are commonly used in risk assessment and risk management, including Cox and Popken's method. Cox and Popken apply the technique to specific examples of enrofloxacin and macrolides antimicrobial drugs, sometimes used in broiler chickens for human food. Although enthusiastically portrayed as a straightforward calculation of QALYs lost for two decision alternatives, Cox and Popken were silent on the pivotal expert judgment subsumed in their method: quality weights for illnesses caused by antimicrobial-resistant and antimicrobial-sensitive microbes are tacitly assumed to be equal. Yet, the costs in terms of prolonged illness, additional medications, repeat medical visits, and dread of more serious sequelae are expected to differ substantially for antimicrobial-resistant versus antimicrobial-sensitive illnesses. Despite their enthusiasm to the contrary, the "human health impacts assessment" by Cox and Popken is not immune from expert judgments in risk management.     

The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?

Good policy making should be based on available scientific knowledge. Sometimes this knowledge is well established through research, but often scientists must simply express their judgment, and this is particularly so in risk scenarios that are characterized by high levels of uncertainty. Usually in such cases, the opinions of several experts will be sought in order to pool knowledge and reduce error, raising the question of whether individual expert judgments should be given different weights. We argue—against the commonly advocated “classical method”—that no significant benefits are likely to accrue from unequal weighting in mathematical aggregation. Our argument hinges on the difficulty of constructing reliable and valid measures of substantive expertise upon which to base weights. Practical problems associated with attempts to evaluate experts are also addressed. While our discussion focuses on one specific weighting scheme that is currently gaining in popularity for expert knowledge elicitation, our general thesis applies to externally imposed unequal weighting schemes more generally.     

A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler-Chicken Processing

A structured expert judgment study was organized to obtain input data for a microbial risk-assessment model describing the transmission of campylobacter during broiler-chicken processing in the Netherlands. More specially, the expert study was aimed at quantifying the uncertainty on input parameters of this model and focused on the contamination of broiler-chicken carcasses with campylobacter during processing. Following the protocol for structured expert judgment studies, expert assessments were elicited individually through subjective probability distribution functions. The classical model was used to aggregate the individual experts' distributions in order to obtain a single combined distribution per variable. Three different weighting schemes were applied, including equal weighting and performance-based weighting with and without optimalization of the combined distributions. The individual experts' weights were based on their performance on the seed variables. Results of the various weighting schemes are presented in terms of performance, robustness, and combined distributions of the seed variables and some of the query variables. All three weighting schemes had adequate performance, with the optimized combined distributions significantly outperforming both the equal weight and the nonoptimized combined distributions. Hence, this weighting scheme, having adequate robustness, was chosen for further processing of the results.     

团队内社会网络质量、工作竞争对团队成员知识转移的影响研究

本研究以团队为对象,考察了在正式的工作关系背景下,当团队成员间存在工作竞争时,团队内社会网络对知识转移的影响.研究认为,反映社会网络质量的网络信任会促进团队内知识转移,而网络认知信任和网络情感信任对团队内知识转移具有不同的作用,其中网络情感信任的影响要强于网络认知信任.工作竞争具有调节这种影响的作用,并且对网络认知信任和情感信任的调节程度会有所不同.研究对所提假设进行了实证检验,并对结果进行了进一步解释,旨在进一步剖析知识活动的规律,从而为企业有效的管理知识提供指导.     

A Regional Decision Support Scheme for Pest Risk Analysis in Southeast Asia

A key justification to support plant health regulations is the ability of quarantine services to conduct pest risk analyses (PRA). Despite the supranational nature of biological invasions and the close proximity and connectivity of Southeast Asian countries, PRAs are conducted at the national level. Furthermore, some countries have limited experience in the development of PRAs, which may result in inadequate phytosanitary responses that put their plant resources at risk to pests vectored via international trade. We review existing decision support schemes for PRAs and, following international standards for phytosanitary measures, propose new methods that adapt existing practices to suit the unique characteristics of Southeast Asia. Using a formal written expert elicitation survey, a panel of regional scientific experts was asked to identify and rate unique traits of Southeast Asia with respect to PRA. Subsequently, an expert elicitation workshop with plant protection officials was conducted to verify the potential applicability of the developed methods. Rich biodiversity, shortage of trained personnel, social vulnerability, tropical climate, agriculture‐dependent economies, high rates of land‐use change, and difficulties in implementing risk management options were identified as challenging Southeast Asian traits. The developed methods emphasize local Southeast Asian conditions and could help support authorities responsible for carrying out PRAs within the region. These methods could also facilitate the creation of other PRA schemes in low‐ and middle‐income tropical countries.     

What Can Decision Analysis Do for Invasive Species Management?

Decisions about management of invasive species are difficult for all the reasons typically addressed by multiattribute decision analysis: uncertain outcomes, multiple and conflicting objectives, and many interested parties with differing views on both facts and values. This article illustrates how the tools of multiattribute analysis can improve management of invasive species, with an emphasis on making explicit the social values and preferences that must inform invasive species management. Risk assessment protocols developed previously for invasive species management typically suffer from two interacting flaws: (1) separating risk assessment from risk management, thus disrupting essential connections between the social values at stake in invasive species decisions and the scientific knowledge necessary to predict the likely impacts of management actions, and (2) relying on expert judgment about risk framed in qualitative and value-laden terms, inadvertently mixing the expert's judgment about what is likely to happen with personal preferences. Using the values structuring and probability-modeling elements of formal decision analysis can remedy these difficulties and make invasive species management responsive to both good science and public values. The management of feral pigs in Hawaiian ecosystems illustrates the need for such an integrated approach.     

Representation,Propagation, and Decision Issues in Risk Analysis Under Incomplete Probabilistic Information

This article tries to clarify the potential role to be played by uncertainty theories such as imprecise probabilities, random sets, and possibility theory in the risk analysis process. Instead of opposing an objective bounding analysis, where only statistically founded probability distributions are taken into account, to the full‐fledged probabilistic approach, exploiting expert subjective judgment, we advocate the idea that both analyses are useful and should be articulated with one another. Moreover, the idea that risk analysis under incomplete information is purely objective is misconceived. The use of uncertainty theories cannot be reduced to a choice between probability distributions and intervals. Indeed, they offer representation tools that are more expressive than each of the latter approaches and can capture expert judgments while being faithful to their limited precision. Consequences of this thesis are examined for uncertainty elicitation, propagation, and at the decision‐making step.     

Modeling Extreme Risks in Ecology

Extreme risks in ecology are typified by circumstances in which data are sporadic or unavailable, understanding is poor, and decisions are urgently needed. Expert judgments are pervasive and disagreements among experts are commonplace. We outline approaches to evaluating extreme risks in ecology that rely on stochastic simulation, with a particular focus on methods to evaluate the likelihood of extinction and quasi‐extinction of threatened species, and the likelihood of establishment and spread of invasive pests. We evaluate the importance of assumptions in these assessments and the potential of some new approaches to account for these uncertainties, including hierarchical estimation procedures and generalized extreme value distributions. We conclude by examining the treatment of consequences in extreme risk analysis in ecology and how expert judgment may better be harnessed to evaluate extreme risks.     

面向专家的知识库优化

知识库的质量是影响智能系统性能的主要因素 ,而知识获取一直是设计智能系统的瓶颈问题 ,这是由于目前人类认识的局限性 ,导致知识工程师和专家之间的不协调关系造成的 .为克服上述不利局面 ,本文利用粗糙集等理论 ,得到含有噪声的初始知识库 ,然后采用遗传算法、可视化技术和知识校验等技术对规则库和案例库进行了优化 .从而在知识获取过程中建立了知识工程师和专家之间的新型的关系 ,其中专家处于中心地位 ,知识工程师只是起辅助作用 ,即整个知识获取过程是面向专家的     

Combining Experts' Risk Judgments on Technology Performance of Phytoremediation: Self-Confidence Ratings, Averaging Procedures, and Formative Consensus Building

Expert panels and averaging procedures are common means for coping with the uncertainty of effects of technology application in complex environments. We investigate the connection between confidence and the validity of expert judgment. Moreover, a formative consensus building procedure (FCB) is introduced that generates probability statements on the performance of technologies, and we compare different algorithms for the statistical aggregation of individual judgments. The case study refers to an expert panel of 10 environmental scientists assessing the performance of a soil cleanup technology that uses the capability of certain plants to accumulate heavy metals from the soil in the plant body (phytoremediation). The panel members first provided individual statements on the effectiveness of a phytoremediation. Such statements can support policymakers, answering the questions concerning the expected performance of the new technology in contaminated areas. The present study reviews (1) the steps of the FCB, (2) the constraints of technology application (contaminants, soil structure, etc.), (3) the measurement of expert knowledge, (4) the statistical averaging and the discursive agreement procedures, and (5) the boundaries of application for the FCB method. The quantitative statement oriented part of FCB generates terms such as: "The probability that the concentration of soil contamination will be reduced by at least 50% is 0.8." The data suggest that taking the median of the individual expert estimates provides the most accurate aggregated estimate. The discursive agreement procedure of FCB appears suitable for deriving politically relevant singular statements rather than for obtaining comprehensive information about uncertainties as represented by probability distributions.