Structured expert elicitation about Listeria monocytogenes cross-contamination in the environment of retail deli operations in the United States

Listeria monocytogenes is among the foodborne pathogens with the highest death toll in the United States. Ready‐to‐eat foods contaminated at retail are an important source of infection. Environmental sites in retail deli operations can be contaminated. However, commonly contaminated sites are unlikely to come into direct contact with food and the public health relevance of environmental contamination has remained unclear. To identify environmental sites that may pose a considerable cross‐contamination risk, to elucidate potential transmission pathways, and to identify knowledge gaps, we performed a structured expert elicitation of 41 experts from state regulatory agencies and the food retail industry with practical experience in retail deli operations. Following the “Delphi” method, the elicitation was performed in three consecutive steps: questionnaire, review and discussion of results, second questionnaire. Hands and gloves were identified as important potential contamination sources. However, bacterial transfers to and from hands or gloves represented a major data gap. Experts agreed about transfer probabilities from cutting boards, scales, deli cases, and deli preparation sinks to product, and about transfer probabilities from floor drains, walk‐in cooler floors, and knife racks to food contact surfaces. Comparison of experts' opinions to observational data revealed a tendency among experts with certain demographic characteristics and professional opinions to overestimate prevalence. Experts’ votes clearly clustered into separate groups not defined by place of employment, even though industry experts may have been somewhat overrepresented in one cluster. Overall, our study demonstrates the value and caveats of expert elicitation to identify data gaps and prioritize research efforts.     

Capturing and Integrating Knowledge for Managing Risks in Tunnel Works

Risk‐related knowledge gained from past construction projects is regarded as potentially extremely useful in risk management. This article describes a proposed approach to capture and integrate risk‐related knowledge to support decision making in construction projects. To ameliorate the problem related to the scarcity of risks information often encountered in construction projects, Bayesian Belief Networks are used and expert judgment is elicited to augment available information. Particularly, the article provides an overview of judgment‐based biases that can appear in the elicitation of judgments for constructing Bayesian Networks and the provisos that can be made in this respect to minimize these types of bias. The proposed approach is successfully applied to develop six models for top risks in tunnel works. More than 30 tunneling experts in the Netherlands and Germany were involved in the investigation to provide information on identifying relevant scenarios than can lead to failure events associated with tunneling risks. The article has provided an illustration of the applicability of the developed approach for the case of “face instability in soft soils using slurry shields.”     

Quantifying experts' uncertainty about the future cost of exotic diseases

Since the foot-and-mouth disease outbreak of 2001 in the United Kingdom, there has been debate about the sharing, between government and industry, both the costs of livestock disease outbreaks and responsibility for the decisions that give rise to them. As part of a consultation into the formation of a new body to manage livestock diseases, government veterinarians and economists produced estimates of the average annual costs for a number of exotic infectious diseases. In this article, we demonstrate how the government experts were helped to quantify their uncertainties about the cost estimates using formal expert elicitation techniques. This has enabled the decisionmakers to have a greater appreciation of government experts' uncertainty in this policy area.     

Expert Elicitation of the Value per Statistical Life in an Air Pollution Context

The monetized value of avoided premature mortality typically dominates the calculated benefits of air pollution regulations; therefore, characterization of the uncertainty surrounding these estimates is key to good policymaking. Formal expert judgment elicitation methods are one means of characterizing this uncertainty. They have been applied to characterize uncertainty in the mortality concentration‐response function, but have yet to be used to characterize uncertainty in the economic values placed on avoided mortality. We report the findings of a pilot expert judgment study for Health Canada designed to elicit quantitative probabilistic judgments of uncertainties in Value‐per‐Statistical‐Life (VSL) estimates for use in an air pollution context. The two‐stage elicitation addressed uncertainties in both a base case VSL for a reduction in mortality risk from traumatic accidents and in benefits transfer‐related adjustments to the base case for an air quality application (e.g., adjustments for age, income, and health status). Results for each expert were integrated to develop example quantitative probabilistic uncertainty distributions for VSL that could be incorporated into air quality models.     

Review and Assessment of Poliovirus Immunity and Transmission: Synthesis of Knowledge Gaps and Identification of Research Needs

With the intensifying global efforts to eradicate wild polioviruses, policymakers face complex decisions related to achieving eradication and managing posteradication risks. These decisions and the expanding use of inactivated poliovirus vaccine (IPV) trigger renewed interest in poliovirus immunity, particularly the role of mucosal immunity in the transmission of polioviruses. Sustained high population immunity to poliovirus transmission represents a key prerequisite to eradication, but poliovirus immunity and transmission remain poorly understood despite decades of studies. In April 2010, the U.S. Centers for Disease Control and Prevention convened an international group of experts on poliovirus immunology and virology to review the literature relevant for modeling poliovirus transmission, develop a consensus about related uncertainties, and identify research needs. This article synthesizes the quantitative assessments and research needs identified during the process. Limitations in the evidence from oral poliovirus vaccine (OPV) challenge studies and other relevant data led to differences in expert assessments, indicating the need for additional data, particularly in several priority areas for research: (1) the ability of IPV‐induced immunity to prevent or reduce excretion and affect transmission, (2) the impact of waning immunity on the probability and extent of poliovirus excretion, (3) the relationship between the concentration of poliovirus excreted and infectiousness to others in different settings, and (4) the relative role of fecal‐oral versus oropharyngeal transmission. This assessment of current knowledge supports the immediate conduct of additional studies to address the gaps.     

A Regional Decision Support Scheme for Pest Risk Analysis in Southeast Asia

A key justification to support plant health regulations is the ability of quarantine services to conduct pest risk analyses (PRA). Despite the supranational nature of biological invasions and the close proximity and connectivity of Southeast Asian countries, PRAs are conducted at the national level. Furthermore, some countries have limited experience in the development of PRAs, which may result in inadequate phytosanitary responses that put their plant resources at risk to pests vectored via international trade. We review existing decision support schemes for PRAs and, following international standards for phytosanitary measures, propose new methods that adapt existing practices to suit the unique characteristics of Southeast Asia. Using a formal written expert elicitation survey, a panel of regional scientific experts was asked to identify and rate unique traits of Southeast Asia with respect to PRA. Subsequently, an expert elicitation workshop with plant protection officials was conducted to verify the potential applicability of the developed methods. Rich biodiversity, shortage of trained personnel, social vulnerability, tropical climate, agriculture‐dependent economies, high rates of land‐use change, and difficulties in implementing risk management options were identified as challenging Southeast Asian traits. The developed methods emphasize local Southeast Asian conditions and could help support authorities responsible for carrying out PRAs within the region. These methods could also facilitate the creation of other PRA schemes in low‐ and middle‐income tropical countries.     

Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.     

Using Expert Models in Human Reliability Analysis—A Dependence Assessment Method Based on Fuzzy Logic

In human reliability analysis (HRA), dependence analysis refers to assessing the influence of the failure of the operators to perform one task on the failure probabilities of subsequent tasks. A commonly used approach is the technique for human error rate prediction (THERP). The assessment of the dependence level in THERP is a highly subjective judgment based on general rules for the influence of five main factors. A frequently used alternative method extends the THERP model with decision trees. Such trees should increase the repeatability of the assessments but they simplify the relationships among the factors and the dependence level. Moreover, the basis for these simplifications and the resulting tree is difficult to trace. The aim of this work is a method for dependence assessment in HRA that captures the rules used by experts to assess dependence levels and incorporates this knowledge into an algorithm and software tool to be used by HRA analysts. A fuzzy expert system (FES) underlies the method. The method and the associated expert elicitation process are demonstrated with a working model. The expert rules are elicited systematically and converted into a traceable, explicit, and computable model. Anchor situations are provided as guidance for the HRA analyst's judgment of the input factors. The expert model and the FES‐based dependence assessment method make the expert rules accessible to the analyst in a usable and repeatable way, with an explicit and traceable basis.     

Risk Perception in the U.K. Oil and Gas Production Industry: Are Expert Loss‐Prevention Managers' Perceptions Different From Those of Members of the Public?

This article investigates potential differences in risk perception between experts (loss-prevention managers in the U.K. oil and gas production industry) and nonexperts (managers and students). Extant research on expert versus nonexpert perceptions of risk is reviewed, followed by the present study concerning risk perceptions of seven pen-picture scenarios involving the occurrence of hazardous events in the U.K. oil and gas production industry. In contrast to many of the earlier studies of expert versus nonexpert perceptions of risk, the present analysis concludes that experts did not judge the overall riskiness of the portrayed hazardous events as less risky than the nonexperts. Nevertheless, the experts believe more strongly than our nonexperts that the risks portrayed in the scenarios pose little threat to future generations, are more precisely known, and are relatively controllable. Use of multiple regression analysis to help uncover the basis of overall riskiness assessments for expert and lay respondents was inconclusive, however. Finally, little evidence was found that nonexperts were any more heterogeneous in their risk perceptions than experts. It may be that the nature of the risks assessed in the present study may account for the general lack of clear expert versus nonexpert differences in overall perceptions of the riskiness of hazardous events in the North Sea. Earlier findings of strong expert versus nonexpert differences in risk perception assessed hazards of major public concern. It is inferred that using such extreme hazards may have resulted in an exaggerated view of differences in expert versus public (nonexpert) perception of risk.     

Toward Improved Communication about Wildland Fire: Mental Models Research to Identify Information Needs for Natural Resource Management

The management of natural- and human-induced wildland fires is an intricate process that must balance two considerations: that of fire as a necessary natural disturbance and that of the risks that fire poses. Reconciling tradeoffs between these risks and benefits is contingent upon informed, directed, and two-way communication between wildland fire managers and stakeholders. In an effort to aid with the design of such a communication effort, this study used mental models research to determine the unique wildland fire information and decision-making needs of stakeholders living at the wildland-urban interface of a fire-prone area. While the analysis revealed many similarities in how stakeholders conceptualized the risks and benefits of wildland fire, many misconceptions and important gaps in understanding on the part of both expert and nonexpert respondents were identified.     

Explosion Probability of Unexploded Ordnance: Expert Beliefs

This article reports on a study to quantify expert beliefs about the explosion probability of unexploded ordnance (UXO). Some 1,976 sites at closed military bases in the United States are contaminated with UXO and are slated for cleanup, at an estimated cost of $15–140 billion. Because no available technology can guarantee 100% removal of UXO, information about explosion probability is needed to assess the residual risks of civilian reuse of closed military bases and to make decisions about how much to invest in cleanup. This study elicited probability distributions for the chance of UXO explosion from 25 experts in explosive ordnance disposal, all of whom have had field experience in UXO identification and deactivation. The study considered six different scenarios: three different types of UXO handled in two different ways (one involving children and the other involving construction workers). We also asked the experts to rank by sensitivity to explosion 20 different kinds of UXO found at a case study site at Fort Ord, California. We found that the experts do not agree about the probability of UXO explosion, with significant differences among experts in their mean estimates of explosion probabilities and in the amount of uncertainty that they express in their estimates. In three of the six scenarios, the divergence was so great that the average of all the expert probability distributions was statistically indistinguishable from a uniform (0, 1) distribution—suggesting that the sum of expert opinion provides no information at all about the explosion risk. The experts' opinions on the relative sensitivity to explosion of the 20 UXO items also diverged. The average correlation between rankings of any pair of experts was 0.41, which, statistically, is barely significant (p= 0.049) at the 95% confidence level. Thus, one expert's rankings provide little predictive information about another's rankings. The lack of consensus among experts suggests that empirical studies are needed to better understand the explosion risks of UXO.     

Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data

Expert judgment (or expert elicitation) is a formal process for eliciting judgments from subject‐matter experts about the value of a decision‐relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal‐weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal‐weight, best‐expert, performance, frequentist, and copula) using simulated expert‐judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well‐calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best‐expert approaches perform better and the equal‐weight combination method performs worse than the alternative approaches.     

Using Probability Boxes to Model Elicited Information: A Case Study

Governments are responsible for making policy decisions, often in the face of severe uncertainty about the factors involved. Expert elicitation can be used to fill information gaps where data are not available, cannot be obtained, or where there is no time for a full‐scale study and analysis. Various features of distributions for variables may be elicited, for example, the mean, standard deviation, and quantiles, but uncertainty about these values is not always recorded. Distributional and dependence assumptions often have to be made in models and although these are sometimes elicited from experts, modelers may also make assumptions for mathematical convenience (e.g., assuming independence between variables). Probability boxes (p‐boxes) provide a flexible methodology to analyze elicited quantities without having to make assumptions about the distribution shape. If information about distribution shape(s) is available, p‐boxes can provide bounds around the results given these possible input distributions. P‐boxes can also be used to combine variables without making dependence assumptions. This article aims to illustrate how p‐boxes may help to improve the representation of uncertainty for analyses based on elicited information. We focus on modeling elicited quantiles with nonparametric p‐boxes, modeling elicited quantiles with parametric p‐boxes where the elicited quantiles do not match the elicited distribution shape, and modeling elicited interval information.     

Probability Elicitation Under Severe Time Pressure: A Rank‐Based Method

Probability elicitation protocols are used to assess and incorporate subjective probabilities in risk and decision analysis. While most of these protocols use methods that have focused on the precision of the elicited probabilities, the speed of the elicitation process has often been neglected. However, speed is also important, particularly when experts need to examine a large number of events on a recurrent basis. Furthermore, most existing elicitation methods are numerical in nature, but there are various reasons why an expert would refuse to give such precise ratio‐scale estimates, even if highly numerate. This may occur, for instance, when there is lack of sufficient hard evidence, when assessing very uncertain events (such as emergent threats), or when dealing with politicized topics (such as terrorism or disease outbreaks). In this article, we adopt an ordinal ranking approach from multicriteria decision analysis to provide a fast and nonnumerical probability elicitation process. Probabilities are subsequently approximated from the ranking by an algorithm based on the principle of maximum entropy, a rule compatible with the ordinal information provided by the expert. The method can elicit probabilities for a wide range of different event types, including new ways of eliciting probabilities for stochastically independent events and low‐probability events. We use a Monte Carlo simulation to test the accuracy of the approximated probabilities and try the method in practice, applying it to a real‐world risk analysis recently conducted for DEFRA (the U.K. Department for the Environment, Farming and Rural Affairs): the prioritization of animal health threats.     

Classification of Listeria monocytogenes Persistence in Retail Delicatessen Environments Using Expert Elicitation and Machine Learning

Increasing evidence suggests that persistence of Listeria monocytogenes in food processing plants has been the underlying cause of a number of human listeriosis outbreaks. This study extracts criteria used by food safety experts in determining bacterial persistence in the environment, using retail delicatessen operations as a model. Using the Delphi method, we conducted an expert elicitation with 10 food safety experts from academia, industry, and government to classify L. monocytogenes persistence based on environmental sampling results collected over six months for 30 retail delicatessen stores. The results were modeled using variations of random forest, support vector machine, logistic regression, and linear regression; variable importance values of random forest and support vector machine models were consolidated to rank important variables in the experts’ classifications. The duration of subtype isolation ranked most important across all expert categories. Sampling site category also ranked high in importance and validation errors doubled when this covariate was removed. Support vector machine and random forest models successfully classified the data with average validation errors of 3.1% and 2.2% (n = 144), respectively. Our findings indicate that (i) the frequency of isolations over time and sampling site information are critical factors for experts determining subtype persistence, (ii) food safety experts from different sectors may not use the same criteria in determining persistence, and (iii) machine learning models have potential for future use in environmental surveillance and risk management programs. Future work is necessary to validate the accuracy of expert and machine classification against biological measurement of L. monocytogenes persistence.     

Eliciting Information from Experts on the Likelihood of Rapid Climate Change

The threat of so‐called rapid or abrupt climate change has generated considerable public interest because of its potentially significant impacts. The collapse of the North Atlantic Thermohaline Circulation or the West Antarctic Ice Sheet, for example, would have potentially catastrophic effects on temperatures and sea level, respectively. But how likely are such extreme climatic changes? Is it possible actually to estimate likelihoods? This article reviews the societal demand for the likelihoods of rapid or abrupt climate change, and different methods for estimating likelihoods: past experience, model simulation, or through the elicitation of expert judgments. The article describes a survey to estimate the likelihoods of two characterizations of rapid climate change, and explores the issues associated with such surveys and the value of information produced. The surveys were based on key scientists chosen for their expertise in the climate science of abrupt climate change. Most survey respondents ascribed low likelihoods to rapid climate change, due either to the collapse of the Thermohaline Circulation or increased positive feedbacks. In each case one assessment was an order of magnitude higher than the others. We explore a high rate of refusal to participate in this expert survey: many scientists prefer to rely on output from future climate model simulations.     

Toward Disaster‐Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments

Resilient infrastructure systems are essential for cities to withstand and rapidly recover from natural and human‐induced disasters, yet electric power, transportation, and other infrastructures are highly vulnerable and interdependent. New approaches for characterizing the resilience of sets of infrastructure systems are urgently needed, at community and regional scales. This article develops a practical approach for analysts to characterize a community's infrastructure vulnerability and resilience in disasters. It addresses key challenges of incomplete incentives, partial information, and few opportunities for learning. The approach is demonstrated for Metro Vancouver, Canada, in the context of earthquake and flood risk. The methodological approach is practical and focuses on potential disruptions to infrastructure services. In spirit, it resembles probability elicitation with multiple experts; however, it elicits disruption and recovery over time, rather than uncertainties regarding system function at a given point in time. It develops information on regional infrastructure risk and engages infrastructure organizations in the process. Information sharing, iteration, and learning among the participants provide the basis for more informed estimates of infrastructure system robustness and recovery that incorporate the potential for interdependent failures after an extreme event. Results demonstrate the vital importance of cross‐sectoral communication to develop shared understanding of regional infrastructure disruption in disasters. For Vancouver, specific results indicate that in a hypothetical M7.3 earthquake, virtually all infrastructures would suffer severe disruption of service in the immediate aftermath, with many experiencing moderate disruption two weeks afterward. Electric power, land transportation, and telecommunications are identified as core infrastructure sectors.     

Different Roles and Viewpoints of Scientific Experts in Advising on Environmental Health Risks

Environmental health risks are often complex, largescale, and uncertain. The uncertainties inherent in these problems permit differences among experts in the appraisal of risks. This raises the question of whether different expert roles exist and, if so, how this affects the policy advice that is given. Here, we present a pilot study of the different roles and viewpoints that can be discerned among scientific experts in the Netherlands. Q methodology was used to empirically explore existing theoretical treatises on different expert roles. In total, 26 electromag­netic field (EMF) experts and 21 particulate matter (PM) experts participated. The responses were analyzed separately for EMF and PM respondents using Q factor analysis. In both the EMF and PM domain, three different expert roles were identified. This suggests that particular expert roles depend on the specific environmental health risk. The results indicate that different expert roles exist among scientists who provide policy advice on environmental health risks. This empirical study adds new data and insights to the literature on expert roles. The results of this study are relevant for the selection and composition of expert committees and the interpretation of expert advice.     

Risk Analysis for U.S. Offshore Wind Farms: The Need for an Integrated Approach

Wind power is becoming an increasingly important part of the global energy portfolio, and there is growing interest in developing offshore wind farms in the United States to better utilize this resource. Wind farms have certain environmental benefits, notably near‐zero emissions of greenhouse gases, particulates, and other contaminants of concern. However, there are significant challenges ahead in achieving large‐scale integration of wind power in the United States, particularly offshore wind. Environmental impacts from wind farms are a concern, and these are subject to a number of on‐going studies focused on risks to the environment. However, once a wind farm is built, the farm itself will face a number of risks from a variety of hazards, and managing these risks is critical to the ultimate achievement of long‐term reductions in pollutant emissions from clean energy sources such as wind. No integrated framework currently exists for assessing risks to offshore wind farms in the United States, which poses a challenge for wind farm risk management. In this “Perspective”, we provide an overview of the risks faced by an offshore wind farm, argue that an integrated framework is needed, and give a preliminary starting point for such a framework to illustrate what it might look like. This is not a final framework; substantial work remains. Our intention here is to highlight the research need in this area in the hope of spurring additional research about the risks to wind farms to complement the substantial amount of on‐going research on the risks from wind farms.     

The Value of Performance Weights and Discussion in Aggregated Expert Judgments

In risky situations characterized by imminent decisions, scarce resources, and insufficient data, policymakers rely on experts to estimate model parameters and their associated uncertainties. Different elicitation and aggregation methods can vary substantially in their efficacy and robustness. While it is generally agreed that biases in expert judgments can be mitigated using structured elicitations involving groups rather than individuals, there is still some disagreement about how to best elicit and aggregate judgments. This mostly concerns the merits of using performance‐based weighting schemes to combine judgments of different individuals (rather than assigning equal weights to individual experts), and the way that interaction between experts should be handled. This article aims to contribute to, and complement, the ongoing discussion on these topics.