Multi‐Exposure Pathway Model to Compare Escherichia coli O157 Risks and Interventions

The relative contributions of exposure pathways associated with cattle‐manure‐borne Escherichia coli O157:H7 on public health have yet to be fully characterized. A stochastic, quantitative microbial risk assessment (QMRA) model was developed to describe a hypothetical cattle farm in order to compare the relative importance of five routes of exposure, including aquatic recreation downstream of the farm, consumption of contaminated ground beef processed with limited interventions, consumption of leafy greens, direct animal contact, and the recreational use of a cattle pasture. To accommodate diverse environmental and hydrological pathways, existing QMRAs were integrated with novel and simplistic climate and field‐level submodels. The model indicated that direct animal contact presents the greatest risk of illness per exposure event during the high pathogen shedding period. However, when accounting for the frequency of exposure, using a high‐risk exposure‐receptor profile, consumption of ground beef was associated with the greatest risk of illness. Additionally, the model was used to evaluate the efficacy of hypothetical interventions affecting one or more exposure routes; concurrent evaluation of multiple routes allowed for the assessment of the combined effect of preharvest interventions across exposure pathways—which may have been previously underestimated—as well as the assessment of the effect of additional downstream interventions. This analysis represents a step towards a full evaluation of the risks associated with multiple exposure pathways; future incorporation of variability associated with environmental parameters and human behaviors would allow for a comprehensive assessment of the relative contribution of exposure pathways at the population level.     

Dose Response for Infection by Escherichia coli O157:H7 from Outbreak Data

In 1996, an outbreak of E. coli O157:H7-associated illness occurred in an elementary school in Japan. This outbreak has been studied in unusual detail, making this an important case for quantitative risk assessment. The availability of stored samples of the contaminated food allowed reliable estimation of exposure to the pathogens. Collection of fecal samples allowed assessment of the numbers infected, including asymptomatic cases. Comparison to other published dose-response studies for E. coli O157:H7 show that the strain that caused the outbreak studied here must have been considerably more infectious. We use this well-documented incident as an example to demonstrate how such information on the response to a single dose can be used for dose-response assessment. In particular, we demonstrate how the high infectivity limits the uncertainty in the low-dose region.     

A Mathematical Model for Pathogen Cross‐Contamination Dynamics during the Postharvest Processing of Leafy Greens

We developed a probabilistic mathematical model for the postharvest processing of leafy greens focusing on Escherichia coli O157:H7 contamination of fresh‐cut romaine lettuce as the case study. Our model can (i) support the investigation of cross‐contamination scenarios, and (ii) evaluate and compare different risk mitigation options. We used an agent‐based modeling framework to predict the pathogen prevalence and levels in bags of fresh‐cut lettuce and quantify spread of E. coli O157:H7 from contaminated lettuce to surface areas of processing equipment. Using an unbalanced factorial design, we were able to propagate combinations of random values assigned to model inputs through different processing steps and ranked statistically significant inputs with respect to their impacts on selected model outputs. Results indicated that whether contamination originated on incoming lettuce heads or on the surface areas of processing equipment, pathogen prevalence among bags of fresh‐cut lettuce and batches was most significantly impacted by the level of free chlorine in the flume tank and frequency of replacing the wash water inside the tank. Pathogen levels in bags of fresh‐cut lettuce were most significantly influenced by the initial levels of contamination on incoming lettuce heads or surface areas of processing equipment. The influence of surface contamination on pathogen prevalence or levels in fresh‐cut bags depended on the location of that surface relative to the flume tank. This study demonstrates that developing a flexible yet mathematically rigorous modeling tool, a “virtual laboratory,” can provide valuable insights into the effectiveness of individual and combined risk mitigation options.     

Risk-Based Analysis of the Danish Pork Salmonella Program: Past and Future

The Danish pork Salmonella control program was initiated in 1993 in response to a prominent pork-related outbreak in Copenhagen. It involved improved efforts at slaughter hygiene (postharvest) and on-farm (preharvest) surveillance and control. After 10 years, 95 million Euros, significant reductions in seropositive herds, Salmonella positive carcasses, and pork-attributable human cases (PAHC), questions have arisen about how best to continue this program. The objective of this study was to provide some analysis and information to address these questions. The methods used include a computer simulation model constructed of a series of Excel workbooks, one for each simulated year and scenario ( http://www.ifss.iastate/DanSalmRisk ). Each workbook has three modules representing the key processes affecting risk: seropositive pigs leaving the farm (Production), carcass contamination after slaughter (Slaughter), and PAHC of Salmonella (Attribution). Parameter estimates are derived from an extensive farm-to-fork database collected by industry and government and managed by the Danish Zoonosis Centre ( http://www.food.dtu.dk ). Retrospective (1994–2003) and prospective (2004–2013) simulations were evaluated. The retrospective simulations showed that, except for the first few years (1994–1998), the on-farm program had minimal impact in reducing the number of positive carcasses and PAHC. Most of the reductions in PAHC up to 2003 were, according to this analysis, due to various improvements in abattoir processes. Prospective simulations showed that minimal reductions in human health risk (PAHC) could be achieved with on-farm programs alone. Carcass decontamination was shown as the most effective means of reducing human risk, reducing PAHC to about 10% of the simulated 2004 level.     

Sensitivity Analysis of a Two‐Dimensional Quantitative Microbiological Risk Assessment: Keeping Variability and Uncertainty Separated

The aim of quantitative microbiological risk assessment is to estimate the risk of illness caused by the presence of a pathogen in a food type, and to study the impact of interventions. Because of inherent variability and uncertainty, risk assessments are generally conducted stochastically, and if possible it is advised to characterize variability separately from uncertainty. Sensitivity analysis allows to indicate to which of the input variables the outcome of a quantitative microbiological risk assessment is most sensitive. Although a number of methods exist to apply sensitivity analysis to a risk assessment with probabilistic input variables (such as contamination, storage temperature, storage duration, etc.), it is challenging to perform sensitivity analysis in the case where a risk assessment includes a separate characterization of variability and uncertainty of input variables. A procedure is proposed that focuses on the relation between risk estimates obtained by Monte Carlo simulation and the location of pseudo‐randomly sampled input variables within the uncertainty and variability distributions. Within this procedure, two methods are used—that is, an ANOVA‐like model and Sobol sensitivity indices—to obtain and compare the impact of variability and of uncertainty of all input variables, and of model uncertainty and scenario uncertainty. As a case study, this methodology is applied to a risk assessment to estimate the risk of contracting listeriosis due to consumption of deli meats.     

Quantitative risk assessment-epidemic curve prediction model for leafy green outbreak investigation

The objective of this study was to leverage quantitative risk assessment to investigate possible root cause(s) of foodborne illness outbreaks related to Shiga toxin-producing Escherichia coli O157:H7 (STEC O157) infections in leafy greens in the United States. To this end, we developed the FDA leafy green quantitative risk assessment epidemic curve prediction model (FDA-LG QRA-EC) that simulated the lettuce supply chain. The model was used to predict the number of reported illnesses and the epidemic curve associated with lettuce contaminated with STEC O157 for a wide range of scenarios representing various contamination conditions and facility processing/sanitation practices. Model predictions were generated for fresh-cut and whole lettuce, quantifying the differing impacts of facility processing and home preparation on predicted illnesses. Our model revealed that the timespan (i.e., number of days with at least one reported illness) and the peak (i.e., day with the most predicted number of reported illnesses) of the epidemic curve of a STEC O157-lettuce outbreak were not strongly influenced by facility processing/sanitation practices and were indications of contamination pattern among incoming lettuce batches received by the facility or distribution center. Through comparisons with observed number of illnesses from recent STEC O157-lettuce outbreaks, the model identified contamination conditions on incoming lettuce heads that could result in an outbreak of similar size, which can be used to narrow down potential root cause hypotheses.     

A Predictive Model for Survival of Escherichia coli O157:H7 and Generic E. coli in Soil Amended with Untreated Animal Manure

This study aimed at developing a predictive model that captures the influences of a variety of agricultural and environmental variables and is able to predict the concentrations of enteric bacteria in soil amended with untreated Biological Soil Amendments of Animal Origin (BSAAO) under dynamic conditions. We developed and validated a Random Forest model using data from a longitudinal field study conducted in mid-Atlantic United States investigating the survival of Escherichia coli O157:H7 and generic E. coli in soils amended with untreated dairy manure, horse manure, or poultry litter. Amendment type, days of rain since the previous sampling day, and soil moisture content were identified as the most influential agricultural and environmental variables impacting concentrations of viable E. coli O157:H7 and generic E. coli recovered from amended soils. Our model results also indicated that E. coli O157:H7 and generic E. coli declined at similar rates in amended soils under dynamic field conditions.The Random Forest model accurately predicted changes in viable E. coli concentrations over time under different agricultural and environmental conditions. Our model also accurately characterized the variability of E. coli concentration in amended soil over time by providing upper and lower prediction bound estimates. Cross-validation results indicated that our model can be potentially generalized to other geographic regions and incorporated into a risk assessment for evaluating the risks associated with application of untreated BSAAO. Our model can be validated for other regions and predictive performance also can be enhanced when data sets from additional geographic regions become available.     

Risk‐Based Decision Making for Reoccupation of Contaminated Areas Following a Wide‐Area Anthrax Release

This article presents an analysis of postattack response strategies to mitigate the risks of reoccupying contaminated areas following a release of Bacillus anthracis spores (the bacterium responsible for causing anthrax) in an urban setting. The analysis is based on a hypothetical attack scenario in which individuals are exposed to B. anthracis spores during an initial aerosol release and then placed on prophylactic antibiotics that successfully protect them against the initial aerosol exposure. The risk from reoccupying buildings contaminated with spores due to their reaerosolization and inhalation is then evaluated. The response options considered include: decontamination of the buildings, vaccination of individuals reoccupying the buildings, extended evacuation of individuals from the contaminated buildings, and combinations of these options. The study uses a decision tree to estimate the costs and benefits of alternative response strategies across a range of exposure risks. Results for best estimates of model inputs suggest that the most cost‐effective response for high‐risk scenarios (individual chance of infection exceeding 11%) consists of evacuation and building decontamination. For infection risks between 4% and 11%, the preferred option is to evacuate for a short period, vaccinate, and then reoccupy once the vaccine has taken effect. For risks between 0.003% and 4%, the preferred option is to vaccinate only. For risks below 0.003%, none of the mitigation actions have positive expected monetary benefits. A sensitivity analysis indicates that for high‐infection‐likelihood scenarios, vaccination is recommended in the case where decontamination efficacy is less than 99.99%.     

A Systems‐Based Risk Assessment Framework for Intentional Electromagnetic Interference (IEMI) on Critical Infrastructures

Modern infrastructures are becoming increasingly dependent on electronic systems, leaving them more vulnerable to electrical surges or electromagnetic interference. Electromagnetic disturbances appear in nature, e.g., lightning and solar wind; however, they may also be generated by man‐made technology to maliciously damage or disturb electronic equipment. This article presents a systematic risk assessment framework for identifying possible, consequential, and plausible intentional electromagnetic interference (IEMI) attacks on an arbitrary distribution network infrastructure. In the absence of available data on IEMI occurrences, we find that a systems‐based risk assessment is more useful than a probabilistic approach. We therefore modify the often applied definition of risk, i.e., a set of triplets containing scenario, probability, and consequence, to a set of quadruplets: scenario, resource requirements, plausibility, and consequence. Probability is “replaced” by resource requirements and plausibility, where the former is the minimum amount and type of equipment necessary to successfully carry out an attack scenario and the latter is a subjective assessment of the extent of the existence of attackers who possess the motivation, knowledge, and resources necessary to carry out the scenario. We apply the concept of intrusion areas and classify electromagnetic source technology according to key attributes. Worst‐case scenarios are identified for different quantities of attacker resources. The most plausible and consequential of these are deemed the most important scenarios and should provide useful decision support in a countermeasures effort. Finally, an example of the proposed risk assessment framework, based on notional data, is provided on a hypothetical water distribution network.     

Supply Chain Vulnerability Analysis Using Scenario‐Based Input‐Output Modeling: Application to Port Operations

Disruptive events such as natural disasters, loss or reduction of resources, work stoppages, and emergent conditions have potential to propagate economic losses across trade networks. In particular, disruptions to the operation of container port activity can be detrimental for international trade and commerce. Risk assessment should anticipate the impact of port operation disruptions with consideration of how priorities change due to uncertain scenarios and guide investments that are effective and feasible for implementation. Priorities for protective measures and continuity of operations planning must consider the economic impact of such disruptions across a variety of scenarios. This article introduces new performance metrics to characterize resiliency in interdependency modeling and also integrates scenario‐based methods to measure economic sensitivity to sudden‐onset disruptions. The methods will be demonstrated on a U.S. port responsible for handling $36.1 billion of cargo annually. The methods will be useful to port management, private industry supply chain planning, and transportation infrastructure management.     

AbSRiM: An Agent‐Based Security Risk Management Approach for Airport Operations

Security risk management is essential for ensuring effective airport operations. This article introduces AbSRiM, a novel agent‐based modeling and simulation approach to perform security risk management for airport operations that uses formal sociotechnical models that include temporal and spatial aspects. The approach contains four main steps: scope selection, agent‐based model definition, risk assessment, and risk mitigation. The approach is based on traditional security risk management methodologies, but uses agent‐based modeling and Monte Carlo simulation at its core. Agent‐based modeling is used to model threat scenarios, and Monte Carlo simulations are then performed with this model to estimate security risks. The use of the AbSRiM approach is demonstrated with an illustrative case study. This case study includes a threat scenario in which an adversary attacks an airport terminal with an improvised explosive device. The approach provides a promising way to include important elements, such as human aspects and spatiotemporal aspects, in the assessment of risk. More research is still needed to better identify the strengths and weaknesses of the AbSRiM approach in different case studies, but results demonstrate the feasibility of the approach and its potential.     

Optimal Reserve Prices in Name‐Your‐Own‐Price Auctions with Bidding and Channel Options

Few papers have explored the optimal reserve prices in the name‐your‐own‐price (NYOP) channel with bidding options in a multiple channel environment. In this paper, we investigate a double‐bid business model in which the consumers can bid twice in the NYOP channel, and compare it with the single‐bid case. We also study the impact of adding a retailer‐own list‐price channel on the optimal reserve prices. This paper focuses on achieving some basic understanding on the potential gain of adding a second bid option to a single‐bid system and on the potential benefits of adding a list‐price channel by the NYOP retailer. We show that a double‐bid scenario can outperform a single‐bid scenario in both single‐channel and dual‐channel situations. The optimal reserve price in the double‐bid scenario is no less than that in the single‐bid case. Furthermore, the addition of a retailer‐own list‐price channel could push up the reserve prices in both single‐bid and double‐bid scenarios.     

A Case Study Evaluating the Risk of Infection from Middle Eastern Respiratory Syndrome Coronavirus (MERS‐CoV) in a Hospital Setting Through Bioaerosols

Middle Eastern respiratory syndrome, an emerging viral infection with a global case fatality rate of 35.5%, caused major outbreaks first in 2012 and 2015, though new cases are continuously reported around the world. Transmission is believed to mainly occur in healthcare settings through aerosolized particles. This study uses Quantitative Microbial Risk Assessment to develop a generalizable model that can assist with interpreting reported outbreak data or predict risk of infection with or without the recommended strategies. The exposure scenario includes a single index patient emitting virus‐containing aerosols into the air by coughing, leading to short‐ and long‐range airborne exposures for other patients in the same room, nurses, healthcare workers, and family visitors. Aerosol transport modeling was coupled with Monte Carlo simulation to evaluate the risk of MERS illness for the exposed population. Results from a typical scenario show the daily mean risk of infection to be the highest for the nurses and healthcare workers (8.49 × 10^?4 and 7.91 × 10^?4, respectively), and the lowest for family visitors and patients staying in the same room (3.12 × 10^?4 and 1.29 × 10^?4, respectively). Sensitivity analysis indicates that more than 90% of the uncertainty in the risk characterization is due to the viral concentration in saliva. Assessment of risk interventions showed that respiratory masks were found to have a greater effect in reducing the risks for all the groups evaluated (>90% risk reduction), while increasing the air exchange was effective for the other patients in the same room only (up to 58% risk reduction).     

Disease Spread Models to Estimate Highly Uncertain Emerging Diseases Losses for Animal Agriculture Insurance Policies: An Application to the U.S. Farm‐Raised Catfish Industry

Emerging diseases (ED) can have devastating effects on agriculture. Consequently, agricultural insurance for ED can develop if basic insurability criteria are met, including the capability to estimate the severity of ED outbreaks with associated uncertainty. The U.S. farm‐raised channel catfish (Ictalurus punctatus) industry was used to evaluate the feasibility of using a disease spread simulation modeling framework to estimate the potential losses from new ED for agricultural insurance purposes. Two stochastic models were used to simulate the spread of ED between and within channel catfish ponds in Mississippi (MS) under high, medium, and low disease impact scenarios. The mean (95% prediction interval (PI)) proportion of ponds infected within disease‐impacted farms was 7.6% (3.8%, 22.8%), 24.5% (3.8%, 72.0%), and 45.6% (4.0%, 92.3%), and the mean (95% PI) proportion of fish mortalities in ponds affected by the disease was 9.8% (1.4%, 26.7%), 49.2% (4.7%, 60.7%), and 88.3% (85.9%, 90.5%) for the low, medium, and high impact scenarios, respectively. The farm‐level mortality losses from an ED were up to 40.3% of the total farm inventory and can be used for insurance premium rate development. Disease spread modeling provides a systematic way to organize the current knowledge on the ED perils and, ultimately, use this information to help develop actuarially sound agricultural insurance policies and premiums. However, the estimates obtained will include a large amount of uncertainty driven by the stochastic nature of disease outbreaks, by the uncertainty in the frequency of future ED occurrences, and by the often sparse data available from past outbreaks.     

Balancing Inventory and Stockout Risk in Retail Supply Chains Using Fast‐Ship

In this study, we consider a two‐retailer, one‐supplier supply chain in which retailers satisfy excess demand by offering to directly ship out‐of‐stock items on an expedited basis at no extra cost to customers. This practice is referred to as the fast‐ship option. We consider two scenarios along with the fast‐ship option. In the first scenario, retailers transship when out of stock, whereas in the second scenario, they do not. If they do not transship, some customers may perform the search on their own. In each scenario, the wholesale prices are either exogenous, or chosen endogenously by the supplier. For both cases, we determine the supplier's and the retailers’ optimal decisions. The key research question we ask and answer is the following: which of the two scenarios is preferred by either player when all decisions are made optimally? We show that when fewer customers are willing to search on their own and wholesale prices are exogenous, both the supplier and the retailers prefer to transship. In addition, the decision maker in a centralized chain will have the exact same preference as that of players in a decentralized setting when the retailers’ and the supplier's preferences coincide and wholesale prices are exogenous. This preference concordance does not hold if wholesale prices are endogenous.     

Quantitative Assessment of the Risk of Release of Foot‐and‐Mouth Disease Virus via Export of Bull Semen from Israel

Various foot‐and‐mouth disease (FMD) virus strains circulate in the Middle East, causing frequent episodes of FMD outbreaks among Israeli livestock. Since the virus is highly resistant in semen, artificial insemination with contaminated bull semen may lead to the infection of the receiver cow. As a non‐FMD‐free country with vaccination, Israel is currently engaged in trading bull semen only with countries of the same status. The purpose of this study was to assess the risk of release of FMD virus through export of bull semen in order to estimate the risk for FMD‐free countries considering purchasing Israeli bull semen. A stochastic risk assessment model was used to estimate this risk, defined as the annual likelihood of exporting at least one ejaculate of bull semen contaminated with viable FMD virus. A total of 45 scenarios were assessed to account for uncertainty and variability around specific parameter estimates and to evaluate the effect of various mitigation measures, such as performing a preexport test on semen ejaculates. Under the most plausible scenario, the annual likelihood of exporting bull semen contaminated with FMD virus had a median of 1.3 * 10^?7 for an export of 100 ejaculates per year. This corresponds to one infected ejaculate exported every 7 million years. Under the worst‐case scenario, the median of the risk rose to 7.9 * 10^?5, which is equivalent to the export of one infected ejaculate every 12,000 years. Sensitivity analysis indicated that the most influential parameter is the probability of viral excretion in infected bulls.     

How Solid Is the Dutch (and the British) National Risk Assessment? Overview and Decision‐Theoretic Evaluation

Internationally, national risk assessment (NRA) is rapidly gaining government sympathy as a science‐based approach toward prioritizing the management of national hazards and threats, with the Netherlands and the United Kingdom in leading positions since 2007. NRAs are proliferating in Europe; they are also conducted in Australia, Canada, New Zealand, and the United States, while regional RAs now exist for over 100 Dutch or British provinces or counties. Focused on the Dutch NRA (DNRA) and supported by specific examples, summaries and evaluations are given of its (1) scenario development, (2) impact assessment, (3) likelihood estimation, (4) risk diagram, and (5) capability analysis. Despite the DNRA's thorough elaboration, apparent weaknesses are lack of stakeholder involvement, possibility of false‐positive risk scenarios, rigid multicriteria impact evaluation, hybrid methods for likelihood estimation, half‐hearted use of a “probability × effect” definition of risk, forced comparison of divergent risk scenarios, and unclear decision rules for risk acceptance and safety enhancement. Such weaknesses are not unique for the DNRA. In line with a somewhat reserved encouragement by the OECD (Studies in Risk Management. Innovation in Country Risk Management. Paris: OECD, 2009), the scientific solidity of NRA results so far is questioned, and several improvements are suggested. One critical point is that expert‐driven NRAs may preempt political judgments and decisions by national security authorities. External review and validation of major NRA components is recommended for strengthening overall results as a reliable basis for national and/or regional safety policies. Meanwhile, a broader, more transactional concept of risk may lead to better national and regional risk assessments.     

The Practice of Scenario Planning: An Analysis of Inter‐ and Intra‐organizational Strategizing

Strategic activity is often punctuated through the application of strategy tools. Despite widespread use, a lack of understanding exists regarding the impact such tools and their practices have on an organization's strategy process. Of the growing body of research tackling the phenomenon, none appears to extend beyond an intra‐organizational setting. Acknowledging the importance of multi‐organizational partnerships, particularly in the public sector, in this paper an attempt is made to help fill this void through examining the application and effect of a scenario planning process at an inter‐organizational level. Conceptualizing scenario planning as a practice of simplexity, where complexity of thought combines with simplicity of action, an in‐depth, longitudinal case study is used to demonstrate the importance and interaction of sensemaking, storytelling and organizing in creating meaning within strategizing activities at the inter‐organizational level. However, also demonstrated is the relative weakness of the output of the scenario planning process ? the stories ? as a boundary object capable of transferring knowledge and meaning to the intra‐organizational level. Through empirical and theoretical integration a model is developed presenting the flow of practices and artefacts used in sensemaking within inter‐ and intra‐organizational strategizing.     

Risk Assessment of Listeriosis Linked to the Consumption of Two Soft Cheeses Made from Raw Milk: Camembert of Normandy and Brie of Meaux

This article reports a quantitative risk assessment of human listeriosis linked to the consumption of soft cheeses made from raw milk. Risk assessment was based on data purposefully acquired inclusively over the period 2000-2001 for two French cheeses, namely: Camembert of Normandy and Brie of Meaux. Estimated Listeria monocytogenes concentration in raw milk was on average 0.8 and 0.3 cells/L, respectively, in Normandy and Brie regions. A Monte Carlo simulation was used to account for the time-temperature history of the milk and cheeses from farm to table. It was assumed that cell progeny did not spread within the solid cheese matrix (as they would be free to do in liquid broth). Interaction between pH and temperature was accounted for in the growth model. The simulated proportion of servings with no L. monocytogenes cell was 88% for Brie and 82% for Camembert. The 99th percentile of L. monocytogenes cell numbers in servings of 27 g of cheese was 131 for Brie and 77 for Camembert at the time of consumption, corresponding respectively to three and five cells of L. monocytogenes per gram. The expected number of severe listeriosis cases would be < or =10(-3) and < or =2.5 x 10(-3) per year for 17 million servings of Brie of Meaux and 480 million servings of Camembert of Normandy, respectively.     

Context‐Specific,Scenario‐Based Risk Scales

Reacting to an emergency requires quick decisions under stressful and dynamic conditions. To react effectively, responders need to know the right actions to take given the risks posed by the emergency. While existing research on risk scales focuses primarily on decision making in static environments with known risks, these scales may be inappropriate for conditions where the decision maker's time and mental resources are limited and may be infeasible if the actual risk probabilities are unknown. In this article, we propose a method to develop context‐specific, scenario‐based risk scales designed for emergency response training. Emergency scenarios are used as scale points, reducing our dependence on known probabilities; these are drawn from the targeted emergency context, reducing the mental resources required to interpret the scale. The scale is developed by asking trainers/trainees to rank order a range of risk scenarios and then aggregating these orderings using a Kemeny ranking. We propose measures to assess this aggregated scale's internal consistency, reliability, and validity, and we discuss how to use the scale effectively. We demonstrate our process by developing a risk scale for subsurface coal mine emergencies and test the reliability of the scale by repeating the process, with some methodological variations, several months later.