Sensitivity Analysis for Importance Assessment

We review briefly some examples that would support an extended role for quantitative sensitivity analysis in the context of model-based analysis (Section 1). We then review what features a quantitative sensitivity analysis needs to have to play such a role (Section 2). The methods that meet these requirements are described in Section 3; an example is provided in Section 4. Some pointers to further research are set out in Section 5.     

A Common Rationale for Global Sensitivity Measures and Their Estimation

Measures of sensitivity and uncertainty have become an integral part of risk analysis. Many such measures have a conditional probabilistic structure, for which a straightforward Monte Carlo estimation procedure has a double‐loop form. Recently, a more efficient single‐loop procedure has been introduced, and consistency of this procedure has been demonstrated separately for particular measures, such as those based on variance, density, and information value. In this work, we give a unified proof of single‐loop consistency that applies to any measure satisfying a common rationale. This proof is not only more general but invokes less restrictive assumptions than heretofore in the literature, allowing for the presence of correlations among model inputs and of categorical variables. We examine numerical convergence of such an estimator under a variety of sensitivity measures. We also examine its application to a published medical case study.     

Comparative Human Health Impact Assessment of Engineered Nanomaterials in the Framework of Life Cycle Assessment

For safe innovation, knowledge on potential human health impacts is essential. Ideally, these impacts are considered within a larger life‐cycle‐based context to support sustainable development of new applications and products. A methodological framework that accounts for human health impacts caused by inhalation of engineered nanomaterials (ENMs) in an indoor air environment has been previously developed. The objectives of this study are as follows: (i) evaluate the feasibility of applying the CF framework for NP exposure in the workplace based on currently available data; and (ii) supplement any resulting knowledge gaps with methods and data from the li fe c ycle a pproach and human r isk a ssessment (LICARA) project to develop a modified case‐specific version of the framework that will enable near‐term inclusion of NP human health impacts in life cycle assessment (LCA) using a case study involving nanoscale titanium dioxide (nanoTiO₂). The intent is to enhance typical LCA with elements of regulatory risk assessment, including its more detailed measure of uncertainty. The proof‐of‐principle demonstration of the framework highlighted the lack of available data for both the workplace emissions and human health effects of ENMs that is needed to calculate generalizable characterization factors using common human health impact assessment practices in LCA. The alternative approach of using intake fractions derived from workplace air concentration measurements and effect factors based on best‐available toxicity data supported the current case‐by‐case approach for assessing the human health life cycle impacts of ENMs. Ultimately, the proposed framework and calculations demonstrate the potential utility of integrating elements of risk assessment with LCA for ENMs once the data are available.     

Measuring Uncertainty Importance: Investigation and Comparison of Alternative Approaches

Uncertainty importance measures are quantitative tools aiming at identifying the contribution of uncertain inputs to output uncertainty. Their application ranges from food safety (Frey & Patil (2002)) to hurricane losses (Iman et al. (2005a, 2005b)). Results and indications an analyst derives depend on the method selected for the study. In this work, we investigate the assumptions at the basis of various indicator families to discuss the information they convey to the analyst/decisionmaker. We start with nonparametric techniques, and then present variance-based methods. By means of an example we show that output variance does not always reflect a decisionmaker state of knowledge of the inputs. We then examine the use of moment-independent approaches to global sensitivity analysis, i.e., techniques that look at the entire output distribution without a specific reference to its moments. Numerical results demonstrate that both moment-independent and variance-based indicators agree in identifying noninfluential parameters. However, differences in the ranking of the most relevant factors show that inputs that influence variance the most are not necessarily the ones that influence the output uncertainty distribution the most.     

Sensitivity Analysis of a Two‐Dimensional Quantitative Microbiological Risk Assessment: Keeping Variability and Uncertainty Separated

The aim of quantitative microbiological risk assessment is to estimate the risk of illness caused by the presence of a pathogen in a food type, and to study the impact of interventions. Because of inherent variability and uncertainty, risk assessments are generally conducted stochastically, and if possible it is advised to characterize variability separately from uncertainty. Sensitivity analysis allows to indicate to which of the input variables the outcome of a quantitative microbiological risk assessment is most sensitive. Although a number of methods exist to apply sensitivity analysis to a risk assessment with probabilistic input variables (such as contamination, storage temperature, storage duration, etc.), it is challenging to perform sensitivity analysis in the case where a risk assessment includes a separate characterization of variability and uncertainty of input variables. A procedure is proposed that focuses on the relation between risk estimates obtained by Monte Carlo simulation and the location of pseudo‐randomly sampled input variables within the uncertainty and variability distributions. Within this procedure, two methods are used—that is, an ANOVA‐like model and Sobol sensitivity indices—to obtain and compare the impact of variability and of uncertainty of all input variables, and of model uncertainty and scenario uncertainty. As a case study, this methodology is applied to a risk assessment to estimate the risk of contracting listeriosis due to consumption of deli meats.     

Global Sensitivity Analysis Applied to a Contamination Assessment Model of Listeria monocytogenes in Cold Smoked Salmon at Consumption

In this study, a variance‐based global sensitivity analysis method was first applied to a contamination assessment model of Listeria monocytogenes in cold smoked vacuum packed salmon at consumption. The impact of the choice of the modeling approach (populational or cellular) of the primary and secondary models as well as the effect of their associated input factors on the final contamination level was investigated. Results provided a subset of important factors, including the food water activity, its storage temperature, and duration in the domestic refrigerator. A refined sensitivity analysis was then performed to rank the important factors, tested over narrower ranges of variation corresponding to their current distributions, using three techniques: ANOVA, Spearman correlation coefficient, and partial least squares regression. Finally, the refined sensitivity analysis was used to rank the important factors.     

A Matrix-Based Approach to Uncertainty and Sensitivity Analysis for Fault Trees1

System unavailabilities for large complex systems such as nuclear power plants are often evaluated through use of fault tree analysis. The system unavailability is obtained from a Boolean representation of a system fault tree. Even after truncation of higher order terms these expressions can be quite large, involving thousands of terms. A general matrix notation is proposed for the representation of Boolean expressions which facilitates uncertainty and sensitivity analysis calculations.     

A Methodology for Determining Interactions in Probabilistic Safety Assessment Models by Varying One Parameter at a Time

In risk analysis problems, the decision‐making process is supported by the utilization of quantitative models. Assessing the relevance of interactions is an essential information in the interpretation of model results. By such knowledge, analysts and decisionmakers are able to understand whether risk is apportioned by individual factor contributions or by their joint action. However, models are oftentimes large, requiring a high number of input parameters, and complex, with individual model runs being time consuming. Computational complexity leads analysts to utilize one‐parameter‐at‐a‐time sensitivity methods, which prevent one from assessing interactions. In this work, we illustrate a methodology to quantify interactions in probabilistic safety assessment (PSA) models by varying one parameter at a time. The method is based on a property of the functional ANOVA decomposition of a finite change that allows to exactly determine the relevance of factors when considered individually or together with their interactions with all other factors. A set of test cases illustrates the technique. We apply the methodology to the analysis of the core damage frequency of the large loss of coolant accident of a nuclear reactor. Numerical results reveal the nonadditive model structure, allow to quantify the relevance of interactions, and to identify the direction of change (increase or decrease in risk) implied by individual factor variations and by their cooperation.     

Uncertainty in Climate Change Modeling: Can Global Sensitivity Analysis Be of Help?

Integrated assessment models offer a crucial support to decisionmakers in climate policy making. For a full understanding and corroboration of model results, analysts ought to identify the exogenous variables that influence the model results the most (key drivers), appraise the relevance of interactions, and the direction of change associated with the simultaneous variation of uncertain variables. We show that such information can be directly extracted from the data set produced by Monte Carlo simulations. Our discussion is guided by the application to the well‐known DICE model of William Nordhaus. The proposed methodology allows analysts to draw robust insights into the dependence of future atmospheric temperature, global emissions, and carbon costs and taxes on the model's exogenous variables.     

Sensitivity Analysis of Model Output with Input Constraints: A Generalized Rationale for Local Methods

In this work, we introduce a generalized rationale for local sensitivity analysis (SA) methods that allows to solve the problems connected with input constraints. Several models in use in the risk analysis field are characterized by the presence of deterministic relationships among the input parameters. However, SA issues related to the presence of constraints have been mainly dealt with in a heuristic fashion. We start with a systematic analysis of the effects of constraints. The findings can be summarized in the following three effects. (i) Constraints make it impossible to vary one parameter while keeping all others fixed. (ii) The model output becomes insensitive to a parameter if a constraint is solved for that parameter. (iii) Sensitivity analysis results depend on which parameter is selected as dependent. The explanation of these effects is found by proposing a result that leads to a natural extension of the local SA rationale introduced in Helton (1993) . We then extend the definitions of the Birnbaum, criticality, and the differential importance measures to the constrained case. In addition, a procedure is introduced that allows to obtain constrained sensitivity results at the same cost as in the absence of constraints. The application to a nonbinary event tree concludes the article, providing a numerical illustration of the above findings.     

Uncertainty and Sensitivity Analysis in Assessment of the Thyroid Cancer Risk Related to Chernobyl Fallout in Eastern France

The increase in the thyroid cancer incidence in France observed over the last 20 years has raised public concern about its association with the 1986 nuclear power plant accident at Chernobyl. At the request of French authorities, a first study sought to quantify the possible risk of thyroid cancer associated with the Chernobyl fallout in France. This study suffered from two limitations. The first involved the lack of knowledge of spontaneous thyroid cancer incidence rates (in the absence of exposure), which was especially necessary to take their trends into account for projections over time; the second was the failure to consider the uncertainties. The aim of this article is to enhance the initial thyroid cancer risk assessment for the period 1991-2007 in the area of France most exposed to the fallout (i.e., eastern France) and thereby mitigate these limitations. We consider the changes over time in the incidence of spontaneous thyroid cancer and conduct both uncertainty and sensitivity analyses. The number of spontaneous thyroid cancers was estimated from French cancer registries on the basis of two scenarios: one with a constant incidence, the other using the trend observed. Thyroid doses were estimated from all available data about contamination in France from Chernobyl fallout. Results from a 1995 pooled analysis published by Ron et al. were used to determine the dose-response relation. Depending on the scenario, the number of spontaneous thyroid cancer cases ranges from 894 (90% CI: 869-920) to 1,716 (90% CI: 1,691-1,741). The number of excess thyroid cancer cases predicted ranges from 5 (90% UI: 1-15) to 63 (90% UI: 12-180). All of the assumptions underlying the thyroid cancer risk assessment are discussed.     

Sensitivity Analysis Using Risk Measures

In a quantitative model with uncertain inputs, the uncertainty of the output can be summarized by a risk measure. We propose a sensitivity analysis method based on derivatives of the output risk measure, in the direction of model inputs. This produces a global sensitivity measure, explicitly linking sensitivity and uncertainty analyses. We focus on the case of distortion risk measures, defined as weighted averages of output percentiles, and prove a representation of the sensitivity measure that can be evaluated on a Monte Carlo sample, as a weighted average of gradients over the input space. When the analytical model is unknown or hard to work with, nonparametric techniques are used for gradient estimation. This process is demonstrated through the example of a nonlinear insurance loss model. Furthermore, the proposed framework is extended in order to measure sensitivity to constant model parameters, uncertain statistical parameters, and random factors driving dependence between model inputs.     

Sensitivity of Concentration and Risk Predictions in the PRESTO and MMSOILS Multimedia Models: Regression Technique Assessment

This paper describes the application of two multimedia models, PRESTO and MMSOILS, to predict contaminant migration from a landfill that contains an organic chemical (methylene chloride) and a radionuclide (uranium-238). Exposure point concentrations and human health risks are predicted, and distributions of those predictions are generated using Monte Carlo techniques. Analysis of exposure point concentrations shows that predictions of uranium-238 in groundwater differ by more than one order of magnitude between models. These differences occur mainly because PRESTO simulates uranium-238 transport through the groundwater using a one-dimensional algorithm and vertically mixes the plume over an effective mixing depth, whereas MMSOILS uses a three-dimensional algorithm and simulates a plume that resides near the surface of the aquifer.A sensitivity analysis, using stepwise multiple linear regression, is performed to evaluate which of the random variables are most important in producing the predicted distributions of exposure point concentrations and health risks. The sensitivity analysis shows that the predicted distributions can be accurately reproduced using a small subset of the random variables. Simple regression techniques are applied, for comparison, to the same scenarios, and results are similar. The practical implication of this analysis is the ability to distinguish between important versus unimportant random variables in terms of their sensitivity to selected endpoints.     

Sensitivity Analysis of a Two-Dimensional Probabilistic Risk Assessment Model Using Analysis of Variance

This article demonstrates application of sensitivity analysis to risk assessment models with two-dimensional probabilistic frameworks that distinguish between variability and uncertainty. A microbial food safety process risk (MFSPR) model is used as a test bed. The process of identifying key controllable inputs and key sources of uncertainty using sensitivity analysis is challenged by typical characteristics of MFSPR models such as nonlinearity, thresholds, interactions, and categorical inputs. Among many available sensitivity analysis methods, analysis of variance (ANOVA) is evaluated in comparison to commonly used methods based on correlation coefficients. In a two-dimensional risk model, the identification of key controllable inputs that can be prioritized with respect to risk management is confounded by uncertainty. However, as shown here, ANOVA provided robust insights regarding controllable inputs most likely to lead to effective risk reduction despite uncertainty. ANOVA appropriately selected the top six important inputs, while correlation-based methods provided misleading insights. Bootstrap simulation is used to quantify uncertainty in ranks of inputs due to sampling error. For the selected sample size, differences in F values of 60% or more were associated with clear differences in rank order between inputs. Sensitivity analysis results identified inputs related to the storage of ground beef servings at home as the most important. Risk management recommendations are suggested in the form of a consumer advisory for better handling and storage practices.     

Variance-Based Importance Analysis Applied to a Complex Probabilistic Performance Assessment

The most important input parameters in a complex probabilistic performance assessment are identified using a variance-based method and compared with those identified using a regression-based method. The variance-based method has the advantage of not requiring assumptions about the functional relationship between input and output parameters. However, it has the drawback of requiring heuristic assessments of threshold variance ratios above which a parameter is considered important, and it also requires numerous executions of the computer program, which may be computationally expensive. Both methods identified the same top 5 and 7 of the top 10 most important parameters for a system having 195 inputs. Although no distinct advantage for the variance-based approach was identified, the ideas which motivate the new approach are sound and suggest new avenues for exploring the relationships between the inputs and the output of a complex system.     

Methods for Uncertainty Analysis: A Comparative Survey

This paper presents a survey and comparative evaluation of methods which have been developed for the determination of uncertainties in accident consequences and probabilities, for use in probabilistic risk assessment. The methods considered are: analytic techniques, Monte Carlo simulation, response surface approaches, differential sensitivity techniques, and evaluation of classical statistical confidence bounds. It is concluded that only the response surface and differential sensitivity approaches are sufficiently general and flexible for use as overall methods of uncertainty analysis in probabilistic risk assessment. The other methods considered, however, are very useful in particular problems.     

A Framework for Understanding Uncertainty in Seismic Risk Assessment

A better understanding of the uncertainty that exists in models used for seismic risk assessment is critical to improving risk-based decisions pertaining to earthquake safety. Current models estimating the probability of collapse of a building do not consider comprehensively the nature and impact of uncertainty. This article presents a model framework to enhance seismic risk assessment and thus gives decisionmakers a fuller understanding of the nature and limitations of the estimates. This can help ensure that risks are not over- or underestimated and the value of acquiring accurate data is appreciated fully. The methodology presented provides a novel treatment of uncertainties in input variables, their propagation through the model, and their effect on the results. The study presents ranges of possible annual collapse probabilities for different case studies on buildings in different parts of the world, exposed to different levels of seismicity, and with different vulnerabilities. A global sensitivity analysis was conducted to determine the significance of uncertain variables. Two key outcomes are (1) that the uncertainty in ground-motion conversion equations has the largest effect on the uncertainty in the calculation of annual collapse probability; and (2) the vulnerability of a building appears to have an effect on the range of annual collapse probabilities produced, i.e., the level of uncertainty in the estimate of annual collapse probability, with less vulnerable buildings having a smaller uncertainty.     

Life cycle impact assessment: a challenge for risk analysts.

Modern technology, together with an advanced economy, can provide a good or service in myriad ways, giving us choices on what to produce and how to produce it. To make those choices more intelligently, society needs to know not only the market price of each alternative, but the associated health and environmental consequences. A fair comparison requires evaluating the consequences across the whole "life cycle"--from the extraction of raw materials and processing to manufacture/construction, use, and end-of-life--of each alternative. Focusing on only one stage (e.g., manufacture) of the life cycle is often misleading. Unfortunately, analysts and researchers still have only rudimentary tools to quantify the materials and energy inputs and the resulting damage to health and the environment. Life cycle assessment (LCA) provides an overall framework for identifying and evaluating these implications. Since the 1960s, considerable progress has been made in developing methods for LCA, especially in characterizing, qualitatively and quantitatively, environmental discharges. However, few of these analyses have attempted to assess the quantitative impact on the environment and health of material inputs and environmental discharges Risk analysis and LCA are connected closely. While risk analysis has characterized and quantified the health risks of exposure to a toxicant, the policy implications have not been clear. Inferring that an occupational or public health exposure carries a nontrivial risk is only the first step in formulating a policy response. A broader framework, including LCA, is needed to see which response is likely to lower the risk without creating high risks elsewhere. Even more important, LCA has floundered at the stage of translating an inventory of environmental discharges into estimates of impact on health and the environment. Without the impact analysis, policymakers must revert to some simple rule, such as that all discharges, regardless of which chemical, which medium, and where they are discharged, are equally toxic. Thus, risk analysts should seek LCA guidance in translating a risk analysis into policy conclusions or even advice to those at risk. LCA needs the help of RA to go beyond simplistic assumptions about the implications of a discharge inventory. We demonstrate the need and rationale for LCA, present a brief history of LCA, present examples of the application of this tool, note the limitations of LCA models, and present several methods for incorporating risk assessment into LCA. However, we warn the reader not to expect too much. A comprehensive comparison of the health and environmental implications of alternatives is beyond the state of the art. LCA is currently not able to provide risk analysts with detailed information on the chemical form and location of the environmental discharges that would allow detailed estimation of the risks to individuals due to toxicants. For example, a challenge for risk analysts is to estimate health and other risks where the location and chemical speciation are not characterized precisely. Providing valuable information to decisionmakers requires advances in both LCA and risk analysis. These two disciplines should be closely linked, since each has much to contribute to the other.     

Life-Cycle Assessment in Relation to Risk Assessment: An Evolving Perspective

A life-cycle approach takes a cradle-to-grave perspective of a product's numerous activities from the raw material extraction to final disposal. There have been recent efforts to develop life-cycle assessment (LCA) to assess both environmental and human health issues. The question then arises: what are the capabilities of LCA, especially in relation to risk assessment? To address this question, this paper first describes the LCA mass-based accounting system and then analyzes the use of this approach for environmental and human health assessment. The key LCA limitations in this respect are loss of spatial, temporal, dose-response, and threshold information. These limitations affect LCA's capability to assess several environmental issues, and human health in particular. This leads to the conclusion that LCA impact assessment does not predict or measure actual effects, quantitate risks, or address safety. Instead, LCA uses mass loadings with simplifying assumptions and subjective judgments to add independent effects and exposures into an overall score. As a result, LCA identifies possible human health issues on a systemwide basis from a worst case, hypothetical hazard perspective. Ideally, the identified issues would then be addressed by more detailed assessment methods, such as risk assessment.     

Sensitivity Analysis to Evaluate the Impact of Uncertain Factors in a Scenario Tree Model for Classical Swine Fever Introduction

Introduction of classical swine fever virus (CSFV) is a continuing threat to the pig production sector in the European Union. A scenario tree model was developed to obtain more insight into the main risk factors determining the probability of CSFV introduction (P(CSFV)). As this model contains many uncertain input parameters, sensitivity analysis was used to indicate which of these parameters influence model results most. Group screening combined with the statistical techniques of design of experiments and meta-modeling was applied to detect the most important uncertain input parameters among a total of 257 parameters. The response variable chosen was the annual P(CSFV) into the Netherlands. Only 128 scenario calculations were needed to specify the final meta-model. A consecutive one-at-a-time sensitivity analysis was performed with the main effects of this meta-model to explore their impact on the ranking of risk factors contributing most to the annual P(CSFV). The results indicated that model outcome is most sensitive to the uncertain input parameters concerning the expected number of classical swine fever epidemics in Germany, Belgium, and the United Kingdom and the probability that CSFV survives in an empty livestock truck traveling over a distance of 0-900 km.