Probabilistic sensitivity analysis of complex models: a Bayesian approach

Summary.  In many areas of science and technology, mathematical models are built to simulate complex real world phenomena. Such models are typically implemented in large computer programs and are also very complex, such that the way that the model responds to changes in its inputs is not transparent. Sensitivity analysis is concerned with understanding how changes in the model inputs influence the outputs. This may be motivated simply by a wish to understand the implications of a complex model but often arises because there is uncertainty about the true values of the inputs that should be used for a particular application. A broad range of measures have been advocated in the literature to quantify and describe the sensitivity of a model's output to variation in its inputs. In practice the most commonly used measures are those that are based on formulating uncertainty in the model inputs by a joint probability distribution and then analysing the induced uncertainty in outputs, an approach which is known as probabilistic sensitivity analysis. We present a Bayesian framework which unifies the various tools of prob- abilistic sensitivity analysis. The Bayesian approach is computationally highly efficient. It allows effective sensitivity analysis to be achieved by using far smaller numbers of model runs than standard Monte Carlo methods. Furthermore, all measures of interest may be computed from a single set of runs.     

Inverse regression for ridge recovery: a data-driven approach for parameter reduction in computer experiments

Parameter reduction can enable otherwise infeasible design and uncertainty studies with modern computational science models that contain several input parameters. In statistical regression, techniques for sufficient dimension reduction (SDR) use data to reduce the predictor dimension of a regression problem. A computational scientist hoping to use SDR for parameter reduction encounters a problem: a computer prediction is best represented by a deterministic function of the inputs, so data comprised of computer simulation queries fail to satisfy the SDR assumptions. To address this problem, we interpret SDR methods sliced inverse regression (SIR) and sliced average variance estimation (SAVE) as estimating the directions of a ridge function, which is a composition of a low-dimensional linear transformation with a nonlinear function. Within this interpretation, SIR and SAVE estimate matrices of integrals whose column spaces are contained in the ridge directions’ span; we analyze and numerically verify convergence of these column spaces as the number of computer model queries increases. Moreover, we show example functions that are not ridge functions but whose inverse conditional moment matrices are low-rank. Consequently, the computational scientist should beware when using SIR and SAVE for parameter reduction, since SIR and SAVE may mistakenly suggest that truly important directions are unimportant.

     

Sensitivity,uncertainty, and decision analyses in the prioritization of research

The evaluation of hazards from complex, large scale, technologically advanced systems often requires the construction of computer implemented mathematical models. These models are used to evaluate the safety of the systems and to evaluate the consequences of modifications to the systems. These evaluations, however, are normally surrounded by significant uncertainties related to the uncertainty inherent in natural phenomena such as the weather and those related to uncertainties in the parameters and models used in the evaluation.

Another use of these models is to evaluate strategies for improving information used in the modeling process itself. While sensitivity analysis is useful in defining variables in the model that are important, uncertainty analysis provides a tool for assessing the importance of uncertainty about these variables. A third complementary technique, is decision analysis. It provides a methodology for explicitly evaluating and ranking potential improvements to the model. Its use in the development of information gathering strategies for a nuclear waste repository are discussed in this paper.     

An efficient computational approach for prior sensitivity analysis and cross‐validation

Prior sensitivity analysis and cross‐validation are important tools in Bayesian statistics. However, due to the computational expense of implementing existing methods, these techniques are rarely used. In this paper, the authors show how it is possible to use sequential Monte Carlo methods to create an efficient and automated algorithm to perform these tasks. They apply the algorithm to the computation of regularization path plots and to assess the sensitivity of the tuning parameter in g‐prior model selection. They then demonstrate the algorithm in a cross‐validation context and use it to select the shrinkage parameter in Bayesian regression. The Canadian Journal of Statistics 38:47–64; 2010 © 2010 Statistical Society of Canada     

Estimating stellar-oscillation-related parameters and their uncertainties with the moment method

Summary.  The moment method is a well-known astronomical mode identification technique in asteroseismology which uses a time series of the first three moments of a spectral line to estimate the discrete oscillation mode parameters l and m . The method, in contrast with many other mode identification techniques, also provides estimates of other important continuous parameters such as the inclination angle α and the rotational velocity v _e. We developed a statistical formalism for the moment method based on so-called generalized estimating equations. This formalism allows an estimation of the uncertainty of the continuous parameters, taking into account that the different moments of a line profile are correlated and that the uncertainty of the observed moments also depends on the model parameters. Furthermore, we set up a procedure to take into account the mode uncertainty, i.e. the fact that often several modes ( l ,  m ) can adequately describe the data. We also introduce a new lack-of-fit function which works at least as well as a previous discriminant function, and which in addition allows us to identify the sign of the azimuthal order m . We applied our method to star HD181558 by using several numerical methods, from which we learned that numerically solving the estimating equations is an intensive task. We report on the numerical results, from which we gain insight in the statistical uncertainties of the physical parameters that are involved in the moment method.     

ANOVA decomposition of conditional Gaussian processes for sensitivity analysis with dependent inputs

Complex computer codes are widely used in science to model physical systems. Sensitivity analysis aims to measure the contributions of the inputs on the code output variability. An efficient tool to perform such analysis is the variance-based methods which have been recently investigated in the framework of dependent inputs. One of their issue is that they require a large number of runs for the complex simulators. To handle it, a Gaussian process (GP) regression model may be used to approximate the complex code. In this work, we propose to decompose a GP into a high-dimensional representation. This leads to the definition of a variance-based sensitivity measure well tailored for non-independent inputs. We give a methodology to estimate these indices and to quantify their uncertainty. Finally, the approach is illustrated on toy functions and on a river flood model.     

Quantifying uncertainty in the biospheric carbon flux for England and Wales

Summary.  A crucial issue in the current global warming debate is the effect of vegetation and soils on carbon dioxide (CO₂) concentrations in the atmosphere. Vegetation can extract CO₂ through photosynthesis, but respiration, decay of soil organic matter and disturbance effects such as fire return it to the atmosphere. The balance of these processes is the net carbon flux. To estimate the biospheric carbon flux for England and Wales, we address the statistical problem of inference for the sum of multiple outputs from a complex deterministic computer code whose input parameters are uncertain. The code is a process model which simulates the carbon dynamics of vegetation and soils, including the amount of carbon that is stored as a result of photosynthesis and the amount that is returned to the atmosphere through respiration. The aggregation of outputs corresponding to multiple sites and types of vegetation in a region gives an estimate of the total carbon flux for that region over a period of time. Expert prior opinions are elicited for marginal uncertainty about the relevant input parameters and for correlations of inputs between sites. A Gaussian process model is used to build emulators of the multiple code outputs and Bayesian uncertainty analysis is then used to propagate uncertainty in the input parameters through to uncertainty on the aggregated output. Numerical results are presented for England and Wales in the year 2000. It is estimated that vegetation and soils in England and Wales constituted a net sink of 7.55 Mt C (1 Mt C = 10¹² g of carbon) in 2000, with standard deviation 0.56 Mt C resulting from the sources of uncertainty that are considered.     

Parameter estimation from interval-valued data using the expectation-maximization algorithm

This paper investigates on the problem of parameter estimation in statistical model when observations are intervals assumed to be related to underlying crisp realizations of a random sample. The proposed approach relies on the extension of likelihood function in interval setting. A maximum likelihood estimate of the parameter of interest may then be defined as a crisp value maximizing the generalized likelihood function. Using the expectation-maximization (EM) to solve such maximizing problem therefore derives the so-called interval-valued EM algorithm (IEM), which makes it possible to solve a wide range of statistical problems involving interval-valued data. To show the performance of IEM, the following two classical problems are illustrated: univariate normal mean and variance estimation from interval-valued samples, and multiple linear/nonlinear regression with crisp inputs and interval output.     

An economic statistical design of the Poisson EWMA control charts for monitoring nonconformities

An economic statistical model of the exponentially weighted moving average (EWMA) control chart for the average number of nonconformities in the sample is proposed. The statistical and economic performance of proposed design are evaluated using the average run length (ARL) and the hourly expected cost, respectively. A Markov chain approach is applied to derive expressions for ARL. The cost model is established based on the general cost function given in Lorenzen and Vance [The economic design of control charts: a unified approach. Technometrics. 1986;28:3–11]. An example is provided to illustrate the application of the proposed model. A sensitivity analysis is also carried out to investigate the effects of model parameters on the solution of the economic statistical design by using the design of experiments (DOE) technique.     

货币、金融深化与经济增长的效率——基于中国的实证

虚拟经济与实体经济的关系历来是理论关注的焦点之一 ,自从维克塞尔 (Ｊ .Ｇ .Ｋ .Ｗｉｃｋｓｅｌｌ)提出货币与经济的一体化理论之后[1] ,以前的将价值理论与货币理论相分离的“两分法”的经济分析方法被打破。随后 ,尤其是 2 0世纪 6 0年代以后 ,雷蒙德·Ｗ·戈德史密斯 (ＲａｙｍｏｎｄＷ .Ｇｏｌｄｓｍｉｔｈ)和爱德华·Ｓ·肖(ＥｄｗａｒｄＳ .Ｓｈａｗ)以及罗纳德·麦金农 (ＲｏｎａｌｄＭｃｋｉｎ ｎｏｎ)等人从实证和理论上对金融在经济发展中作用进行了深入地研究。改革开放后 ,随着我国金融中介机构的增多和金融流量的急剧扩大 ,…     

Expert Knowledge Elicitation: Subjective but Scientific

ABSTRACT

Expert opinion and judgment enter into the practice of statistical inference and decision-making in numerous ways. Indeed, there is essentially no aspect of scientific investigation in which judgment is not required. Judgment is necessarily subjective, but should be made as carefully, as objectively, and as scientifically as possible.

Elicitation of expert knowledge concerning an uncertain quantity expresses that knowledge in the form of a (subjective) probability distribution for the quantity. Such distributions play an important role in statistical inference (for example as prior distributions in a Bayesian analysis) and in evidence-based decision-making (for example as expressions of uncertainty regarding inputs to a decision model). This article sets out a number of practices through which elicitation can be made as rigorous and scientific as possible.

One such practice is to follow a recognized protocol that is designed to address and minimize the cognitive biases that experts are prone to when making probabilistic judgments. We review the leading protocols in the field, and contrast their different approaches to dealing with these biases through the medium of a detailed case study employing the SHELF protocol.

The article ends with discussion of how to elicit a joint probability distribution for multiple uncertain quantities, which is a challenge for all the leading protocols. Supplementary materials for this article are available online.     

RISK PREFERENCES UNDER PRICE UNCERTAINTIES AND PRODUCTION RISK

This paper deals with estimation of risk preferences of producers when they face uncertainties in output and input prices, in addition to uncertainty in production (usually labeled as production risk). All these uncertainty components are modeled in the context of production theory where the producers maximize expected utility of anticipated profit. Risk preference functions associated with these uncertainties are derived without assuming a specific form of the utility function. Moreover, no distributional assumptions are made on the distributions of the random variables representing price and production uncertainties. A multi-stage estimation procedure is developed to estimate the parameters of the production function and risk preference functions associated with output price uncertainty, input price uncertainty and production risk. Production risk is specified in such a way that one can identify inputs with increasing, decreasing and constant production risks. Similarly, risk aversion behavior is specified in such a way that one can test for different types of risk aversion behavior.     

Statistical Modeling for the Evaluation of Consumers’ Preferences

In 2009 a survey was performed in Veneto, a region in the north-east of Italy, to study the demand for wine and specifically for Passito, a typical Italian wine. The main goal of the study consisted in analyzing how the preferences and consumption habits of Passito vary depending on consumers’ characteristics. Specifically two kinds of statistical methods were applied: Covariate Uniform Binomial (CUB) model, a statistical approach for ordinal data to study the feeling toward Passito and the uncertainty of the respondents; classical logistic regression analysis, to describe how the attitude toward passito can be modeled as function of consumers’ covariates. Gender and residence were the most important covariates, useful in defining segments of consumers with significant differences in terms of Passito's preferences and consumption behavior. The logistic regression analysis allowed to complete the statistical analysis based on CUB models validating the results of the CUB model and estimating a model useful to predict the attitude toward the considered product for specific sub-groups of consumers.     

Reliability of global sensitivity indices

Uncertainty and sensitivity analysis is an essential ingredient of model development and applications. For many uncertainty and sensitivity analysis techniques, sensitivity indices are calculated based on a relatively large sample to measure the importance of parameters in their contributions to uncertainties in model outputs. To statistically compare their importance, it is necessary that uncertainty and sensitivity analysis techniques provide standard errors of estimated sensitivity indices. In this paper, a delta method is used to analytically approximate standard errors of estimated sensitivity indices for a popular sensitivity analysis method, the Fourier amplitude sensitivity test (FAST). Standard errors estimated based on the delta method were compared with those estimated based on 20 sample replicates. We found that the delta method can provide a good approximation for the standard errors of both first-order and higher-order sensitivity indices. Finally, based on the standard error approximation, we also proposed a method to determine a minimum sample size to achieve the desired estimation precision for a specified sensitivity index. The standard error estimation method presented in this paper can make the FAST analysis computationally much more efficient for complex models.     

Maximum-Entropy Prior Uncertainty and Correlation of Statistical Economic Data

Empirical estimates of source statistical economic data such as trade flows, greenhouse gas emissions, or employment figures are always subject to uncertainty (stemming from measurement errors or confidentiality) but information concerning that uncertainty is often missing. This article uses concepts from Bayesian inference and the maximum entropy principle to estimate the prior probability distribution, uncertainty, and correlations of source data when such information is not explicitly provided. In the absence of additional information, an isolated datum is described by a truncated Gaussian distribution, and if an uncertainty estimate is missing, its prior equals the best guess. When the sum of a set of disaggregate data is constrained to match an aggregate datum, it is possible to determine the prior correlations among disaggregate data. If aggregate uncertainty is missing, all prior correlations are positive. If aggregate uncertainty is available, prior correlations can be either all positive, all negative, or a mix of both. An empirical example is presented, which reports relative uncertainties and correlation priors for the County Business Patterns database. In this example, relative uncertainties range from 1% to 80% and 20% of data pairs exhibit correlations below ?0.9 or above 0.9. Supplementary materials for this article are available online.     

Theoretic and empirical data-inclusive process characterization

Summary.  In process characterization the quality of information that is obtained depends directly on the quality of process model. The current quality revolution is now providing a strong stimulus for rethinking and re-evaluating many statistical ideas. Among these are the role of theoretic knowledge and data in statistical inference and some issues in theoretic–empirical modelling. With this concern the paper takes a broad, pragmatic view of statistical inference to include all aspects of model formulation. The estimation of model parameters traditionally assumes that a model has a prespecified known form and takes no account of possible uncertainty regarding model structure. But in practice model structural uncertainty is a fact of life and is likely to be more serious than other sources of uncertainty which have received far more attention. This is true whether the model is specified on subject-matter grounds or when a model is formulated, fitted and checked on the same data set in an iterative interactive way. For that reason novel modelling techniques have been fashioned for reducing model uncertainty. Using available knowledge for theoretic model elaboration the techniques that have been created approximate the exact unknown process model concurrently by accessible theoretic and polynomial empirical functions. The paper examines the effects of uncertainty for hybrid theoretic–empirical models and, for reducing uncertainty, additive and multiplicative methods of model formulation are fashioned. Such modelling techniques have been successfully applied to perfect a steady flow model for an air gauge sensor. Validation of the models elaborated has revealed that the multiplicative modelling approach allows us to attain a satisfactory model with small discrepancy from empirical evidence.     

An Interactive Forecasting System

Time sharing computer configurations have introduced a new dimension in applying statistical and mathematical models to sequential decision problems. When the outcome of one step in the process influences subsequent decisions, then an interactive time-sharing system is of great help. Since the forecasting function involves such a sequential process, it can be handled particularly well with an appropriate time-shared computer system. This paper describes such as system which allows the user to do preliminary analysis of his data to identify the forecasting technique or class of techniques most appropriate for his situation and to apply those in developing a forecast. This interactive forecasting system has met with excellent success both in teaching the fundamentals of forecasting for business decision making and in actually applying those techniques in management situations.     

Machine learning and design of experiments with an application to product innovation in the chemical industry

Industrial statistics plays a major role in the areas of both quality management and innovation. However, existing methodologies must be integrated with the latest tools from the field of Artificial Intelligence. To this end, a background on the joint application of Design of Experiments (DOE) and Machine Learning (ML) methodologies in industrial settings is presented here, along with a case study from the chemical industry. A DOE study is used to collect data, and two ML models are applied to predict responses which performance show an advantage over the traditional modeling approach. Emphasis is placed on causal investigation and quantification of prediction uncertainty, as these are crucial for an assessment of the goodness and robustness of the models developed. Within the scope of the case study, the models learned can be implemented in a semi-automatic system that can assist practitioners who are inexperienced in data analysis in the process of new product development.