Convergence properties of the expected improvement algorithm with fixed mean and covariance functions

This paper deals with the convergence of the expected improvement algorithm, a popular global optimization algorithm based on a Gaussian process model of the function to be optimized. The first result is that under some mild hypotheses on the covariance function k of the Gaussian process, the expected improvement algorithm produces a dense sequence of evaluation points in the search domain, when the function to be optimized is in the reproducing kernel Hilbert space generated by k  . The second result states that the density property also holds for P-almost$\mathsf{P}\text{-almost}$ all continuous functions, where P is the (prior) probability distribution induced by the Gaussian process.     

A Note On Weighted Distributions

A representation of the "original" random variable (r.v.) in terms of the "weighted" r.v. is given and the Inverse Gaussian distribution is characterized through a distributional property the "weighted" r.v. observed under 1ength biased sampling.     

Gamma-Generalized Inverse Gaussian Class of Distributions with Applications

In this article, a new family of probability distributions with domain in ?⁺ is introduced. This class can be considered as a natural extension of the exponential-inverse Gaussian distribution in Bhattacharya and Kumar (1986 Bhattacharya , S. K. , Kumar , S. ( 1986 ). E-IG model in life testing . Calcutta Statist. Assoc. Bull. 35 : 85 – 90 . [Google Scholar]) and Frangos and Karlis (2004 Frangos , N. , Karlis , D. ( 2004 ). Modelling losses using an exponential-inverse Gaussian distribution . Insur. Math. Econo. 35 : 53 – 67 .[Crossref], [Web of Science ®] , [Google Scholar]). This new family is obtained through the mixture of gamma distribution with generalized inverse Gaussian distribution. We also show some important features such as expressions of probability density function, moments, etc. Special attention is paid to the mixture with the inverse Gaussian distribution, as a particular case of the generalized inverse Gaussian distribution. From the exponential-inverse Gaussian distribution two one-parameter family of distributions are obtained to derive risk measures and credibility expressions. The versatility of this family has been proven in numerical examples.     

EM-based maximum likelihood parameter estimation for multivariate generalized hyperbolic distributions with fixed λ

Generalized Hyperbolic distribution (Barndorff-Nielsen 1977) is a variance-mean mixture of a normal distribution with the Generalized Inverse Gaussian distribution. Recently subclasses of these distributions (e.g., the hyperbolic distribution and the Normal Inverse Gaussian distribution) have been applied to construct stochastic processes in turbulence and particularly in finance, where multidimensional problems are of special interest. Parameter estimation for these distributions based on an i.i.d. sample is a difficult task even for a specified one-dimensional subclass (subclass being uniquely defined by ) and relies on numerical methods. For the hyperbolic subclass ( = 1), computer program hyp (Blæsild and Sørensen 1992) estimates parameters via ML when the dimensionality is less than or equal to three. To the best of the author's knowledge, no successful attempts have been made to fit any given subclass when the dimensionality is greater than three. This article proposes a simple EM-based (Dempster, Laird and Rubin 1977) ML estimation procedure to estimate parameters of the distribution when the subclass is known regardless of the dimensionality. Our method relies on the ability to numerically evaluate modified Bessel functions of the third kind and their logarithms, which is made possible by currently available software. The method is applied to fit the five dimensional Normal Inverse Gaussian distribution to a series of returns on foreign exchange rates.     

Approximations for weighted Kolmogorov–Smirnov distributions via boundary crossing probabilities

A statistical application to Gene Set Enrichment Analysis implies calculating the distribution of the maximum of a certain Gaussian process, which is a modification of the standard Brownian bridge. Using the transformation into a boundary crossing problem for the Brownian motion and a piecewise linear boundary, it is proved that the desired distribution can be approximated by an n-dimensional Gaussian integral. Fast approximations are defined and validated by Monte Carlo simulation. The performance of the method for the genomics application is discussed.     

Bandwidth selection in pre-smoothed particle filters

For the purpose of maximum likelihood estimation of static parameters, we apply a kernel smoother to the particles in the standard SIR filter for non-linear state space models with additive Gaussian observation noise. This reduces the Monte Carlo error in the estimates of both the posterior density of the states and the marginal density of the observation at each time point. We correct for variance inflation in the smoother, which together with the use of Gaussian kernels, results in a Gaussian (Kalman) update when the amount of smoothing turns to infinity. We propose and study of a criterion for choosing the optimal bandwidth h in the kernel smoother. Finally, we illustrate our approach using examples from econometrics. Our filter is shown to be highly suited for dynamic models with high signal-to-noise ratio, for which the SIR filter has problems.     

An extension of almost sure central limit theorem for the maximum of stationary Gaussian random fields

Let X₁, X₂, … be a sequence of stationary standardized Gaussian random fields. The almost sure limit theorem for the maxima of stationary Gaussian random fields is established. Our results extend and improve the results in Csáki and Gonchigdanzan (2002 Csáki, E., Gonchigdanzan, K. (2002). Almost sure limit theorems for the maximum of stationary Gaussian sequences. Stat. Probab. Lett. 58:195–203.[Crossref], [Web of Science ®] , [Google Scholar]) and Choi (2010 Choi, H. (2010). Almost sure limit theorem for stationary Gaussian random fields. J. Korean Stat. Soc. 39:449–454.[Crossref], [Web of Science ®] , [Google Scholar]).     

An elliptically symmetric angular Gaussian distribution

We define a distribution on the unit sphere \(\mathbb {S}^{d-1}\) called the elliptically symmetric angular Gaussian distribution. This distribution, which to our knowledge has not been studied before, is a subfamily of the angular Gaussian distribution closely analogous to the Kent subfamily of the general Fisher–Bingham distribution. Like the Kent distribution, it has ellipse-like contours, enabling modelling of rotational asymmetry about the mean direction, but it has the additional advantages of being simple and fast to simulate from, and having a density and hence likelihood that is easy and very quick to compute exactly. These advantages are especially beneficial for computationally intensive statistical methods, one example of which is a parametric bootstrap procedure for inference for the directional mean that we describe.     

Extremes Values of Discrete and Continuous Time Strongly Dependent Gaussian Processes

The joint limit distribution of the maximum of a continuous, strongly dependent stationary Gaussian process and the maximum of this process sampled at discrete time points is studied. It is shown that these two extreme values are asymptotically totally dependent if the grid of the discrete time points is sufficiently dense, and asymptotically dependent if the the grid points are sparse or Pickands grids. Our results are motivated by the deep contributions Piterbarg (2004 Piterbarg , V. I. ( 2004 ). Discrete and continuous time extremes of Gaussian processes . Extremes 7 : 161 – 177 .[Crossref] , [Google Scholar]) and Hüsler (2004 Hüsler , J. ( 2004 ). Dependence between extreme values of discrete and continuous time locally stationary Gaussian processes . Extremes 7 : 179 – 190 .[Crossref] , [Google Scholar]).     

Asymptotics of maxima and sums for a type of strongly dependent isotropic Gaussian random fields

For a type of strongly dependent isotropic Gaussian random fields introduced by Mittal (1976 Mittal, Y. 1976. A class of isotropic covariances functions. Pacific Journal of Mathematics 64:517–38.[Crossref], [Web of Science ®] , [Google Scholar]), the joint limiting distribution of the maximum and the sum for the Gaussian random fields is derived. The asymptotic relation between the maximum and sum of the continuous time strongly dependent isotropic Gaussian random fields and the maximum and sum of this fields sampled at discrete time points is also obtained.     

Testing for additivity and joint effects in multivariate nonparametric regression using Fourier and wavelet methods

We consider the problem of testing for additivity and joint effects in multivariate nonparametric regression when the data are modelled as observations of an unknown response function observed on a d-dimensional (d 2) lattice and contaminated with additive Gaussian noise. We propose tests for additivity and joint effects, appropriate for both homogeneous and inhomogeneous response functions, using the particular structure of the data expanded in tensor product Fourier or wavelet bases studied recently by Amato and Antoniadis (2001) and Amato, Antoniadis and De Feis (2002). The corresponding tests are constructed by applying the adaptive Neyman truncation and wavelet thresholding procedures of Fan (1996), for testing a high-dimensional Gaussian mean, to the resulting empirical Fourier and wavelet coefficients. As a consequence, asymptotic normality of the proposed test statistics under the null hypothesis and lower bounds of the corresponding powers under a specific alternative are derived. We use several simulated examples to illustrate the performance of the proposed tests, and we make comparisons with other tests available in the literature.     

ON THE NULL DISTRIBUTIONS OF THE ENTROPY TESTS FOR THE GAUSSIAN AND INVERSE GAUSSIAN MODELS

Vasicek [1] Vasicek, O. 1976. A test for normality based on sample entropy. J. R. Statist. Soc. B, 38: 54–59.  [Google Scholar]used the “convolution of twelve uniforms” for a Monte Carlo tabulation of the 5% critical values for his entropy test for normality. We employ a superior normal generator to construct a corrected and extended tabulation for his test. Interestingly, it is shown that, the same tables can be used for implementing Mudholkar and Tian's [2] Mudholkar, G. S. and Tian, L. 1999. “An entropy characterization of the inverse Gaussian distribution and related goodness-of-fit test”. In Tech. Rep., University of Rochester Rochester, NY Submitted for publication [Google Scholar]entropy test for the composite inverse Gaussian hypothesis. The finding extends the known Gaussian, inverse Gaussian analogies.     

Optimal network design for Bayesian spatial prediction of multivariate non-Gaussian environmental data

This paper deals with the problem of increasing air pollution monitoring stations in Tehran city for efficient spatial prediction. As the data are multivariate and skewed, we introduce two multivariate skew models through developing the univariate skew Gaussian random field proposed by Zareifard and Jafari Khaledi [21 H. Zareifard and M. Jafari Khaledi, Non-Gaussian modeling of spatial data using scale mixing of a unified skew Gaussian process, J. Multivariate Anal. 114 (2013), pp. 16–28. doi: 10.1016/j.jmva.2012.07.003[Crossref], [Web of Science ®] , [Google Scholar]]. These models provide extensions of the linear model of coregionalization for non-Gaussian data. In the Bayesian framework, the optimal network design is found based on the maximum entropy criterion. A Markov chain Monte Carlo algorithm is developed to implement posterior inference. Finally, the applicability of two proposed models is demonstrated by analyzing an air pollution data set.     

Inference on Overlap for Two Inverse Gaussian Populations: Equal Means Case

The inverse Gaussian distribution is often suited for modeling positive and/or positively skewed data (see Chhikara and Folks, 1989 Chhikara , R. S. , Folks , J. L. ( 1989 ). The Inverse Gaussian Distribution . New York : Marcel Dekker . [Google Scholar]) and presents an interesting alternative to the Gaussian model in such cases. We note here that overlap coefficients and their variants are widely studied in the literature for Gaussian populations (see Mulekar and Mishra, 1994 Mulekar , M. , Mishra , S. N. ( 1994 ). Overlap coefficients of two normal densities: equal means case . J. Japan. Statist. Soc. 24 : 169 – 180 . [Google Scholar], 2000 Mulekar , M. , Mishra , S. N. ( 2000 ). Confidence interval estimation of overlap: equal means case . Computat. Statist. Data Anal. 34 : 121 – 137 .[Crossref], [Web of Science ®] , [Google Scholar], and references therein for further details). This article studies the properties and addresses point estimation for large samples of commonly used measures of overlap when the populations are described by inverse Gaussian distributions. The bias and mean square error properties of the estimators are studied through a simulation study.     

Edgeworth Expansion for Linear Regression Processes with Long-Memory Errors

This article provides an Edgeworth expansion for the distribution of the log-likelihood derivative LLD of the parameter of a time series generated by a linear regression model with Gaussian, stationary, and long-memory errors. Under some sets of conditions on the regression coefficients, the spectral density function, and the parameter values, an Edgeworth expansion of the density as well as the distribution function of a vector of centered and normalized derivatives of the plug-in log-likelihood PLL function of arbitrarily large order is established. This is done by extending the results of Lieberman et al. (2003 Lieberman , O. , Rousseau , J. , Zucker , D. M. ( 2003 ). Valid edgeworth expansions for the maximum likelihood estimator of the parameter of a stationary. gaussian, strongly dependent processes. it Ann. Statist. 31:586–612 . [Google Scholar]), who provided an Edgeworth expansion for the Gaussian stationary long-memory case, to our present model, which is a linear regression process with stationary Gaussian long-memory errors.     

Random projections for Bayesian regression

This article deals with random projections applied as a data reduction technique for Bayesian regression analysis. We show sufficient conditions under which the entire d-dimensional distribution is approximately preserved under random projections by reducing the number of data points from n to \(k\in O({\text {poly}}(d/\varepsilon ))\) in the case \(n\gg d\). Under mild assumptions, we prove that evaluating a Gaussian likelihood function based on the projected data instead of the original data yields a \((1+O(\varepsilon ))\)-approximation in terms of the \(\ell _2\) Wasserstein distance. Our main result shows that the posterior distribution of Bayesian linear regression is approximated up to a small error depending on only an \(\varepsilon \)-fraction of its defining parameters. This holds when using arbitrary Gaussian priors or the degenerate case of uniform distributions over \(\mathbb {R}^d\) for \(\beta \). Our empirical evaluations involve different simulated settings of Bayesian linear regression. Our experiments underline that the proposed method is able to recover the regression model up to small error while considerably reducing the total running time.     

Independence Structure of Natural Conjugate Densities to Exponential Families and the Gibbs' Sampler

In this paper the independence between a block of natural parameters and the complementary block of mean value parameters holding for densities which are natural conjugate to some regular exponential families is used to design in a convenient way a Gibbs' sampler with block updates. Even when the densities of interest are obtained by conditioning to zero a block of natural parameters in a density conjugate to a larger "saturated" model, the updates require only the computation of marginal distributions under the "unconditional" density. For exponential families which are closed under marginalization, including both the zero mean Gaussian family and the cross-classified Bernoulli family such an implementation of the Gibbs' sampler can be seen as an Iterative Proportional Fitting algorithm with random inputs.     

Skew Gaussian Process for Nonlinear Regression

In this article, we extend the Gaussian process for regression model by assuming a skew Gaussian process prior on the input function and a skew Gaussian white noise on the error term. Under these assumptions, the predictive density of the output function at a new fixed input is obtained in a closed form. Also, we study the Gaussian process predictor when the errors depart from the Gaussianity to the skew Gaussian white noise. The bias is derived in a closed form and is studied for some special cases. We conduct a simulation study to compare the empirical distribution function of the Gaussian process predictor under Gaussian white noise and skew Gaussian white noise.     

Computation of Gaussian orthant probabilities in high dimension

We study the computation of Gaussian orthant probabilities, i.e. the probability that a Gaussian variable falls inside a quadrant. The Geweke–Hajivassiliou–Keane (GHK) algorithm (Geweke, Comput Sci Stat 23:571–578 1991, Keane, Simulation estimation for panel data models with limited dependent variables, 1993, Hajivassiliou, J Econom 72:85–134, 1996, Genz, J Comput Graph Stat 1:141–149, 1992) is currently used for integrals of dimension greater than 10. In this paper, we show that for Markovian covariances GHK can be interpreted as the estimator of the normalizing constant of a state-space model using sequential importance sampling. We show for an AR(1) the variance of the GHK, properly normalized, diverges exponentially fast with the dimension. As an improvement we propose using a particle filter. We then generalize this idea to arbitrary covariance matrices using Sequential Monte Carlo with properly tailored MCMC moves. We show empirically that this can lead to drastic improvements on currently used algorithms. We also extend the framework to orthants of mixture of Gaussians (Student, Cauchy, etc.), and to the simulation of truncated Gaussians.