Design and analysis of computer experiments when the output is highly correlated over the input space

To build a predictor, the output of a deterministic computer model or “code” is often treated as a realization of a stochastic process indexed by the code's input variables. The authors consider an asymptotic form of the Gaussian correlation function for the stochastic process where the correlation tends to unity. They show that the limiting best linear unbiased predictor involves Lagrange interpolating polynomials; linear model terms are implicitly included. The authors then develop optimal designs based on minimizing the limiting integrated mean squared error of prediction. They show through several examples that these designs lead to good prediction accuracy.     

Optimal estimating function for estimation and prediction in semi-parametric models

In a seminal paper, Godambe [1985. The foundations of finite sample estimation in stochastic processes. Biometrika 72, 419–428.] introduced the ‘estimating function’ approach to estimation of parameters in semi-parametric models under a filtering associated with a martingale structure. Later, Godambe [1987. The foundations of finite sample estimation in stochastic processes II. Bernoulli, Vol. 2. V.N.V. Science Press, 49–54.] and Godambe and Thompson [1989. An extension of quasi-likelihood Estimation. J. Statist. Plann. Inference 22, 137–172.] replaced this filtering by a more flexible conditioning. Abraham et al. [1997. On the prediction for some nonlinear time-series models using estimating functions. In: Basawa, I.V., et al. (Eds.), IMS Selected Proceedings of the Symposium on Estimating Functions, Vol. 32. pp. 259–268.] and Thavaneswaran and Heyde [1999. Prediction via estimating functions. J. Statist. Plann. Inference 77, 89–101.] invoked the theory of estimating functions for one-step ahead prediction in time-series models. This paper addresses the problem of simultaneous estimation of parameters and multi-step ahead prediction of a vector of future random variables in semi-parametric models by extending the inimitable approach of 13 and 14. The proposed technique is in conformity with the paradigm of the modern theory of estimating functions leading to finite sample optimality within a chosen class of estimating functions, which in turn are used to get the predictors. Particular applications of the technique give predictors that enjoy optimality properties with respect to other well-known criteria.     

A new class of measurement-error models,with applications to dietary data

Measurement-error modelling occurs when one cannot observe a covariate, but instead has possibly replicated surrogate versions of this covariate measured with error. The vast majority of the literature in measurement-error modelling assumes (typically with good reason) that given the value of the true but unobserved (latent) covariate, the replicated surrogates are unbiased for latent covariate and conditionally independent. In the area of nutritional epidemiology, there is some evidence from biomarker studies that this simple conditional independence model may break down due to two causes: (a) systematic biases depending on a person's body mass index, and (b) an additional random component of bias, so that the error structure is the same as a one-way random-effects model. We investigate this problem in the context of (1) estimating distribution of usual nutrient intake, (2) estimating the correlation between a nutrient instrument and usual nutrient intake, and (3) estimating the true relative risk from an estimated relative risk using the error-prone covariate. While systematic bias due to body mass index appears to have little effect, the additional random effect in the variance structure is shown to have a potentially important effect on overall results, both on corrections for relative risk estimates and in estimating the distribution of usual nutrient intake. However, the effect of dietary measurement error on both factors is shown via examples to depend strongly on the data set being used. Indeed, one of our data sets suggests that dietary measurement error may be masking a strong risk of fat on breast cancer, while for a second data set this masking is not so clear. Until further understanding of dietary measurement is available, measurement-error corrections must be done on a study-specific basis, sensitivity analyses should be conducted, and even then results of nutritional epidemiology studies relating diet to disease risk should be interpreted cautiously.     

Estimation and Prediction in the Presence of Spatial Confounding for Spatial Linear Models

In studies that produce data with spatial structure, it is common that covariates of interest vary spatially in addition to the error. Because of this, the error and covariate are often correlated. When this occurs, it is difficult to distinguish the covariate effect from residual spatial variation. In an i.i.d. normal error setting, it is well known that this type of correlation produces biased coefficient estimates, but predictions remain unbiased. In a spatial setting, recent studies have shown that coefficient estimates remain biased, but spatial prediction has not been addressed. The purpose of this paper is to provide a more detailed study of coefficient estimation from spatial models when covariate and error are correlated and then begin a formal study regarding spatial prediction. This is carried out by investigating properties of the generalized least squares estimator and the best linear unbiased predictor when a spatial random effect and a covariate are jointly modelled. Under this setup, we demonstrate that the mean squared prediction error is possibly reduced when covariate and error are correlated.     

Applications of statistical methods to American football

The rules of American football favor the strategic placement of the 11 players per team making the identification of statistical tendencies a particularly useful capability. Gambling on American football games is explained. Several automated prediction techniques are discussed and compared, including least squares, weighted least squares, James-Stein, and Harville. A more data-intensive approach is discussed. That approach has coaching implications as well as predictive ability.     

Prediction of Chaotic Time Series in the Presence of Measurement Error: the Importance of Initial Conditions

In this paper we argue that even if a dynamic relationship can be well described by a deterministic system, retrieving this relationship from an empirical time series has to take into account some, although possibly very small measurement error in the observations. Therefore, measuring the initial conditions for prediction may become much more difficult since one now has a combination of deterministic and stochastic elements. We introduce a partial smoothing estimator for estimating the unobserved initial conditions. We will show that this estimator allows to reduce the effects of measurement error for predictions although the reduction may be small in the presence of strong chaotic dynamics. This will be illustrated using the logistic map.     

Factor analysis regression

The paper describes two regression models—principal components and maximum-likelihood factor analysis—which may be used when the stochastic predictor varibles are highly intereorrelated and/or contain measurement error. The two problems can occur jointly, for example in social-survey data where the true (but unobserved) covariance matrix can be singular. Departure from singularity of the sample dispersion matrix is then due to measurement error. We first consider the more elementary principal components regression model, where it is shown that it can be derived as a special case of (i) canonical correlation, and (ii) restricted least squares. The second part consists of the more general maximum-likelihood factor-analysis regression model, which is derived from the generalized inverse of the product of two singular matrices. Also, it is proved that factor-analysis regression can be considered as an instrumental variables estimator and therefore does not depend on whether factors have been “properly” identified in terms of substantive behaviour. Consequently the additional task of rotating factors to “simple structure” does not arise.     

A fractionally integrated Wishart stochastic volatility model

There has recently been growing interest in modeling and estimating alternative continuous time multivariate stochastic volatility models. We propose a continuous time fractionally integrated Wishart stochastic volatility (FIWSV) process, and derive the conditional Laplace transform of the FIWSV model in order to obtain a closed form expression of moments. A two-step procedure is used, namely estimating the parameter of fractional integration via the local Whittle estimator in the first step, and estimating the remaining parameters via the generalized method of moments in the second step. Monte Carlo results for the procedure show a reasonable performance in finite samples. The empirical results for the S&P 500 and FTSE 100 indexes show that the data favor the new FIWSV process rather than the one-factor and two-factor models of the Wishart autoregressive process for the covariance structure.     

Variance Estimation in Spatial Regression Using a Non-parametric Semivariogram Based on Residuals

Abstract.  The empirical semivariogram of residuals from a regression model with stationary errors may be used to estimate the covariance structure of the underlying process. For prediction (kriging) the bias of the semivariogram estimate induced by using residuals instead of errors has only a minor effect because the bias is small for small lags. However, for estimating the variance of estimated regression coefficients and of predictions, the bias due to using residuals can be quite substantial. Thus we propose a method for reducing this bias. The adjusted empirical semivariogram is then isotonized and made conditionally negative-definite and used to estimate the variance of estimated regression coefficients in a general estimating equations setup. Simulation results for least squares and robust regression show that the proposed method works well in linear models with stationary correlated errors.     

An empirical investigation of different operating characteristics of several estimators of the intraclass correlation in the analysis of binary data

This paper is concerned with properties (bias, standard deviation, mean square error and efficiency) of twenty six estimators of the intraclass correlation in the analysis of binary data. Our main interest is to study these properties when data are generated from different distributions. For data generation we considered three over-dispersed binomial distributions, namely, the beta-binomial distribution, the probit normal binomial distribution and a mixture of two binomial distributions. The findings regarding bias, standard deviation and mean squared error of all these estimators, are that (a) in general, the distributions of biases of most of the estimators are negatively skewed. The biases are smallest when data are generated from the beta-binomial distribution and largest when data are generated from the mixture distribution; (b) the standard deviations are smallest when data are generated from the beta-binomial distribution; and (c) the mean squared errors are smallest when data are generated from the beta-binomial distribution and largest when data are generated from the mixture distribution. Of the 26, nine estimators including the maximum likelihood estimator, an estimator based on the optimal quadratic estimating equations of Crowder (1987), and an analysis of variance type estimator is found to have least amount of bias, standard deviation and mean squared error. Also, the distributions of the bias, standard deviation and mean squared error for each of these estimators are, in general, more symmetric than those of the other estimators. Our findings regarding efficiency are that the estimator based on the optimal quadratic estimating equations has consistently high efficiency and least variability in the efficiency results. In the important range in which the intraclass correlation is small (≤0 5), on the average, this estimator shows best efficiency performance. The analysis of variance type estimator seems to do well for larger values of the intraclass correlation. In general, the estimator based on the optimal quadratic estimating equations seems to show best efficiency performance for data from the beta-binomial distribution and the probit normal binomial distribution, and the analysis of variance type estimator seems to do well for data from the mixture distribution.     

Comparison of GEE1 and GEE2 estimation applied to clustered logistic regression

Generalized estimating equations (GEE) have become a popular method for marginal regression modelling of data that occur in clusters. Features of the GEE methodology are the use of a ‘working covariance’, an approximation to the underlying covariance, which is used to improve the efficiency in estimating the regression coefficients, and the ‘sandwich’ estimate of variance, which provides a way of consistently estimating their standard errors. These techniques have been extended to include estimating equations for the underlying correlation structure, both to improve the efficiency of the regression coefficient estimates and to provide estimates of correlations between units in a cluster, when these are of interest. If the mean structure is of primary interest, then a simpler set of equations (GEE1) can be used, whereas if the underlying covariance structure is of interest in its own right, the use of the more complex GEE2 estimating equations is often recommended. In this paper, we compare the effect of increasing the complexity of the ‘working covariances’ on the variance of the parameter estimates, as well as the mean-squared error of the ‘sandwich’ estimate of variance. We give asymptotic expressions for these variances and mean-squared error terms. We use these to study the behaviour of different variants of GEE1 and GEE2 when we change the number of clusters, the cluster size, and the within-cluster correlation. We conclude that the extra complexity of the full GEE2 approach is not usually justified if the mean structure is of primary interest.     

Estimating prediction error in microarray classification: Modifications of the 0.632+ bootstrap when ${\bf n} < {\bf p}$

We are interested in estimating prediction error for a classification model built on high dimensional genomic data when the number of genes (p) greatly exceeds the number of subjects (n). We examine a distance argument supporting the conventional 0.632+ bootstrap proposed for the $n > p$ scenario, modify it for the $n < p$ situation and develop learning curves to describe how the true prediction error varies with the number of subjects in the training set. The curves are then applied to define adjusted resampling estimates for the prediction error in order to achieve a balance in terms of bias and variability. The adjusted resampling methods are proposed as counterparts of the 0.632+ bootstrap when $n < p$ , and are found to improve on the 0.632+ bootstrap and other existing methods in the microarray study scenario when the sample size is small and there is some level of differential expression. The Canadian Journal of Statistics 41: 133–150; 2013 © 2012 Statistical Society of Canada     

Orthogonal projections and the geometry of estimating functions

In this paper, a notion of generalized inner product spaces is introduced to study optimal estimating functions. The basic technique involves an idea of orthogonal projection first introduced by Small and McLeish (1988, 1989, 1991, 1992, 1994). A characterization of orthogonal projections in generalized inner product spaces is given. It is shown that the orthogonal projection of the score function into a linear subspace of estimating functions is optimal in that subspace, and a characterization of optimal estimating functions is given. As special cases of the main results of this paper, we derive the results of Godambe (1985) on the foundation of estimation in stochastic processes, the result of Godambe and Thompson (1989) on the extension of quasi-likelihood, and the generalized estimating equations for multivariate data due to Liang and Zeger (1986). Also we have derived optimal estimating functions in the Bayesian framework.     

The multivariate linear model with multivariatet and intra-class covariance structure

The prediction distribution of future responses from a multivariate linear model with error having a multivariatet-distribution and intra-class covariance structure has been derived. The distribution depends on ρ, the intra-class correlation coefficient. For unknown ρ, the marginal likelihood function of ρ has been obtained and the prediction distribution has been approximated by the estimate of ρ. As an application, a β-expectation tolerance region for the model has been constructed.     

Automatic selection of a linear predictor through frequency domain cross-validation

Given data from a weakly stationary stochastic process in discrete time, and any L-step ahead linear predictor estimated from that data, we will construct an approximately unbiased estimator of the resulting mean squared error of L-step ahead linear prediction. The motivation for the estimator is based on frequency domain cross-validation, and hence the range of validity and applicability of the resulting selection method is not limited by particular assumptions about the structure of the underlying stochastic process or the form of the fitted linear predictors. We also propose a new frequency domain predictor fitting method. The method provides a natural finite-past analog to the existing spectral factorization techniques, and it compares favorably with the existing techniques, both asymptotically and for finite samples. In a Monte Carlo study, we compare several predictor selection methods, at lead times one and five. The performance criterion used is the mean squared prediction error of the selected predictor. The new selection methods work well, and a comparison of results for the two different lead times underscores the need for tailoring the selection criterion to suit the lead time.     

Higher Moments and Prediction‐Based Estimation for the COGARCH(1,1) Model

COGARCH models are continuous time versions of the well‐known GARCH models of financial returns. The first aim of this paper is to show how the method of prediction‐based estimating functions can be applied to draw statistical inference from observations of a COGARCH(1,1) model if the higher‐order structure of the process is clarified. A second aim of the paper is to provide recursive expressions for the joint moments of any fixed order of the process. Asymptotic results are given, and a simulation study shows that the method of prediction‐based estimating function outperforms the other available estimation methods.     

Another look at subset selection using linear least squares

The QR-factorization provides a set of orthogonal variables which has advantages over other orthogonal representations, such as principal components and the singular-value decomposition, in selecting subsets of regression variables by least squares methods. Stopping rules, in particular, are easily understood. A new stopping rule is derived for prediction. This is derived by approximately minimizing the mean squared error in estimating the squared error of prediction. A clear distinction is made between the kind of stopping rule which is relevant when the objective is prediction, and when the objective is asymptotic consistency. Progress with reducing the bias due to the model selection procedure is briefly summarized.     

A generalized least squares estimation method for VARMA models

In this paper a new generalized least squares procedure for estimating VARMA models is proposed. This method differs from existing ones in explicitly considering the stochastic structure of the approximation error that arises when lagged innovations are replaced with lagged residuals obtained from a long VAR. Simulation results indicate that this method performs better than the Double Regression method and similar to Mauricio's (1995) exact maximum likelihood estimation procedure.     

Multivariate Stochastic Volatility via Wishart Processes: A Comment

This comment refers to an error in the methodology for estimating the parameters of the model developed by Philipov and Glickman for modeling multivariate stochastic volatility via Wishart processes. For estimation they used Bayesian techniques. The derived expressions for the full conditionals of the model parameters as well as the expression for the acceptance ratio of the covariance matrix are erroneous. In this erratum all necessary formulae are given to guarantee an appropriate implementation and application of the model.     

The use of Smooth Bootstrap Techniques for Estimating the Error Rate of a Prediction Rule

In this paper we present a simulation study for comparing differents methods for estimating the prediction error rate in a discrimination problem. We consider the Cross-validation, Bootstrap and Bayesian Bootstrap methods for such as problem, while also elaborating on both simple and Bayesian Bootstrap methods by smoothing techniques. We observe as the smoothing procedure lead to improvements in the estimation of the true error rate of the discrimination rule, specially in the case of the smooth Bayesian Bootstrap estimator, whose reduction in M.S.E. resulted from the high positive correlation between the true error rate and its estimations based in this method.