The peculiar shrinkage properties of partial least squares regression

Partial least squares regression has been widely adopted within some areas as a useful alternative to ordinary least squares regression in the manner of other shrinkage methods such as principal components regression and ridge regression. In this paper we examine the nature of this shrinkage and demonstrate that partial least squares regression exhibits some undesirable properties.     

Partial least squares estimator for single-index models

The partial least squares (PLS) approach first constructs new explanatory variables, known as factors (or components), which are linear combinations of available predictor variables. A small subset of these factors is then chosen and retained for prediction. We study the performance of PLS in estimating single-index models, especially when the predictor variables exhibit high collinearity. We show that PLS estimates are consistent up to a constant of proportionality. We present three simulation studies that compare the performance of PLS in estimating single-index models with that of sliced inverse regression (SIR). In the first two studies, we find that PLS performs better than SIR when collinearity exists. In the third study, we learn that PLS performs well even when there are multiple dependent variables, the link function is non-linear and the shape of the functional form is not known.     

Linearity of regression for non-adjacent record values

Let X₁,X₂,… be a sequence of iid random variables having a continuous distribution; by R₁,R₂,… denote the corresponding record values. All the distributions allowing linearity of regressions either E(R_m+k|R_m) or E(R_m|R_m+k) are identified.     

Bayes linear analysis for graphical models: The geometric approach to local computation and interpretive graphics

This paper concerns the geometric treatment of graphical models using Bayes linear methods. We introduce Bayes linear separation as a second order generalised conditional independence relation, and Bayes linear graphical models are constructed using this property. A system of interpretive and diagnostic shadings are given, which summarise the analysis over the associated moral graph. Principles of local computation are outlined for the graphical models, and an algorithm for implementing such computation over the junction tree is described. The approach is illustrated with two examples. The first concerns sales forecasting using a multivariate dynamic linear model. The second concerns inference for the error variance matrices of the model for sales, and illustrates the generality of our geometric approach by treating the matrices directly as random objects. The examples are implemented using a freely available set of object-oriented programming tools for Bayes linear local computation and graphical diagnostic display.     

Diagnostic checks for discrete data regression models using posterior predictive simulations

Model checking with discrete data regressions can be difficult because the usual methods such as residual plots have complicated reference distributions that depend on the parameters in the model. Posterior predictive checks have been proposed as a Bayesian way to average the results of goodness-of-fit tests in the presence of uncertainty in estimation of the parameters. We try this approach using a variety of discrepancy variables for generalized linear models fitted to a historical data set on behavioural learning. We then discuss the general applicability of our findings in the context of a recent applied example on which we have worked. We find that the following discrepancy variables work well, in the sense of being easy to interpret and sensitive to important model failures: structured displays of the entire data set, general discrepancy variables based on plots of binned or smoothed residuals versus predictors and specific discrepancy variables created on the basis of the particular concerns arising in an application. Plots of binned residuals are especially easy to use because their predictive distributions under the model are sufficiently simple that model checks can often be made implicitly. The following discrepancy variables did not work well: scatterplots of latent residuals defined from an underlying continuous model and quantile–quantile plots of these residuals.     

Factors influencing Soay sheep survival

We present a survival analysis of Soay sheep mark recapture and recovery data. Unlike previous conditional analyses, it is not necessary to assume equality of recovery and recapture probabilities; instead these are estimated by maximum likelihood. Male and female sheep are treated separately, with the higher numbers and survival probabilities of the females resulting in a more complex model than that used for the males. In both cases, however, age and time aspects need to be included and there is a strong indication of a reduction in survival for sheep aged 7 years or more. Time variation in survival is related to the size of the population and selected weather variables, by using logistic regression. The size of the population significantly affects the survival probabilities of male and female lambs, and of female sheep aged 7 or more years. March rainfall and a measure of the North Atlantic oscillation are found to influence survival significantly for all age groups considered, for both males and females. Either of these weather variables can be used in a model. Several phenotypic and genotypic individual covariates are also fitted. The only covariate which is found to influence survival significantly is the type of horn of first-year female sheep. There is a substantial variation in the recovery probabilities over time, reflecting in part the increased effort when a population crash was expected. The goodness of fit of the model is checked by using graphical procedures.     

Two ways of modelling overdispersion in non-normal data

For non-normal data assumed to have distributions, such as the Poisson distribution, which have an a priori dispersion parameter, there are two ways of modelling overdispersion: by a quasi-likelihood approach or with a random-effect model. The two approaches yield different variance functions for the response, which may be distinguishable if adequate data are available. The epilepsy data of Thall and Vail and the fabric data of Bissell are used to exemplify the ideas.     

Adaptive Estimation of the Integral of Squared Regression Derivatives

A problem of estimating the integral of a squared regression function and of its squared derivatives has been addressed in a number of papers. For the case of a heteroscedastic model where smoothness of the underlying regression function, the design density, and the variance of errors are known, the asymptotically sharp minimax lower bound and a sharp estimator were found in Pastuchova & Khasminski (1989). However, there are apparently no results on the either rate optimal or sharp optimal adaptive, or data-driven, estimation when neither the degree of regression function smoothness nor design density, scale function and distribution of errors are known. After a brief review of main developments in non-parametric estimation of non-linear functionals, we suggest a simple adaptive estimator for the integral of a squared regression function and its derivatives and prove that it is sharp-optimal whenever the estimated derivative is sufficiently smooth.     

Practical Maximum Pseudolikelihood for Spatial Point Patterns(with Discussion)

This paper describes a technique for computing approximate maximum pseudolikelihood estimates of the parameters of a spatial point process. The method is an extension of Berman & Turner's (1992) device for maximizing the likelihoods of inhomogeneous spatial Poisson processes. For a very wide class of spatial point process models the likelihood is intractable, while the pseudolikelihood is known explicitly, except for the computation of an integral over the sampling region. Approximation of this integral by a finite sum in a special way yields an approximate pseudolikelihood which is formally equivalent to the (weighted) likelihood of a loglinear model with Poisson responses. This can be maximized using standard statistical software for generalized linear or additive models, provided the conditional intensity of the process takes an 'exponential family' form. Using this approach a wide variety of spatial point process models of Gibbs type can be fitted rapidly, incorporating spatial trends, interaction between points, dependence on spatial covariates, and mark information.     

Theory & Methods: Krige,Smooth, Both or Neither?

Both kriging and non-parametric regression smoothing can model a non-stationary regression function with spatially correlated errors. However comparisons have mainly been based on ordinary kriging and smoothing with uncorrelated errors. Ordinary kriging attributes smoothness of the response to spatial autocorrelation whereas non-parametric regression attributes trends to a smooth regression function. For spatial processes it is reasonable to suppose that the response is due to both trend and autocorrelation. This paper reviews methodology for non-parametric regression with autocorrelated errors which is a natural compromise between the two methods. Re-analysis of the one-dimensional stationary spatial data of Laslett (1994) and a clearly non-stationary time series demonstrates the rather surprising result that for these data, ordinary kriging outperforms more computationally intensive models including both universal kriging and correlated splines for spatial prediction. For estimating the regression function, non-parametric regression provides adaptive estimation, but the autocorrelation must be accounted for in selecting the smoothing parameter.