Penalized angular regression for personalized predictions

Personalization is becoming an important aspect of many predictive applications. We introduce a penalized regression method which inherently implements personalization. Personalized angle (PAN) regression constructs regression coefficients that are specific to the covariate vector for which one is producing a prediction, thus personalizing the regression model itself. This is achieved by penalizing the normalized prediction for a given covariate vector. The method therefore penalizes the normalized regression coefficients, or the angles of the regression coefficients in a hyperspherical parametrization, introducing a new angle-based class of penalties. PAN hence combines two novel concepts: penalizing the normalized coefficients and personalization. For an orthogonal design matrix, we show that the PAN estimator is the solution to a low-dimensional eigenvector equation. Based on the hyperspherical parametrization, we construct an efficient algorithm to calculate the PAN estimator. We propose a parametric bootstrap procedure for selecting the tuning parameter, and simulations show that PAN regression can outperform ordinary least squares, ridge regression and other penalized regression methods in terms of prediction error. Finally, we demonstrate the method in a medical application.     

Multivariate-multiple circular regression

We introduce a fully model-based approach of studying functional relationships between a multivariate circular-dependent variable and several circular covariates, enabling inference regarding all model parameters and related prediction. Two multiple circular regression models are presented for this approach. First, for an univariate circular-dependent variable, we propose the least circular mean-square error (LCMSE) estimation method, and asymptotic properties of the LCMSE estimators and inferential methods are developed and illustrated. Second, using a simulation study, we provide some practical suggestions for model selection between the two models. An illustrative example is given using a real data set from protein structure prediction problem. Finally, a straightforward extension to the case with a multivariate-dependent circular variable is provided.     

Multivariate Calibration — Direct and Indirect Regression Methodology

This paper tries first to introduce and motivate the methodology of multivariate calibration. Next a review is given, mostly avoiding technicalities, of the somewhat messy theory of the subject. Two approaches are distinguished: the estimation approach (controlled calibration) and the prediction approach (natural calibration). Among problems discussed are the choice of estimator, the choice of confidence region, methodology for handling situations with more variables than observations, near-collinearities (with counter-measures like ridge type regression, principal components regression, partial least squares regression and continuum regression), pretreatment of data, and cross-validation vs true prediction. Examples discussed in detail concern estimation of the age of a rhinoceros from its horn lengths (low-dimensional), and nitrate prediction in waste-water from high-dimensional spectroscopic measurements.     

A comparison of various methods for multivariate regression with highly collinear variables

Regression tends to give very unstable and unreliable regression weights when predictors are highly collinear. Several methods have been proposed to counter this problem. A subset of these do so by finding components that summarize the information in the predictors and the criterion variables. The present paper compares six such methods (two of which are almost completely new) to ordinary regression: Partial least Squares (PLS), Principal Component regression (PCR), Principle covariates regression, reduced rank regression, and two variants of what is called power regression. The comparison is mainly done by means of a series of simulation studies, in which data are constructed in various ways, with different degrees of collinearity and noise, and the methods are compared in terms of their capability of recovering the population regression weights, as well as their prediction quality for the complete population. It turns out that recovery of regression weights in situations with collinearity is often very poor by all methods, unless the regression weights lie in the subspace spanning the first few principal components of the predictor variables. In those cases, typically PLS and PCR give the best recoveries of regression weights. The picture is inconclusive, however, because, especially in the study with more real life like simulated data, PLS and PCR gave the poorest recoveries of regression weights in conditions with relatively low noise and collinearity. It seems that PLS and PCR are particularly indicated in cases with much collinearity, whereas in other cases it is better to use ordinary regression. As far as prediction is concerned: Prediction suffers far less from collinearity than recovery of the regression weights.     

Constructive Cross-Validation in Linear Prediction

Approaches for regressor construction in the linear prediction problem are investigated in a framework similar to partial least squares and continuum regression, but weighted to allow for custom specification of an evaluative scheme. A cross-validatory continuum regression procedure is proposed, and shown to compare well with ordinary continuum regression in empirical demonstrations.     

A finite sample comparison of nonparametric estimates of the effective dose in quantal bioassay

To estimate the effective dose level ED_α in the common binary response model, several parametric and nonparametric estimators have been proposed in the literature. In the present article, we focus on nonparametric methods and present a detailed numerical comparison of four different approaches to estimate the ED_α nonparametrically. The methods are briefly reviewed and their finite sample properties are studied by means of a detailed simulation study. Moreover, a data example is presented to illustrate the different concepts.     

Comparison of prediction methods for multicollinear data

In this paper we discuss the partial least squares (PLS) prediction method. The method is compared to the predictor based on principal component regression (PCR). Both theoretical considerations and computations on artificial and real data are presented.     

Another look at subset selection using linear least squares

The QR-factorization provides a set of orthogonal variables which has advantages over other orthogonal representations, such as principal components and the singular-value decomposition, in selecting subsets of regression variables by least squares methods. Stopping rules, in particular, are easily understood. A new stopping rule is derived for prediction. This is derived by approximately minimizing the mean squared error in estimating the squared error of prediction. A clear distinction is made between the kind of stopping rule which is relevant when the objective is prediction, and when the objective is asymptotic consistency. Progress with reducing the bias due to the model selection procedure is briefly summarized.     

Estimating Prediction Error: Cross-Validation vs. Accumulated Prediction Error

We study the validation of prediction rules such as regression models and classification algorithms through two out-of-sample strategies, cross-validation and accumulated prediction error. We use the framework of Efron (1983 Efron , B. ( 1983 ). Estimating the error rate of a prediction rule: improvement on cross-validation . Journal of the American Statistical Association 78 : 316 – 331 .[Taylor & Francis Online], [Web of Science ®] , [Google Scholar]) where measures of prediction errors are defined as sample averages of expected errors and show through exact finite sample calculations that cross-validation and accumulated prediction error yield different smoothing parameter choices in nonparametric regression. The difference in choice does not vanish as sample size increases.     

Confidence intervals for prediction intervals

When working with a single random variable, the simplest and most obvious approach when estimating a 1???γ prediction interval, is to estimate the γ/2 and 1???γ/2 quantiles. The paper compares the small-sample properties of several methods aimed at estimating an interval that contains the 1???γ prediction interval with probability 1???α. In effect, the goal is to compute a 1???α confidence interval for the true 1???γ prediction interval. The only successful method when the sample size is small is based in part on an adaptive kernel estimate of the underlying density. Some simulation results are reported on how an extension to non-parametric regression performs, based on a so-called running interval smoother.     

Bayesian regression with multivariate linear splines

We present a Bayesian analysis of a piecewise linear model constructed by using basis functions which generalizes the univariate linear spline to higher dimensions. Prior distributions are adopted on both the number and the locations of the splines, which leads to a model averaging approach to prediction with predictive distributions that take into account model uncertainty. Conditioning on the data produces a Bayes local linear model with distributions on both predictions and local linear parameters. The method is spatially adaptive and covariate selection is achieved by using splines of lower dimension than the data.     

Minicommunication computation of moments of generalized ridge estimators

This paper deals with a formal identification of outliers in regression based on tests of hypotheses. The hypothesis is not the standard one but is based on performance criteria that relates to the coefficient estimation and predictive capabilities of the model. The cri-teria include the trace of the mean square error matrix on the coefficients and integrated mean square error of prediction. Both the mean shift outlier model and the variance in-flation model are discussed.     

Prediction under uncertainty of degree of polynomial in growth curve models

Random coefficient polynomial regression model has been considered for prediction purpose when there is uncertainty about the degree of the polynomialo Expressions for mean square errors of two predictors based on simple estimators have been derived and their perfomaiices have been compared when parameters are estimated from the sample. A modified predictor has also been suggested when parameters in the predicting equations are to be estimated from the sample. Perform-ance ofseveral predictors haife been compared by cross validation technique from a real set of data.     

On efficient point prediction systems

Assume that a forecaster observes a sequence of random variables and issues predictions according to a point prediction systems, i.e. a rule which, at every time t , issues a point prediction for the next observation at time t +1. We introduce the concept of efficiency of a point prediction system, for the case that the joint distribution of the sequence of observations is known to belong to a parametric family, and performance is assessed by the long run sum of squared prediction errors. Independence is not a requirement. Under weak conditions, the class of efficient point prediction systems is non-empty, and any two efficient point prediction systems will, in a certain strong sense, make asymptotically identical predictions for the infinite future. We discuss the efficiency of point prediction systems based on Bayesian predictive means, and on plugging in parameter estimates. The results are applied to probability forecasting and stochastic regression.     

Simultaneous Confidence Regions for Predictions

After observing n independent responses at n corresponding design points in a linear regression setting, one wishes to make a confidence statement about future responses that will apply simultaneously to all possible design points. Two appropriate prediction regions are derived using normal theory.     

Adaptive lifting for nonparametric regression

Many wavelet shrinkage methods assume that the data are observed on an equally spaced grid of length of the form 2^J for some J. These methods require serious modification or preprocessed data to cope with irregularly spaced data. The lifting scheme is a recent mathematical innovation that obtains a multiscale analysis for irregularly spaced data. A key lifting component is the “predict” step where a prediction of a data point is made. The residual from the prediction is stored and can be thought of as a wavelet coefficient. This article exploits the flexibility of lifting by adaptively choosing the kind of prediction according to a criterion. In this way the smoothness of the underlying ‘wavelet’ can be adapted to the local properties of the function. Multiple observations at a point can readily be handled by lifting through a suitable choice of prediction. We adapt existing shrinkage rules to work with our adaptive lifting methods. We use simulation to demonstrate the improved sparsity of our techniques and improved regression performance when compared to both wavelet and non-wavelet methods suitable for irregular data. We also exhibit the benefits of our adaptive lifting on the real inductance plethysmography and motorcycle data.     

How fair can employers be?

In this article we examine three concepts of fairness in employment decisions. Two of these concepts are widely known in the literature as “Fairness 1”and “Fairness 2”. The third concept, which we refer to as “Fairness 0”, is defined and introduced here. Fairness 0 applies to the hiring stage, whereas Fairness 1 and Fairness 2 apply to the placement or promotion stages of employment. Our results have important policy implications. We show that the three concepts of fairness can only rarely be achieved simultaneously.     

On Combining Wavelets Expansion and Sparse Linear Models for Regression on Metabolomic Data and Biomarker Selection

Wavelet thresholding of spectra has to be handled with care when the spectra are the predictors of a regression problem. Indeed, a blind thresholding of the signal followed by a regression method often leads to deteriorated predictions. The scope of this article is to show that sparse regression methods, applied in the wavelet domain, perform an automatic thresholding: the most relevant wavelet coefficients are selected to optimize the prediction of a given target of interest. This approach can be seen as a joint thresholding designed for a predictive purpose. The method is illustrated on a real world problem where metabolomic data are linked to poison ingestion. This example proves the usefulness of wavelet expansion and the good behavior of sparse and regularized methods. A comparison study is performed between the two-steps approach (wavelet thresholding and regression) and the one-step approach (selection of wavelet coefficients with a sparse regression). The comparison includes two types of wavelet bases, various thresholding methods, and various regression methods and is evaluated by calculating prediction performances. Information about the location of the most important features on the spectra was also obtained and used to identify the most relevant metabolites involved in the mice poisoning.     

BUSDM – an algorithm for the bottom-up search of departures from a model

Searching for regions of the input space where a statistical model is inappropriate is useful in many applications. The study proposes an algorithm for finding local departures from a regression-type prediction model. The algorithm returns low-dimensional hypercubes where the average prediction error clearly departs from zero. The study describes the developed algorithm, and shows successful applications on the simulated and real data from the steel plate production. The algorithms that have been originally developed for searching regions of the high-response value from the input space are reviewed and considered as alternative methods for locating model departures. The proposed algorithm succeeds in locating the model departure regions better than the compared alternatives. The algorithm can be utilized in sequential follow-up of a model as time goes along and new data are observed.     

Transductive versions of the LASSO and the Dantzig Selector

Transductive methods are useful in prediction problems when the training dataset is composed of a large number of unlabeled observations and a smaller number of labeled observations. In this paper, we propose an approach for developing transductive prediction procedures that are able to take advantage of the sparsity in the high dimensional linear regression. More precisely, we define transductive versions of the LASSO (Tibshirani, 1996) and the Dantzig Selector (Candès and Tao, 2007). These procedures combine labeled and unlabeled observations of the training dataset to produce a prediction for the unlabeled observations. We propose an experimental study of the transductive estimators that shows that they improve the LASSO and Dantzig Selector in many situations, and particularly in high dimensional problems when the predictors are correlated. We then provide non-asymptotic theoretical guarantees for these estimation methods. Interestingly, our theoretical results show that the Transductive LASSO and Dantzig Selector satisfy sparsity inequalities under weaker assumptions than those required for the “original” LASSO.