Geometric characterization of planar regression

The geometric characterization of linear regression in terms of the ‘concentration ellipse’ by Galton [Galton, F., 1886, Family likeness in stature (with Appendix by Dickson, J.D.H.). Proceedings of the Royal Society of London, 40, 42–73.] and Pearson [Pearson, K., 1901, On lines and planes of closest fit to systems of points in space. Philosophical Magazine, 2, 559–572.] was extended to the case of unequal variances of the presumably uncorrelated errors in the experimental data [McCartin, B.J., 2003, A geometric characterization of linear regression. Statistics, 37(2), 101–117.]. In this paper, this geometric characterization is further extended to planar (and also linear) regression in three dimensions where a beautiful interpretation in terms of the concentration ellipsoid is developed.     

The peculiar shrinkage properties of partial least squares regression

Partial least squares regression has been widely adopted within some areas as a useful alternative to ordinary least squares regression in the manner of other shrinkage methods such as principal components regression and ridge regression. In this paper we examine the nature of this shrinkage and demonstrate that partial least squares regression exhibits some undesirable properties.     

Component selection norms for principal components regression

Multicollinearity or near exact linear dependence among the vectors of regressor variables in a multiple linear regression analysis can have important effects on the quality of least squares parameter estimates. One frequently suggested approach for these problems is principal components regression. This paper investigates alternative variable selection procedures and their implications for such an analysis.     

Tests of regression coefficients under ridge regression models

This paper investigates two “non-exact” t-type tests, t( k₂) and t(k₂), of the individual coefficients of a linear regression model, based on two ordinary ridge estimators. The reported results are built on a simulation study covering 84 different models. For models with large standard errors, the ridge-based t-tests have correct levels with considerable gain in powers over those of the least squares t-test, t(0). For models with small standard errors, t(k₁) is found to be liberal and is not safe to use while, t(k₂) is found to slightly exceed the nominal level in few cases. When tie two ridge tests art: not winners, the results indicate that they don't loose much against t(0).     

Quantile regression via iterative least squares computations

We present an estimating framework for quantile regression where the usual L ₁-norm objective function is replaced by its smooth parametric approximation. An exact path-following algorithm is derived, leading to the well-known ‘basic’ solutions interpolating exactly a number of observations equal to the number of parameters being estimated. We discuss briefly possible practical implications of the proposed approach, such as early stopping for large data sets, confidence intervals, and additional topics for future research.     

On hsu's model in regression analysis

The paper examplifies with Hsu’s model a general pattern as how to derive results of variance component estimation from well known results on mean estimation, as far as linear model theory is concerned. This ’ dispersion-mean-correspondence‘provides new and short proofs for various theorems from the literature, concerning unbiased invariant quadratic estimators with minimum BAYES risk or minimum variance. For pure variance component models, unbiased non-negative quadratic estimability is characterized in terms of the design matrices.     

Penalized regression with model‐based penalties

Nonparametric regression techniques such as spline smoothing and local fitting depend implicitly on a parametric model. For instance, the cubic smoothing spline estimate of a regression function ∫ μ based on observations ti, Yi is the minimizer of Σ{Yi ‐ μ(ti)}² + λ∫(μ′′)². Since ∫(μ″)² is zero when μ is a line, the cubic smoothing spline estimate favors the parametric model μ(t) = α_o + α₁t. Here the authors consider replacing ∫(μ″)² with the more general expression ∫(Lμ)² where L is a linear differential operator with possibly nonconstant coefficients. The resulting estimate of μ performs well, particularly if Lμ is small. They present an O(n) algorithm for the computation of μ. This algorithm is applicable to a wide class of L's. They also suggest a method for the estimation of L. They study their estimates via simulation and apply them to several data sets.     

A simulation study on SPSS ridge regression and ordinary least squares regression procedures for multicollinearity data

This study compares the SPSS ordinary least squares (OLS) regression and ridge regression procedures in dealing with multicollinearity data. The LS regression method is one of the most frequently applied statistical procedures in application. It is well documented that the LS method is extremely unreliable in parameter estimation while the independent variables are dependent (multicollinearity problem). The Ridge Regression procedure deals with the multicollinearity problem by introducing a small bias in the parameter estimation. The application of Ridge Regression involves the selection of a bias parameter and it is not clear if it works better in applications. This study uses a Monte Carlo method to compare the results of OLS procedure with the Ridge Regression procedure in SPSS.     

Study of partial least squares and ridge regression methods

This article considers both Partial Least Squares (PLS) and Ridge Regression (RR) methods to combat multicollinearity problem. A simulation study has been conducted to compare their performances with respect to Ordinary Least Squares (OLS). With varying degrees of multicollinearity, it is found that both, PLS and RR, estimators produce significant reductions in the Mean Square Error (MSE) and Prediction Mean Square Error (PMSE) over OLS. However, from the simulation study it is evident that the RR performs better when the error variance is large and the PLS estimator achieves its best results when the model includes more variables. However, the advantage of the ridge regression method over PLS is that it can provide the 95% confidence interval for the regression coefficients while PLS cannot.     

Measurement error in regression analysis

Consider the linear regression model Y = Xθ+ ε where Y denotes a vector of n observations on the dependent variable, X is a known matrix, θ is a vector of parameters to be estimated and e is a random vector of uncorrelated errors. If X'X is nearly singular, that is if the smallest characteristic root of X'X s small then a small perurbation in the elements of X, such as due to measurement errors, induces considerable variation in the least squares estimate of θ. In this paper we examine for the asymptotic case when n is large the effect of perturbation with regard to the bias and mean squared error of the estimate.     

Trimmed least squares estimator as best trimmed linear conditional estimator for linear regression model

A class of trimmed linear conditional estimators based on regression quantiles for the linear regression model is introduced. This class serves as a robust analogue of non-robust linear unbiased estimators. Asymptotic analysis then shows that the trimmed least squares estimator based on regression quantiles ( Koenker and Bassett ( 1978 ) ) is the best in this estimator class in terms of asymptotic covariance matrices. The class of trimmed linear conditional estimators contains the Mallows-type bounded influence trimmed means ( see De Jongh et al ( 1988 ) ) and trimmed instrumental variables estimators. A large sample methodology based on trimmed instrumental variables estimator for confidence ellipsoids and hypothesis testing is also provided.     

Comparisons among regression estimators under the generalized mean square error criterion

It is not always prossible to establish a preference ordering among regression estimators in terms of the generalized mean square error criterion. In the paper, we determine when it is feasible to use this criteion to couduct comparisons among ordinary least squares, principal components, ridge regression, and shrunken least squares estimators.     

Multiple linear regression model with stochastic design variables

In a simple multiple linear regression model, the design variables have traditionally been assumed to be non-stochastic. In numerous real-life situations, however, they are stochastic and non-normal. Estimators of parameters applicable to such situations are developed. It is shown that these estimators are efficient and robust. A real-life example is given.     

Comparison of prediction methods for multicollinear data

In this paper we discuss the partial least squares (PLS) prediction method. The method is compared to the predictor based on principal component regression (PCR). Both theoretical considerations and computations on artificial and real data are presented.     

Superiority comparisons between mixed regression estimators

This paper gives necessary and sufficient conditions for a mixed regression estimator to be superior to another mixed estimator. The comparisons are based on the mean square error matrices of the estimators. Both estimators are allowed to be biased.     

Partial least squares Cox regression for genome-wide data

Most methods for survival prediction from high-dimensional genomic data combine the Cox proportional hazards model with some technique of dimension reduction, such as partial least squares regression (PLS). Applying PLS to the Cox model is not entirely straightforward, and multiple approaches have been proposed. The method of Park et al. (Bioinformatics 18(Suppl. 1):S120–S127, 2002) uses a reformulation of the Cox likelihood to a Poisson type likelihood, thereby enabling estimation by iteratively reweighted partial least squares for generalized linear models. We propose a modification of the method of park et al. (2002) such that estimates of the baseline hazard and the gene effects are obtained in separate steps. The resulting method has several advantages over the method of park et al. (2002) and other existing Cox PLS approaches, as it allows for estimation of survival probabilities for new patients, enables a less memory-demanding estimation procedure, and allows for incorporation of lower-dimensional non-genomic variables like disease grade and tumor thickness. We also propose to combine our Cox PLS method with an initial gene selection step in which genes are ordered by their Cox score and only the highest-ranking k% of the genes are retained, obtaining a so-called supervised partial least squares regression method. In simulations, both the unsupervised and the supervised version outperform other Cox PLS methods.     

Prediction of response values in linear regression models from replicated experiments

This paper considers the problem of prediction in a linear regression model when data sets are available from replicated experiments. Pooling the data sets for the estimation of regression parameters, we present three predictors — one arising from the least squares method and two stemming from Stein-rule method. Efficiency properties of these predictors are discussed when they are used to predict actual and average values of response variable within/outside the sample. Received: November 17, 1999; revised version: August 10, 2000     

Estimation of the linear-linear segmented regression model in the presence of measurement error

A simple segmented regression model in which the independent variable is measured with error is considered. The method of moments is used to obtain parameter estimates and the joint asymptotic distribution of the estimators is presented. The small sample properties of the inference procedures based on the asymptotic distribution of the estimators are studied numerically.     

Bayesian least squares estimates of univariate regression functions

The regression function R(?) to be estimated is assumed to have an expansion in terms of specified functions, orthogonalized vich respect to values of the explanatory variable. Relative precisions of OBSERVATION are assumed known. The estimate is the posterior linear mean of R(?) given the data. The investigator plots graphs of appropriate functions as an aid in eliciting his prior means and precisions for the coefficients in the expansion. The method is illustrated by an example using simulated data, an example in which effects of various dosages of Vitamin D are estimated, and an example in which a utility function is estimated.