Classification trees aided mixed regression model

This paper introduces a novel hybrid regression method (MixReg) combining two linear regression methods, ordinary least square (OLS) and least squares ratio (LSR) regression. LSR regression is a method to find the regression coefficients minimizing the sum of squared error rate while OLS minimizes the sum of squared error itself. The goal of this study is to combine two methods in a way that the proposed method superior both OLS and LSR regression methods in terms of R² statistics and relative error rate. Applications of MixReg, on both simulated and real data, show that MixReg method outperforms both OLS and LSR regression.     

Study of partial least squares and ridge regression methods

This article considers both Partial Least Squares (PLS) and Ridge Regression (RR) methods to combat multicollinearity problem. A simulation study has been conducted to compare their performances with respect to Ordinary Least Squares (OLS). With varying degrees of multicollinearity, it is found that both, PLS and RR, estimators produce significant reductions in the Mean Square Error (MSE) and Prediction Mean Square Error (PMSE) over OLS. However, from the simulation study it is evident that the RR performs better when the error variance is large and the PLS estimator achieves its best results when the model includes more variables. However, the advantage of the ridge regression method over PLS is that it can provide the 95% confidence interval for the regression coefficients while PLS cannot.     

Efficiency analysis of ten estimation procedures for quantitative linear models with autocorrelated errors

Many estimation procedures for quantitative linear models with autocorrelated errors have been proposed in the literature. A number of these procedures have been compared in various ways for different sample sizes and autocorrelation parameters values and for structured or random explanatory vaiables. In this paper, we revisit three situations that were considered to some extent in previous studies, by comparing ten estimation procedures: Ordinary Least Squares (OLS), Generalized Least Squares (GLS), estimated Generalized Least Squares (six procedures), Maximum Likelihood (ML), and First Differences (FD). The six estimated GLS procedures and the ML procedure differ in the way the error autocovariance matrix is estimated. The three situations can be defined as follows: Case 1, the explanatory variable x in the simple linear regression is fixed; Case 2,x is purely random; and Case 3x is first-order autoregressive. Following a theoretical presentation, the ten estimation procedures are compared in a Monte Carlo study conducted in the time domain, where the errors are first-order autoregressive in Cases 1-3. The measure of comparison for the estimation procedures is their efficiency relative to OLS. It is evaluated as a function of the time series length and the magnitude and sign of the error autocorrelation parameter. Overall, knowledge of the model of the time series process generating the errors enhances efficiency in estimated GLS. Differences in the efficiency of estimation procedures between Case 1 and Cases 2 and 3 as well as differences in efficiency among procedures in a given situation are observed and discussed.     

Estimating the duration of dynamic effects with temporally aggregated observations

This paper discusses how an Ordinary Least Squares (OLS) estimator can be used to obtain reasonably accurate estimates of the duration of dynamic effects in a Koyck model framework without knowledge of the true level of temporal aggregation of the data. With proper changes in the analytic derivation, the approach can be extended to other dynamic models.     

On the asymptotic bias of OLS in dynamic regression models with autocorrelated errors

It is well-known that Ordinary Least Squares (OLS) yields inconsistent estimates if applied to a regression equation with lagged dependent variables and correlated errors. Bias expressions which appear in the literature usually assume the exogenous variables to be non-stochastic. Due to this assumption the numerical sizes of these expressions cannot be determined. Further, the analysis is mostly restricted to very simple models. In this paper the problem of calculating the asymptotic bias of OLS is generalized to stationary dynamic regression models, where the errors follow a stationary ARMA process. A general bias expression is derived and a method is introduced by which its actual size can be computed numerically.     

OLS与ML：回归模型两种参数估计方法的比较研究

最小二乘法(OLS)和最大似然法(ML)是回归模型参数估计的两种最重要的方法。 但二者有着明显的差别,本文就二者之间的有关差别进行比较。     

Efficiency and Validity Analyses of Two-Stage Estimation Procedures and Derived Testing Procedures in Quantitative Linear Models with AR(1) Errors

Abstract

In a quantitative linear model with errors following a stationary Gaussian, first-order autoregressive or AR(1) process, Generalized Least Squares (GLS) on raw data and Ordinary Least Squares (OLS) on prewhitened data are efficient methods of estimation of the slope parameters when the autocorrelation parameter of the error AR(1) process, ρ, is known. In practice, ρ is generally unknown. In the so-called two-stage estimation procedures, ρ is then estimated first before using the estimate of ρ to transform the data and estimate the slope parameters by OLS on the transformed data. Different estimators of ρ have been considered in previous studies. In this article, we study nine two-stage estimation procedures for their efficiency in estimating the slope parameters. Six of them (i.e., three noniterative, three iterative) are based on three estimators of ρ that have been considered previously. Two more (i.e., one noniterative, one iterative) are based on a new estimator of ρ that we propose: it is provided by the sample autocorrelation coefficient of the OLS residuals at lag 1, denoted r(1). Lastly, REstricted Maximum Likelihood (REML) represents a different type of two-stage estimation procedure whose efficiency has not been compared to the others yet. We also study the validity of the testing procedures derived from GLS and the nine two-stage estimation procedures. Efficiency and validity are analyzed in a Monte Carlo study. Three types of explanatory variable x in a simple quantitative linear model with AR(1) errors are considered in the time domain: Case 1, x is fixed; Case 2, x is purely random; and Case 3, x follows an AR(1) process with the same autocorrelation parameter value as the error AR(1) process. In a preliminary step, the number of inadmissible estimates and the efficiency of the different estimators of ρ are compared empirically, whereas their approximate expected value in finite samples and their asymptotic variance are derived theoretically. Thereafter, the efficiency of the estimation procedures and the validity of the derived testing procedures are discussed in terms of the sample size and the magnitude and sign of ρ. The noniterative two-stage estimation procedure based on the new estimator of ρ is shown to be more efficient for moderate values of ρ at small sample sizes. With the exception of small sample sizes, REML and its derived F-test perform the best overall. The asymptotic equivalence of two-stage estimation procedures, besides REML, is observed empirically. Differences related to the nature, fixed or random (uncorrelated or autocorrelated), of the explanatory variable are also discussed.     

Generalized mean squared error properties of regression estimators

Theobald (1974) compares Ordinary Least Squares and Ridge Regression estimators of regression parameters using a generalized mean squared error criterion. This paper presents the generalized mean squared error of a Principal Components Regression estimator and comparisons are made with each of the above estimators. In general the choice of which estimator to use depends on the magnitude and the orientation of the unknown parameter vector.     

Local linear regression with adaptive orthogonal fitting for the wind power application

Short-term forecasting of wind generation requires a model of the function for the conversion of meteorological variables (mainly wind speed) to power production. Such a power curve is nonlinear and bounded, in addition to being nonstationary. Local linear regression is an appealing nonparametric approach for power curve estimation, for which the model coefficients can be tracked with recursive Least Squares (LS) methods. This may lead to an inaccurate estimate of the true power curve, owing to the assumption that a noise component is present on the response variable axis only. Therefore, this assumption is relaxed here, by describing a local linear regression with orthogonal fit. Local linear coefficients are defined as those which minimize a weighted Total Least Squares (TLS) criterion. An adaptive estimation method is introduced in order to accommodate nonstationarity. This has the additional benefit of lowering the computational costs of updating local coefficients every time new observations become available. The estimation method is based on tracking the left-most eigenvector of the augmented covariance matrix. A robustification of the estimation method is also proposed. Simulations on semi-artificial datasets (for which the true power curve is available) underline the properties of the proposed regression and related estimation methods. An important result is the significantly higher ability of local polynomial regression with orthogonal fit to accurately approximate the target regression, even though it may hardly be visible when calculating error criteria against corrupted data.     

A NOTE ON ESTIMATING LINEAR TREND IN A REGRESSION MODEL WITH SERIALLY CORRELATED ERROR COMPONENTS

ABSTRACT

We consider a linear trend regression model when the disturbances follow a serially correlated one-way error component model. In this model, we investigate the performance of the Ordinary Least Squares Esitmator (OLSE), First Difference Estimator (FDE), Generalized Least Squares Estimator (GLSE) and the Cochrane-Orcutt-Transformation Estimator (COTE) of slope coefficient in terms of efficiency. The main findings are as follows: (1) when the autocorrelation is close to unity, then the FDE is approximately the GLSE; (2) the OLSE is better than the COTE; and (3) when the value of the autocorrelation is kept constant and T → ∞, the OLSE, COTE and GLSE are asymptotically equivalent whereas the FDE is worse than the other estimators in terms of efficiency.     

Comparison of estimators for heteroskedastic models

Ordinary least squares (OLS) yield inefficient parameter estimates and inconsistent estimates of the covariance matrix in case of heteroskedastic errors. Robinson's adaptive estimator and the Cragg estimator avoid any explicit parameterization of heteroskedasticity, and reduce the danger of misspecification. A small Monte Carlo experiment is performed to compare the behavior of the adaptive estimator with the performance of the Cragg estimator. The Monte Carlo experiment includes simulations of the Generalized Least Squares (GLS) estimator. Indeed, an interesting question is how more sophisticated techniques, like the adaptive estimator, compare with GLS when the latter relies on an incorrect specification of the heteroskedastic process. It turns out that the regression parameters, when estimated adaptively, display small mean squared errors and great efficiency in case of medium or high heteroskedasticity. The covariance matrix, instead, is better estimated by the Cragg estimator or by GLS based on a misspecified error term, since the adaptive estimator overpredicts the standard errors of the regression parameters.     

A new biased estimator based on ridge estimation

In this paper we introduce a new biased estimator for the vector of parameters in a linear regression model and discuss its properties. We show that our new biased estimator is superior, in the mean square error(mse) sense, to the ordinary least squares (OLS) estimator, the ordinary ridge regression (ORR) estimator and the Liu estimator. We also compare the performance of our new biased estimator with two other special Liu-type estimators proposed in Liu (2003). We illustrate our findings with a numerical example based on the widely analysed dataset on Portland cement.     

[HDDA] sparse subspace constrained partial least squares

ABSTRACT

In this paper, we investigate the objective function and deflation process for sparse Partial Least Squares (PLS) regression with multiple components. While many have considered variations on the objective for sparse PLS, the deflation process for sparse PLS has not received as much attention. Our work highlights a flaw in the Statistically Inspired Modification of Partial Least Squares (SIMPLS) deflation method when applied in sparse PLS regression. We also consider the Nonlinear Iterative Partial Least Squares (NIPALS) deflation in sparse PLS regression. To remedy the flaw in the SIMPLS method, we propose a new sparse PLS method wherein the direction vectors are constrained to be sparse and lie in a chosen subspace. We give insight into this new PLS procedure and show through examples and simulation studies that the proposed technique can outperform alternative sparse PLS techniques in coefficient estimation. Moreover, our analysis reveals a simple renormalization step that can be used to improve the estimation of sparse PLS direction vectors generated using any convex relaxation method.     

Bias in the ordinary least squares estimator in the dynamic linear regression model with autocorrelated disturbances

We consider the bias in the Ordinary Least Squares estimator in the linear regression model with a lagged dependent variable as regressor. Results are obtained with independent and auto-correlated disturbances. Asymptotic results are obtained analytically, and finite sample results based on a Monte Carlo study. The substantial biases found suggest the need for an alternative estimator to Ordinary Least Squares and powerful tests for autocorrelated disturbances in the dynamic model.     

Stepwise Regression in Mixed Quantitative Linear Models with Autocorrelated Errors

ABSTRACT

In the stepwise procedure of selection of a fixed or a random explanatory variable in a mixed quantitative linear model with errors following a Gaussian stationary autocorrelated process, we have studied the efficiency of five estimators relative to Generalized Least Squares (GLS): Ordinary Least Squares (OLS), Maximum Likelihood (ML), Restricted Maximum Likelihood (REML), First Differences (FD), and First-Difference Ratios (FDR). We have also studied the validity and power of seven derived testing procedures, to assess the significance of the slope of the candidate explanatory variable x ₂ to enter the model in which there is already one regressor x ₁. In addition to five testing procedures of the literature, we considered the FDR t-test with n ? 3 df and the modified t-test with n? ? 3 df for partial correlations, where n? is Dutilleul's effective sample size. Efficiency, validity, and power were analyzed by Monte Carlo simulations, as functions of the nature, fixed vs. random (purely random or autocorrelated), of x ₁ and x ₂, the sample size and the autocorrelation of random terms in the regression model. We report extensive results for the autocorrelation structure of first-order autoregressive [AR(1)] type, and discuss results we obtained for other autocorrelation structures, such as spherical semivariogram, first-order moving average [MA(1)] and ARMA(1,1), but we could not present because of space constraints. Overall, we found that:

1. the efficiency of slope estimators and the validity of testing procedures depend primarily on the nature of x ₂, but not on that of x ₁;

2. FDR is the most inefficient slope estimator, regardless of the nature of x ₁ and x ₂;

3. REML is the most efficient of the slope estimators compared relative to GLS, provided the specified autocorrelation structure is correct and the sample size is large enough to ensure the convergence of its optimization algorithm;

4. the FDR t-test, the modified t-test and the REML t-test are the most valid of the testing procedures compared, despite the inefficiency of the FDR and OLS slope estimators for the former two;

5. the FDR t-test, however, suffers from a lack of power that varies with the nature of x ₁ and x ₂; and

6. the modified t-test for partial correlations, which does not require the specification of an autocorrelation structure, can be recommended when x ₁ is fixed or random and x ₂ is random, whether purely random or autocorrelated. Our results are illustrated by the environmental data that motivated our work.

     

To be or not to be valid in testing the significance of the slope in simple quantitative linear models with autocorrelated errors

In this article, the validity of procedures for testing the significance of the slope in quantitative linear models with one explanatory variable and first-order autoregressive [AR(1)] errors is analyzed in a Monte Carlo study conducted in the time domain. Two cases are considered for the regressor: fixed and trended versus random and AR(1). In addition to the classical t -test using the Ordinary Least Squares (OLS) estimator of the slope and its standard error, we consider seven t -tests with n-2\,\hbox{df} built on the Generalized Least Squares (GLS) estimator or an estimated GLS estimator, three variants of the classical t -test with different variances of the OLS estimator, two asymptotic tests built on the Maximum Likelihood (ML) estimator, the F -test for fixed effects based on the Restricted Maximum Likelihood (REML) estimator in the mixed-model approach, two t -tests with n - 2 df based on first differences (FD) and first-difference ratios (FDR), and four modified t -tests using various corrections of the number of degrees of freedom. The FDR t -test, the REML F -test and the modified t -test using Dutilleul's effective sample size are the most valid among the testing procedures that do not assume the complete knowledge of the covariance matrix of the errors. However, modified t -tests are not applicable and the FDR t -test suffers from a lack of power when the regressor is fixed and trended ( i.e. , FDR is the same as FD in this case when observations are equally spaced), whereas the REML algorithm fails to converge at small sample sizes. The classical t -test is valid when the regressor is fixed and trended and autocorrelation among errors is predominantly negative, and when the regressor is random and AR(1), like the errors, and autocorrelation is moderately negative or positive. We discuss the results graphically, in terms of the circularity condition defined in repeated measures ANOVA and of the effective sample size used in correlation analysis with autocorrelated sample data. An example with environmental data is presented.     

New difference-based estimator of parameters in semiparametric regression models

The paper introduces a new difference-based Liu estimator β?_Ldiff=([Xtilde]′[Xtilde]+I)^?1([Xtilde]′[ytilde]+η β?_diff) of the regression parameters β in the semiparametric regression model, y=Xβ+f+?. Difference-based estimator, β?_diff=([Xtilde]′[Xtilde])^?1[Xtilde]′[ytilde] and difference-based Liu estimator are analysed and compared with respect to mean-squared error (mse) criterion. Finally, the performance of the new estimator is evaluated for a real data set. Monte Carlo simulation is given to show the improvement in the scalar mse of the estimator.     

Model selection using information criteria under a new estimation method: least squares ratio

In this study, we evaluate several forms of both Akaike-type and Information Complexity (ICOMP)-type information criteria, in the context of selecting an optimal subset least squares ratio (LSR) regression model. Our simulation studies are designed to mimic many characteristics present in real data – heavy tails, multicollinearity, redundant variables, and completely unnecessary variables. Our findings are that LSR in conjunction with one of the ICOMP criteria is very good at selecting the true model. Finally, we apply these methods to the familiar body fat data set.