On identification of transfer function models by biased regression methods

This paper investigates a biased regression approach to the preliminary estimation of the Box-Jenkins transfer function weights. Using statistical simulation to generate time series, 14 estimators (various OLS, ridge and principal components estimators) are compared in terms of MSE and standard error of the weight estimators. The estimators are investigated for different levels of multicollinearity, signal-to-noise ratio, number of independent variables, length of time series and number of lags included in the estimation. The results show that the ridge estimators nearly always give lower MSE than the OLS estimator, and in the computationally difficult cases give much lower MSE than the OLS estimator. The principal components estimators can give lower MSE than the OLS, but also higher values. All biased estimators nearly always give much lower estimated standard error than OLS when estimating the weights.     

Using Liu-Type Estimator to Combat Collinearity

Abstract

Linear regression model and least squares method are widely used in many fields of natural and social sciences. In the presence of collinearity, the least squares estimator is unstable and often gives misleading information. Ridge regression is the most common method to overcome this problem. We find that when there exists severe collinearity, the shrinkage parameter selected by existing methods for ridge regression may not fully address the ill conditioning problem. To solve this problem, we propose a new two-parameter estimator. We show using both theoretic results and simulation that our new estimator has two advantages over ridge regression. First, our estimator has less mean squared error (MSE). Second, our estimator can fully address the ill conditioning problem. A numerical example from literature is used to illustrate the results.     

Orthogonal sets of balanced incomplete block designs with nested rows and column

Known series of balanced incomplete block designs with nested rows and columns are used to find orthogonal sets of these designs, producing main effects plans in nested rows and columns. Two infinite series are so constructed and shown to be universally optimum for the analysis with recovery of row and column information, a benefit produced by the additional higher strata orthogonality they enjoy. One of these series achieves orthogonality with just v − 1 replicates of v treatments, fewer than required by Latin squares.     

Local influence in ridge semiparametric models

ABSTRACT

Ridge penalized least-squares estimators has been suggested as an alternative to the minimum penalized sum of squares estimates in the presence of collinearity among the explanatory variables in semiparametric regression models (SPRMs). This paper studies the local influence of minor perturbations on the ridge estimates in the SPRM. The diagnostics under the perturbation of ridge penalized sum of squares, response variable, explanatory variables and ridge parameter are considered. Some local influence diagnostics are given. A Monte Carlo simulation study and a real example are used to illustrate the proposed perturbations.     

Obtaining efficient estimates of park use and testing for the structural adequacy of models

This paper provides an examination of the problem of heteroscedasticity as it relates to estimating park use, although the results can also be applied to a wide variety of flow problems involving traffic, people or commodities. The major issue is that estimates of flows obtained using ordinary least squares, OLS, often yield statistically significant results while still giving rise to large differences between observed and predicted flows (residuals). The paper presents results which show that for the flow estimation problem of concern, more accurate use estimates may be obtained by using generalized least squares, GLS, rather than using OLS. Weights to use in GLS regression are developed taking into account the variance to be expected in origin-destination flows. It is shown that deriving the correct weights, estimates of variances, to use in a regression analysis results in an ‘absolute’ test for the structural appropriateness of the regression model. Tests related to the ‘absolute’ adequacy test are introduced and their use to identify specific structural problems with a model is illustrated.     

Good ridge estimators based on prior information

Ridge regression is re-examined and ridge estimators based on prior information are introduced. A necessary and sufficient condition is given for such ridge estimators to yield estimators of every nonnull linear combination of the regression coefficients with smaller mean square error than that of the Gauss-Markov best linear unbiased estimator.     

On mitigating collinearity through mixtures

In linear models having near collinear columns of X, ridge and surrogate estimators often are used to mitigate collinearity. A new class of estimators is based on mixtures, either of X and a design minimal in an ordered class or of the Fisher information and a scalar matrix. Comparisons are drawn among choices for the mixing parameter, and the estimators are found to be admissible relative to ordinary least squares. Case studies demonstrate that selected mixture designs are perturbed from the original design to a lesser extent than are those of the surrogate method, while retaining reasonable efficiency characteristics.     

EXACT FINITE-SAMPLE PROPERTIES OF A PRE-TEST ESTIMATOR IN RIDGE REGRESSION

This paper discusses a pre-test regression estimator which uses the least squares estimate when it is “large” and a ridge regression estimate for “small” regression coefficients, where the preliminary test is applied separately to each regression coefficient in turn to determine whether it is “large” or “small.” For orthogonal regressors, the exact finite-sample bias and mean squared error of the pre-test estimator are derived. The latter is less biased than a ridge estimator, and over much of the parameter space the pre-test estimator has smaller mean squared error than least squares. A ridge estimator is found to be inferior to the pre-test estimator in terms of mean squared error in many situations, and at worst the latter estimator is only slightly less efficient than the former at commonly used significance levels.     

Ridge Regression: Degrees of Freedom in the Analysis of Variance

It appears to be common practice with ridge regression to obtain a decomposition of the total sum of squares, and assign degrees of freedom, according to established least squares theory. This discussion notes the obvious fallacies of such an approach, and introduces a decomposition based on orthogonality, and degrees of freedom based on expected mean squares, for non-stochastic k.     

Collinearity: revisiting the variance inflation factor in ridge regression

Ridge regression has been widely applied to estimate under collinearity by defining a class of estimators that are dependent on the parameter k. The variance inflation factor (VIF) is applied to detect the presence of collinearity and also as an objective method to obtain the value of k in ridge regression. Contrarily to the definition of the VIF, the expressions traditionally applied in ridge regression do not necessarily lead to values of VIFs equal to or greater than 1. This work presents an alternative expression to calculate the VIF in ridge regression that satisfies the aforementioned condition and also presents other interesting properties.     

ROBUST RIDGE REGRESSION BASED ON AN M-ESTIMATOR

Consider the linear regression model y =β₀1 +Xβ+ in the usual notation. It is argued that the class of ordinary ridge estimators obtained by shrinking the least squares estimator by the matrix (X¹X + kI)^-1X'X is sensitive to outliers in the ^variable. To overcome this problem, we propose a new class of ridge-type M-estimators, obtained by shrinking an M-estimator (instead of the least squares estimator) by the same matrix. Since the optimal value of the ridge parameter k is unknown, we suggest a procedure for choosing it adaptively. In a reasonably large scale simulation study with a particular M-estimator, we found that if the conditions are such that the M-estimator is more efficient than the least squares estimator then the corresponding ridge-type M-estimator proposed here is better, in terms of a Mean Squared Error criteria, than the ordinary ridge estimator with k chosen suitably. An example illustrates that the estimators proposed here are less sensitive to outliers in the y-variable than ordinary ridge estimators.     

Developing Ridge Parameters for SUR Model

This paper proposes a number of procedures for developing new biased estimators of the seemingly unrelated regression (SUR) parameters, when the explanatory variables are affected by multicollinearity. Several ridge parameters are proposed and then compared in terms of the trace mean squared error (TMSE) and (PR) criteria. The PR criterion is the proportion of replication (out of 1,000) for which the SUR version of the generalized least squares (SGLS) estimator has a smaller TMSE than others. The study was performed using Monte Carlo simulations where the number of equations in the system, the number of observations, the correlation among equations, and the correlation between explanatory variables have been varied. For each model, we performed 1,000 replications. Our results show that under certain conditions some of the proposed SUR ridge parameters, (R _Sgeom , R _Skmed , R _Sqarith , and R _Sqmax ), performed well when compared, in terms of TMSE and PR criteria, with other proposed and popular existing ridge parameters. In large samples and when the collinearity between the explanatory variables is not high, the unbiased SUR estimator (SGLS), performed better than the other ridge parameters.     

A Population Approach to Analysis of Variance Models

Several authors have contributed to what can now be considered a rather complete theory for analysis of variance in cases with orthogonal factors. By using this theory on an assumed basic reference population, the orthogonality concept gives a natural definition of independence between factors in the population. By looking upon the treated units in designed experiments as a formal sample from a future population about which we want to make inference, a natural parametrization of expectations and variances connected to such experiments arises. This approach seems to throw light upon several controversial questions in the theory of mixed models. Also, it gives a framework for discussing the choice of conditioning in models     

Construction of orthogonal Latin hypercube designs with flexible run sizes

Latin hypercube designs (LHDs) have recently found wide applications in computer experiments. A number of methods have been proposed to construct LHDs with orthogonality among main-effects. When second-order effects are present, it is desirable that an orthogonal LHD satisfies the property that the sum of elementwise products of any three columns (whether distinct or not) is 0. The orthogonal LHDs constructed by Ye (1998), Cioppa and Lucas (2007), Sun et al. (2009) and Georgiou (2009) all have this property. However, the run size n of any design in the former three references must be a power of two (n=2^c) or a power of two plus one (n=2^c+1), which is a rather severe restriction. In this paper, we construct orthogonal LHDs with more flexible run sizes which also have the property that the sum of elementwise product of any three columns is 0. Further, we compare the proposed designs with some existing orthogonal LHDs, and prove that any orthogonal LHD with this property, including the proposed orthogonal LHD, is optimal in the sense of having the minimum values of ave(|t|), t_max, ave(|q|) and q_max.     

Some new classes of orthogonal Latin hypercube designs

Orthogonal Latin hypercube (OLH) is a good design choice in a polynomial function model for computer experiments, because it ensures uncorrelated estimation of linear effects when a first-order model is fitted. However, when a second-order model is adopted, an OLH also needs to satisfy the additional property that each column is orthogonal to the elementwise square of all columns and orthogonal to the elementwise product of every pair of columns. Such class of OLHs is called OLHs of order two while the former class just possessing two-dimensional orthogonality is called OLHs of order one. In this paper we present a general method for constructing OLHs of orders one and two for n=s^m$n=s^m$ runs, where s and m may be any positive integers greater than one, by rotating a grouped orthogonal array with a column-orthogonal rotation matrix. The Kronecker product and the stacking methods are revisited and combined to construct some new classes of OLHs of orders one and two with other flexible numbers of runs. Some useful OLHs of order one or two with larger factor-to-run ratio and moderate runs are tabulated and discussed.     

Consistency and asymptotic unbiasedness of S2 in the serially correlated error components regression model for panel data

TheOLS-estimator of the disturbance variance in the linear regression model for panel data is shown to be asymptotically unbiased and weakly consistent when the disturbances follow an error component model with serially correlated time effects.     

r − k Class estimator in the linear regression model with correlated errors

Autocorrelation in errors and multicollinearity among the regressors are serious problems in regression analysis. The aim of this paper is to examine multicollinearity and autocorrelation problems concurrently and to compare the r ? k class estimator to the generalized least squares estimator, the principal components regression estimator and the ridge regression estimator by the scalar and matrix mean square error criteria in the linear regression model with correlated errors.     

Another twist on the equality of OLS and GLS

We give a neccessary and sufficient condition for the equality of theOLS andGLS estimator of a subset of the regression coefficients in a linear model.     

Selecting the optimum k in ridge regression

Two ridge rules are proposed for selecting the optimal k in ridge regression . Since the sampling distribution of the proposed rules are mathematically in tractable , a Monte Carlo study is conducted to examine their statisticl properties . Numerical results of the simulations in dicate that the performance of ridge rules depends upon the risk function used. Nevertheless, one of the ridge rules does produce a smaller mean squared error than the least squares estimator with the probability greater than 0.57 for all situations.     

Simulation study of new estimators combining the SUR ridge regression and the restricted least squares methodologies

In this paper, we propose two SUR type estimators based on combining the SUR ridge regression and the restricted least squares methods. In the sequel these estimators are designated as the restricted ridge Liu estimator and the restricted ridge HK estimator (see Liu in Commun Statist Thoery Methods 22(2):393–402, 1993; Sarkar in Commun Statist A 21:1987–2000, 1992). The study has been made using Monte Carlo techniques, (1,000 replications), under certain conditions where a number of factors that may effect their performance have been varied. The performance of the proposed and some of the existing estimators are evaluated by means of the TMSE and the PR criteria. Our results indicate that the proposed SUR restricted ridge estimators based on K _SUR, K _Sratio, K _Mratio and [(K)\ddot]{\ddot{K}} produced smaller TMSE and/or PR values than the remaining estimators. In contrast with other ridge estimators, components of [(K)\ddot]{\ddot{K}} are defined in terms of the eigenvalues of X^*￠ X^*{X^{{\ast^{\prime}}} X^{\rm \ast}} and all lie in the open interval (0, 1).