Regression Shrinkage Methods and Autoregressive Time Series Prediction

The pros and cons of applying regression shrinkage prediction arguments and methods to autoregressive time series forecasting are discussed. Simulation evidence of the performance of a Stein regression prediction formula suggests that the overall dominance of the shrunken predictor over least squares in regression no longer holds in time series samples of a reasonable length. Rather, shrinkage appears the better of the two, with respect to prediction mean squared error, only for weaker relationships and seems to be inferior to the least squares predictor when the autoregressive relationship is strong.     

A simulation study of five biased estimators for straight line regression

Five biased estimators of the slope in straight line regression are considered. For each, the estimate of the “bias parameter”, k, is a function of N, the number of observations, and [rcirc]² , the square of the least squares estimate of the standardized slope, β. The estimators include that of Farebrother, the ridge estimator of Hoerl, Kennard, and Baldwin, Vinod's shrunken estimators., and a new modification of one of the latter. Properties of the estimators are studied for 13 combinations of N and 3. Results of simulation experiments provide empirical evidence concerning the values of means and variances of the biased estimators of the slope and estimates of the “bias parameter”, the mean square errors of the estimators, and the frequency of improvement relative to least squares. Adjustments to degrees of freedom in the biased regression analysis of variance table are also considered. An extension of the new modification to the case of p> 1 independent variables is presented in an Appendix.     

EXACT FINITE-SAMPLE PROPERTIES OF A PRE-TEST ESTIMATOR IN RIDGE REGRESSION

This paper discusses a pre-test regression estimator which uses the least squares estimate when it is “large” and a ridge regression estimate for “small” regression coefficients, where the preliminary test is applied separately to each regression coefficient in turn to determine whether it is “large” or “small.” For orthogonal regressors, the exact finite-sample bias and mean squared error of the pre-test estimator are derived. The latter is less biased than a ridge estimator, and over much of the parameter space the pre-test estimator has smaller mean squared error than least squares. A ridge estimator is found to be inferior to the pre-test estimator in terms of mean squared error in many situations, and at worst the latter estimator is only slightly less efficient than the former at commonly used significance levels.     

Applying Least Absolute Deviation Regression to Regression-type Estimation of the Index of a Stable Distribution Using the Characteristic Function

Least absolute deviation regression is applied using a fixed number of points for all values of the index to estimate the index and scale parameter of the stable distribution using regression methods based on the empirical characteristic function. The recognized fixed number of points estimation procedure uses ten points in the interval zero to one, and least squares estimation. It is shown that using the more robust least absolute regression based on iteratively re-weighted least squares outperforms the least squares procedure with respect to bias and also mean square error in smaller samples.     

Bias Reduction through First-order Mean Correction, Bootstrapping and Recursive Mean Adjustment

Standard methods of estimation for autoregressive models are known to be biased in finite samples, which has implications for estimation, hypothesis testing, confidence interval construction and forecasting. Three methods of bias reduction are considered here: first-order bias correction, FOBC, where the total bias is approximated by the O(T^-1) bias; bootstrapping; and recursive mean adjustment, RMA. In addition, we show how first-order bias correction is related to linear bias correction. The practically important case where the AR model includes an unknown linear trend is considered in detail. The fidelity of nominal to actual coverage of confidence intervals is also assessed. A simulation study covers the AR(1) model and a number of extensions based on the empirical AR(p) models fitted by Nelson & Plosser (1982). Overall, which method dominates depends on the criterion adopted: bootstrapping tends to be the best at reducing bias, recursive mean adjustment is best at reducing mean squared error, whilst FOBC does particularly well in maintaining the fidelity of confidence intervals.     

Iterated Bootstrap-t Confidence Intervals for Density Functions

Abstract.  Conventional bootstrap- t intervals for density functions based on kernel density estimators exhibit poor coverages due to failure of the bootstrap to estimate the bias correctly. The problem can be resolved by either estimating the bias explicitly or undersmoothing the kernel density estimate to undermine its bias asymptotically. The resulting bias-corrected intervals have an optimal coverage error of order arbitrarily close to second order for a sufficiently smooth density function. We investigated the effects on coverage error of both bias-corrected intervals when the nominal coverage level is calibrated by the iterated bootstrap. In either case, an asymptotic reduction of coverage error is possible provided that the bias terms are handled using an extra round of smoothed bootstrapping. Under appropriate smoothness conditions, the optimal coverage error of the iterated bootstrap- t intervals has order arbitrarily close to third order. Examples of both simulated and real data are reported to illustrate the iterated bootstrap procedures.     

Estimating expected sojourn times for a markov renewal process

We consider the estimation of the expected sojourn time in a Markov renewal process under the data condition that only the counts of the exits from the states are available for fixed intervals of time. For analytical and illustrative purposes we concentrate on the two-state process case. We present least squares and method of moments estimators and compare their statistical properties both analytically and empirically. We also present modified estimators with improved properties based upon an overlapping interval sampling strategy. The major results indicate that the least squares estimator is biased in general with the bias depending on the size of the sampling interval and the first two moments of the sojourn time distribution function. The bias becomes negligible as the size of the sampling interval increases. Analytical and empirical results indicate that the method of moments estimator is less sensitive to the size of the sampling interval and has slightly better mean squared error properties than the least squares estimator.     

Improving the finite sample performance of autoregression estimators in dynamic factor models: A bootstrap approach

We investigate the finite sample properties of the estimator of a persistence parameter of an unobservable common factor when the factor is estimated by the principal components method. When the number of cross-sectional observations is not sufficiently large, relative to the number of time series observations, the autoregressive coefficient estimator of a positively autocorrelated factor is biased downward, and the bias becomes larger for a more persistent factor. Based on theoretical and simulation analyses, we show that bootstrap procedures are effective in reducing the bias, and bootstrap confidence intervals outperform naive asymptotic confidence intervals in terms of the coverage probability.     

A new estimator for the unit root

We compare the ordinary least squares, weighted symmetric, modified weighted symmetric (MWS), maximum likelihood, and our new modification for least squares (MLS) estimator for first-order autoregressive in the case of unit root using Monte Carlo method. The Monte Carlo study sheds some light on how well the estimators and the predictors perform on different samples sizes. We found that MLS estimator is less biased and has less mean squared error (MSE) than any other estimators, and MWS predictor error performs well, in the sense of MSE, than any other predictors’ methods. The sample percentiles for the distribution of the τ statistic for the first, second, and third periods in the future, for alternative estimators, are reported to know if it agrees with those of normal distribution or not.     

A Generalized Diagonal Ridge-type Estimator in Linear Regression

This article introduces a general class of biased estimator, namely a generalized diagonal ridge-type (GDR) estimator, for the linear regression model when multicollinearity occurs. The estimator represents different kinds of biased estimators when different parameters are obtained. Some properties of this estimator are discussed and an iterative procedure is provided for selecting the parameters. A Monte Carlo simulation study and an application show that the GDR estimator performs much better than the ordinary least squares (OLS) estimator under the mean square error (MSE) criterion when severe multicollinearity is present.     

Improper use of the ordinary least squares estimator in the switching regression model

The paper considers the consequences of incorrectly using the ordinary least squares estimator, when the true but unknown model is a switching regression. Bias and mean square error express ons are given for slope and residual variance estimators. Except for in very specialized cases the estimators are biased. A numerical exarnple illustrates some of the issues raised and provides a conpelison between the ordinary least squares and maximum likelihood estimators.     

Ridge and related estimation procedures: theory and practice

For the classical linear regression problem, a number of estimators alternative to least squares have been proposed for situations in which multicollinearity is a problem. There is, however, relatively little known about how these estimators behave in practice. This paper investigates mean square error properties for a number of biased regression estimators, and discusses some practical implications of the use of such estimators, A conclusion is that certain types of ridge estimatorsappear to have good mean square error properties, and this may be useful in situations in which mean square error is important     

An M-Estimator for Estimating the Extended Burr Type III Parameters with Outliers

In this article, we present an M-estimator to estimate the parameters of the extended three-parameter Burr Type III distribution for complete data with outliers. The confidence intervals for all parameters can be obtained by the M-estimator's normal approximation. The simulation results show that the M-estimator generally outperforms the maximum likelihood and least squares methods in terms of bias and root mean square errors. We also investigate the M-estimator's impact on different quantiles and the mean for the Weibull and normal distributions with outliers. Two numerical examples are used to demonstrate the performance of our proposed method.     

Estimation and Prediction in the Presence of Spatial Confounding for Spatial Linear Models

In studies that produce data with spatial structure, it is common that covariates of interest vary spatially in addition to the error. Because of this, the error and covariate are often correlated. When this occurs, it is difficult to distinguish the covariate effect from residual spatial variation. In an i.i.d. normal error setting, it is well known that this type of correlation produces biased coefficient estimates, but predictions remain unbiased. In a spatial setting, recent studies have shown that coefficient estimates remain biased, but spatial prediction has not been addressed. The purpose of this paper is to provide a more detailed study of coefficient estimation from spatial models when covariate and error are correlated and then begin a formal study regarding spatial prediction. This is carried out by investigating properties of the generalized least squares estimator and the best linear unbiased predictor when a spatial random effect and a covariate are jointly modelled. Under this setup, we demonstrate that the mean squared prediction error is possibly reduced when covariate and error are correlated.     

Improved multivariate prediction regions for Markov process models

This paper concerns the specification of multivariate prediction regions which may be useful in time series applications whenever we aim at considering not just one single forecast but a group of consecutive forecasts. We review a general result on improved multivariate prediction and we use it in order to calculate conditional prediction intervals for Markov process models so that the associated coverage probability turns out to be close to the target value. This improved solution is asymptotically superior to the estimative one, which is simpler but it may lead to unreliable predictive conclusions. An application to general autoregressive models is presented, focusing in particular on AR and ARCH models.     

Some finite sample properties of generalized ridge regression estimators

In this paper we study the mean square error properties of the generalized ridge estimator. We obtain the exact and the approximate bias and the mean square error of the operational generalized ridge estimator in terms of G( ) functions. We show, among other things, that the operational generalized ridge estimator does not dominate the ordinary least squares estimator up to a certain order of approximation. Finally, we note that the iterative procedures to obtain coverging ridge estimators should be used with caution.     

Weighted-Average Least Squares Prediction

Prediction under model uncertainty is an important and difficult issue. Traditional prediction methods (such as pretesting) are based on model selection followed by prediction in the selected model, but the reported prediction and the reported prediction variance ignore the uncertainty from the selection procedure. This article proposes a weighted-average least squares (WALS) prediction procedure that is not conditional on the selected model. Taking both model and error uncertainty into account, we also propose an appropriate estimate of the variance of the WALS predictor. Correlations among the random errors are explicitly allowed. Compared to other prediction averaging methods, the WALS predictor has important advantages both theoretically and computationally. Simulation studies show that the WALS predictor generally produces lower mean squared prediction errors than its competitors, and that the proposed estimator for the prediction variance performs particularly well when model uncertainty increases.     

AN EXAMINATION OF ESTIMATED RESIDUALS IN A REGRESSION WITH AN INFINITE ORDER PARAMETRIC MODEL

We consider a linear regression with the error term that obeys an autoregressive model of infinite order and estimate parameters of the models. The parameters of the autoregressive model should be estimated based on estimated residuals obtained by means of the method of ordinary least squares, because the errors are unobservable. The consistency of the coefficients, variance and spectral density of the model obeyed by the error term is shown. Further, we estimate the coefficients of the linear regression by means of the method of estimated generalized least squares. We also show the consistency of the estimator.

     

Prediction Techniques for Box–Cox Regression Models

This article reviews several techniques useful for forming point and interval predictions in regression models with Box-Cox transformed variables. The techniques reviewed—plug-in, mean squared error analysis, predictive likelihood, and stochastic simulation—take account of nonnormality and parameter uncertainty in varying degrees. A Monte Carlo study examining their small-sample accuracy indicates that uncertainty about the Box–Cox transformation parameter may be relatively unimportant. For certain parameters, deterministic point predictions are biased, and plug-in prediction intervals are also biased. Stochastic simulation, as usually carried out, leads to badly biased predictions. A modification of the usual approach renders stochastic simulation predictions largely unbiased.     

On the bias and mean square error of the least square estimator in a regression model with two inequality constraints and multivariate t error terms

We derive and numerically evaluate the bias and mean square error of the inequality constrained least squares estimator in a model with two inequality constraints and multivariate terror terms. Our results suggest that qualitatively, the estimator properties found for models with normal errors carry over to the case of multivariate terrors.