Nonparametric multistep-ahead prediction in time series analysis

Summary.  We consider the problem of multistep-ahead prediction in time series analysis by using nonparametric smoothing techniques. Forecasting is always one of the main objectives in time series analysis. Research has shown that non-linear time series models have certain advantages in multistep-ahead forecasting. Traditionally, nonparametric k -step-ahead least squares prediction for non-linear autoregressive AR( d ) models is done by estimating E ( X _{t + k}  | X _t , …,  X _{t − d +1}) via nonparametric smoothing of X _{t + k} on ( X _t , …,  X _{t − d +1}) directly. We propose a multistage nonparametric predictor. We show that the new predictor has smaller asymptotic mean-squared error than the direct smoother, though the convergence rate is the same. Hence, the predictor proposed is more efficient. Some simulation results, advice for practical bandwidth selection and a real data example are provided.     

Shrinkage Structure of Partial Least Squares

Partial least squares regression (PLS) is one method to estimate parameters in a linear model when predictor variables are nearly collinear. One way to characterize PLS is in terms of the scaling (shrinkage or expansion) along each eigenvector of the predictor correlation matrix. This characterization is useful in providing a link between PLS and other shrinkage estimators, such as principal components regression (PCR) and ridge regression (RR), thus facilitating a direct comparison of PLS with these methods. This paper gives a detailed analysis of the shrinkage structure of PLS, and several new results are presented regarding the nature and extent of shrinkage.     

Relative efficiency of first difference estimator in panel data regression with serially correlated error components

We consider the pooled cross-sectional and time series regression model when the disturbances follow a serially correlated one-way error components. In this context we discovered that the first difference estimator for the regression coefficients is equivalent to the generalized least squares estimator irrespective of the particular form of the regressor matrix when the disturbances are generated by a first order autoregressive process where the autocorrelation is close to unity.     

Shrinkage estimation for linear regression with ARMA errors

In this paper, we extend the modified lasso of Wang et al. (2007) to the linear regression model with autoregressive moving average (ARMA) errors. Such an extension is far from trivial because new devices need to be called for to establish the asymptotics due to the existence of the moving average component. A shrinkage procedure is proposed to simultaneously estimate the parameters and select the informative variables in the regression, autoregressive, and moving average components. We show that the resulting estimator is consistent in both parameter estimation and variable selection, and enjoys the oracle properties. To overcome the complexity in numerical computation caused by the existence of the moving average component, we propose a procedure based on a least squares approximation to implement estimation. The ordinary least squares formulation with the use of the modified lasso makes the computation very efficient. Simulation studies are conducted to evaluate the finite sample performance of the procedure. An empirical example of ground-level ozone is also provided.     

Empirical assessment of the applicability of shrinkage predictors in regression when past and future data differ

Jones and Copas (1986) present theoretical and simulation results on the relative merits of a Stein predictor (Copas, 1983) and the ordinary least squares predictor in the usual linear multiple regression model, when certain distributional properties of the regressor variables arising in the past differ from those for which predictions are to be made. Here, extension is made to the practical situation where the true regression parameters are unknown. A hypothesis testing procedure is developed to help determine which of shrinkage and least squares is preferable in any given instance. This approach is applied to explain some empirical evidence on the comparative merits of the two procedures, recently given by Berk (1984).     

The peculiar shrinkage properties of partial least squares regression

Partial least squares regression has been widely adopted within some areas as a useful alternative to ordinary least squares regression in the manner of other shrinkage methods such as principal components regression and ridge regression. In this paper we examine the nature of this shrinkage and demonstrate that partial least squares regression exhibits some undesirable properties.     

Bent‐cable regression with autoregressive noise

Motivated by time series of atmospheric concentrations of certain pollutants the authors develop bent‐cable regression for autocorrelated errors. Bent‐cable regression extends the popular piecewise linear (broken‐stick) model, allowing for a smooth change region of any non‐negative width. Here the authors consider autoregressive noise added to a bent‐cable mean structure, with unknown regression and time series parameters. They develop asymptotic theory for conditional least‐squares estimation in a triangular array framework, wherein each segment of the bent cable contains an increasing number of observations while the autoregressive order remains constant as the sample size grows. They explore the theory in a simulation study, develop implementation details, apply the methodology to the motivating pollutant dataset, and provide a scientific interpretation of the bent‐cable change point not discussed previously. The Canadian Journal of Statistics 38: 386–407; 2010 © 2010 Statistical Society of Canada     

Recursive Estimation of GARCH Models

This article develops three recursive on-line algorithms, based on a two-stage least squares scheme for estimating generalized autoregressive conditionally heteroskedastic (GARCH) models. The first one, denoted by 2S-RLS, is an adaptation of the recursive least squares method for estimating autoregressive conditionally heteroskedastic (ARCH) models. The second and the third ones (denoted, respectively, by 2S-PLR and 2S-RML) are adapted versions of the pseudolinear regression (PLR) and the recursive maximum likelihood (RML) methods to the GARCH case. We show that the proposed algorithms give consistent estimators and that the 2S-RLS and the 2S-RML estimators are asymptotically Gaussian. These methods seem very adequate for modeling the sequential feature of financial time series, which are observed on a high-frequency basis. The performance of these algorithms is shown via a simulation study.     

On shrinkage least squares estimation in a parallelism problem

In a multi-sample simple regression model, generally, homogeneity of the regression slopes leads to improved estimation of the intercepts. Analogous to the preliminary test estimators, (smooth) shrinkage least squares estimators of Intercepts based on the James-Stein rule on regression slopes are considered. Relative pictures on the (asymptotic) risk of the classical, preliminary test and the shrinkage least squares estimators are also presented. None of the preliminary test and shrinkage least squares estimators may dominate over the other, though each of them fares well relative to the other estimators.     

Estimation Bias in the First-Order Autoregressive Model and Its Impact on Predictions and Prediction Intervals

The least squares estimate of the autoregressive coefficient in the AR(1) model is known to be biased towards zero, especially for parameters close to the stationarity boundary. Several methods for correcting the autoregressive parameter estimate for the bias have been suggested. Using simulations, we study the bias and the mean square error of the least squares estimate and the bias-corrections proposed by Kendall and Quenouille.

We also study the mean square forecast error and the coverage of the 95% prediction interval when using the biased least squares estimate or one of its bias-corrected versions. We find that the estimation bias matters little for point forecasts, but that it affects the coverage of the prediction intervals. Prediction intervals for forecasts more than one step ahead, when calculated with the biased least squares estimate, are too narrow.     

A recursive approach to parameter estimation in regression and time series models

In this paper we discuss the recursive (or on line) estimation in (i) regression and (ii) autoregressive integrated moving average (ARIMA) time series models. The adopted approach uses Kalman filtering techniques to calculate estimates recursively. This approach is used for the estimation of constant as well as time varying parameters. In the first section of the paper we consider the linear regression model. We discuss recursive estimation both for constant and time varying parameters. For constant parameters, Kalman filtering specializes to recursive least squares. In general, we allow the parameters to vary according to an autoregressive integrated moving average process and update the parameter estimates recursively. Since the stochastic model for the parameter changes will "be rarely known, simplifying assumptions have to be made. In particular we assume a random walk model for the time varying parameters and show how to determine whether the parameters are changing over time. This is illustrated with an example.     

Comparison of prediction methods for multicollinear data

In this paper we discuss the partial least squares (PLS) prediction method. The method is compared to the predictor based on principal component regression (PCR). Both theoretical considerations and computations on artificial and real data are presented.     

Variable Selection via Regression Trees in the Presence of Irrelevant Variables

Many tree algorithms have been developed for regression problems. Although they are regarded as good algorithms, most of them suffer from loss of prediction accuracy when there are many irrelevant variables and the number of predictors exceeds the number of observations. We propose the multistep regression tree with adaptive variable selection to handle this problem. The variable selection step and the fitting step comprise the multistep method.

The multistep generalized unbiased interaction detection and estimation (GUIDE) with adaptive forward selection (fg) algorithm, as a variable selection tool, performs better than some of the well-known variable selection algorithms such as efficacy adaptive regression tube hunting (EARTH), FSR (false selection rate), LSCV (least squares cross-validation), and LASSO (least absolute shrinkage and selection operator) for the regression problem. The results based on simulation study show that fg outperforms other algorithms in terms of selection result and computation time. It generally selects the important variables correctly with relatively few irrelevant variables, which gives good prediction accuracy with less computation time.     

Testing for parameter stability in RCA(1) time series

We utilize strong invariance principles to construct tests for the stability of model parameters determining a random coefficient autoregressive time series of order one. The test statistics are based on (conditional) least squares estimators for the unknown parameters.     

Penalized regression models with autoregressive error terms

Penalized regression methods have recently gained enormous attention in statistics and the field of machine learning due to their ability of reducing the prediction error and identifying important variables at the same time. Numerous studies have been conducted for penalized regression, but most of them are limited to the case when the data are independently observed. In this paper, we study a variable selection problem in penalized regression models with autoregressive (AR) error terms. We consider three estimators, adaptive least absolute shrinkage and selection operator, bridge, and smoothly clipped absolute deviation, and propose a computational algorithm that enables us to select a relevant set of variables and also the order of AR error terms simultaneously. In addition, we provide their asymptotic properties such as consistency, selection consistency, and asymptotic normality. The performances of the three estimators are compared with one another using simulated and real examples.     

AN EXAMINATION OF ESTIMATED RESIDUALS IN A REGRESSION WITH AN INFINITE ORDER PARAMETRIC MODEL

We consider a linear regression with the error term that obeys an autoregressive model of infinite order and estimate parameters of the models. The parameters of the autoregressive model should be estimated based on estimated residuals obtained by means of the method of ordinary least squares, because the errors are unobservable. The consistency of the coefficients, variance and spectral density of the model obeyed by the error term is shown. Further, we estimate the coefficients of the linear regression by means of the method of estimated generalized least squares. We also show the consistency of the estimator.

     

Tree-structured generalized autoregressive conditional heteroscedastic models

We propose a new generalized autoregressive conditional heteroscedastic (GARCH) model with tree-structured multiple thresholds for the estimation of volatility in financial time series. The approach relies on the idea of a binary tree where every terminal node parameterizes a (local) GARCH model for a partition cell of the predictor space. The fitting of such trees is constructed within the likelihood framework for non-Gaussian observations: it is very different from the well-known regression tree procedure which is based on residual sums of squares. Our strategy includes the classical GARCH model as a special case and allows us to increase model complexity in a systematic and flexible way. We derive a consistency result and conclude from simulation and real data analysis that the new method has better predictive potential than other approaches.     

On small sample properties of the mixed regression predictor under misspecification

This paper relaxes the Mittelhammer's (1981) assumption that the value of the true variance is known in the mixed regression model and examines the small sample, properties of the feasible mixed regression predictor under misspecification. The paper shows that the feasible mixed regression predictor is not always superior to the ordinary least squares predictor in terms of the weak mean square error when there exist omitted variables in the model. Further it shows that misspecificstion works favorably for the ordinary least squares predictor.     

Penalized weighted composite quantile regression in the linear regression model with heavy-tailed autocorrelated errors

In this paper, a penalized weighted composite quantile regression estimation procedure is proposed to estimate unknown regression parameters and autoregression coefficients in the linear regression model with heavy-tailed autoregressive errors. Under some conditions, we show that the proposed estimator possesses the oracle properties. In addition, we introduce an iterative algorithm to achieve the proposed optimization problem, and use a data-driven method to choose the tuning parameters. Simulation studies demonstrate that the proposed new estimation method is robust and works much better than the least squares based method when there are outliers in the dataset or the autoregressive error distribution follows heavy-tailed distributions. Moreover, the proposed estimator works comparably to the least squares based estimator when there are no outliers and the error is normal. Finally, we apply the proposed methodology to analyze the electricity demand dataset.     

A Strongly Consistent Criterion to Decide Between I(1) and I(0) Processes Based on Different Convergence Rates

The usual procedure to determine whether a univariate time series is stationary or first-difference stationary is to perform some unit root test. In this article, an alternative methodology is presented that leads to a strongly consistent two-step criterion to estimate the number of unit roots. The criterion is based on estimating some autoregressive polynomials using regression procedures and exploiting the fact that the nonstationary roots converge at a faster rate than the stationary ones. The proposed procedure requires at most four regressions and is easy to implement. A simulation study demonstrates that it can perform significantly better in practice than the Dickey–Fuller and the generalized least squares (GLS)-detrended Dickey–Fuller tests.