Bootstrapping regression models with BLUS residuals

To bootstrap a regression problem, pairs of response and explanatory variables or residuals can be resam‐pled, according to whether we believe that the explanatory variables are random or fixed. In the latter case, different residuals have been proposed in the literature, including the ordinary residuals (Efron 1979), standardized residuals (Bickel & Freedman 1983) and Studentized residuals (Weber 1984). Freedman (1981) has shown that the bootstrap from ordinary residuals is asymptotically valid when the number of cases increases and the number of variables is fixed. Bickel & Freedman (1983) have shown the asymptotic validity for ordinary residuals when the number of variables and the number of cases both increase, provided that the ratio of the two converges to zero at an appropriate rate. In this paper, the authors introduce the use of BLUS (Best Linear Unbiased with Scalar covariance matrix) residuals in bootstrapping regression models. The main advantage of the BLUS residuals, introduced in Theil (1965), is that they are uncorrelated. The main disadvantage is that only n —p residuals can be computed for a regression problem with n cases and p variables. The asymptotic results of Freedman (1981) and Bickel & Freedman (1983) for the ordinary (and standardized) residuals are generalized to the BLUS residuals. A small simulation study shows that even though only n — p residuals are available, in small samples bootstrapping BLUS residuals can be as good as, and sometimes better than, bootstrapping from standardized or Studentized residuals.     

Residuals in the Extended Growth Curve Model

Abstract.  The Extended Growth Curve model is considered. It turns out that the estimated mean of the model is the projection of the observations on the space generated by the design matrices which turns out to be the sum of two tensor product spaces. The orthogonal complement of this space is decomposed into four orthogonal spaces and residuals are defined by projecting the observation matrix on the resulting components. The residuals are interpreted and some remarks are given as to why we should not use ordinary residuals, what kind of information our residuals give and how this information might be used to validate model assumptions and detect outliers and influential observations. It is shown that the residuals are symmetrically distributed around zero and are uncorrelated with each other. The covariance between the residuals and the estimated model as well as the dispersion matrices for the residuals are also given.     

Theory & Methods: Residual Diagnostic Plots for Checking for Model Mis‐specification in Time Series Regression

This paper considers residuals for time series regression. Despite much literature on visual diagnostics for uncorrelated data, there is little on the autocorrelated case. To examine various aspects of the fitted time series regression model, three residuals are considered. The fitted regression model can be checked using orthogonal residuals; the time series error model can be analysed using marginal residuals; and the white noise error component can be tested using conditional residuals. When used together, these residuals allow identification of outliers, model mis‐specification and mean shifts. Due to the sensitivity of conditional residuals to model mis‐specification, it is suggested that the orthogonal and marginal residuals be examined first.     

Some properties of,and relationships among,several uncorrelated and homoscedastic residual vectors

In this paper we examine the properties of four types of residual vectors, arising from fitting a linear regression model to a set of data by least squares. The four types of residuals are (i) the Stepwise residuals (Hedayat and Robson, 1970), (ii) the Recursive residuals (Brown, Durbin, and Evans, 1975), (iii) the Sequentially Adjusted residuals (to be defined herein), and (iv) the BLUS residuals (Theil, 1965, 1971). We also study the relationships among the four residual vectors. It is found that, for any given sequence of observations, (i) the first three sets of residuals are identical, (ii) each of the first three sets, being identical, is a member of Thei’rs (1965, 1971) family of residuals; specifically, they are Linear Unbiased with a Scalar covariance matrix (LUS) but not Best Linear Unbiased with a Scalar covariance matrix (BLUS). We find the explicit form of the transformation matrix and show that the first three sets of residual vectors can be written as an orthogonal transformation of the BLUS residual vector. These and other properties may prove to be useful in the statistical analysis of residuals.     

Simple correspondence analysis using adjusted residuals

Correspondence analysis is a versatile statistical technique that allows the user to graphically identify the association that may exist between variables of a contingency table. For two categorical variables, the classical approach involves applying singular value decomposition to the Pearson residuals of the table. These residuals allow for one to use a simple test to determine those cells that deviate from what is expected under independence. However, the assumptions concerning these residuals are not always satisfied and so such results can lead to questionable conclusions.One may consider instead, an adjustment of the Pearson residual, which is known to have properties associated with the standard normal distribution. This paper explores the application of these adjusted residuals to correspondence analysis and determines how they impact upon the configuration of points in the graphical display.     

On the disribution of residuals form fitted parametric models

Results of a simulation study of the fit of data to an estimated parametric model are reported. Three particular models including the two-parameter normal and exponential distributions, and the simple linear regression model are considered. A number of scaled versions of the least squares residuals from the regression model and quantities that we call residuals from the other two models arc seen follow the parent distribution form loo well. i.e., to be supernormal and superexponential. A point of particular interest is that this tendency does not appear to decrease with increasing sample size, at least for the sample sizes considered here.     

Average run lengths for cusum control charts applied to residuals

A common approach to building control charts for autocorrelated data is to apply classical SPC to the residuals from a time series model of the process. However, Shewhart charts and even CUSUM charts are less sensitive to small shifts in the process mean when applied to residuals than when applied to independent data. Using an approximate analytical model, we show that the average run length of a CUSUM chart for residuals can be reduced substantially by modifying traditional chart design guidelines to account for the degree of autocorrelation in the data.     

Checking the Grouped Data Version of the Cox Model for Interval-grouped Survival Data

Abstract.  Epidemiology research often entails the analysis of failure times subject to grouping. In large cohorts interval grouping also offers a feasible choice of data reduction to actually facilitate an analysis of the data. Based on an underlying Cox proportional hazards model for the exact failure times one may deduce a grouped data version of this model which may then be used to analyse the data. The model bears a lot of resemblance to a generalized linear model, yet due to the nature of data one also needs to incorporate censoring. In the case of non-trivial censoring this precludes model checking procedures based on ordinary residuals as calculation of these requires knowledge of the censoring distribution. In this paper, we represent interval grouped data in a dynamical way using a counting process approach. This enables us to identify martingale residuals which can be computed without knowledge of the censoring distribution. We use these residuals to construct graphical as well as numerical model checking procedures. An example from epidemiology is provided.     

The Performance of Randomization Tests that Use Permutations of Independent Variables

ABSTRACT

In this article we evaluate the performance of a randomization test for a subset of regression coefficients in a linear model. This randomization test is based on random permutations of the independent variables. It is shown that the method maintains its level of significance, except for extreme situations, and has power that approximates the power of another randomization test, which is based on the permutation of residuals from the reduced model. We also show, via an example, that the method of permuting independent variables is more valuable than other randomization methods because it can be used in connection with the downweighting of outliers.     

RECURSIVE ESTIMATION OF THE GENERAL LINEAR MODEL WITH DEPENDENT ERRORS AND MULTIPLE ADDITIONAL OBSERVATIONS1

The techniques for recursive estimation of the general linear model with dependent errors and known second order properties, is generalised to allow for simultaneous addition of an arbitrary number of additional observations. Computational formulae for recursive updating of parameter estimates are derived, together with a sequence of univariate recursive residuals for testing the constancy of the regression relation over time.     

Checking the linear transformation model for clustered failure time observations

The linear transformation model is a semiparametric model which contains the Cox proportional hazards model and the proportional odds model as special cases. Cai et al. (Biometrika 87:867-878, 2000) have proposed an inference procedure for the linear transformation model with correlated censored observations. In this article, we develop formal and graphical model checking techniques for the linear transformation models based on cumulative sums of martingale-type residuals. The proposed method is illustrated with a clinical trial data.     

An evaluation of bootstrap methods for outlier detection in least squares regression

Outlier detection is a critical part of data analysis, and the use of Studentized residuals from regression models fit using least squares is a very common approach to identifying discordant observations in linear regression problems. In this paper we propose a bootstrap approach to constructing critical points for use in outlier detection in the context of least-squares Studentized residuals, and find that this approach allows naturally for mild departures in model assumptions such as non-Normal error distributions. We illustrate our methodology through both a real data example and simulated data.     

Influence diagnostics in the Gamma regression model with adjusted deviance residuals

In this study, we develop the adjusted deviance residuals for the gamma regression model (GRM) by following Cordeiro's (2004) method. These adjusted deviance residuals under the GRM are used for influence diagnostics. A comparative analysis has been sorted out between our proposed method of the adjusted deviance residuals and an existing method for influence diagnostics. These results are illustrated by a simulation study and using a real data set. They are presented for different values of dispersion and sample sizes and indicate the significant role of the GRM inferences.     

Comments on “bootstrapping time series models”

We take issue with the main suggestion in Li and Maddala (LiMa) that bootstrapping residuals is always the preferred approach and question some of their guidelines. We show that it can be potentially misleading to mimic the autocovariance structure of residuals, since it can be very different from that of true errors. We emphasize that the residuals are sensitive to model misspecification and generally not a part of the information set. We make constructive suggestions and propose a semiparametric method.     

Diagnostic Measures for Generalized Linear Models with Missing Covariates

Abstract.  In this paper, we carry out an in-depth investigation of diagnostic measures for assessing the influence of observations and model misspecification in the presence of missing covariate data for generalized linear models. Our diagnostic measures include case-deletion measures and conditional residuals. We use the conditional residuals to construct goodness-of-fit statistics for testing possible misspecifications in model assumptions, including the sampling distribution. We develop specific strategies for incorporating missing data into goodness-of-fit statistics in order to increase the power of detecting model misspecification. A resampling method is proposed to approximate the p -value of the goodness-of-fit statistics. Simulation studies are conducted to evaluate our methods and a real data set is analysed to illustrate the use of our various diagnostic measures.     

Iterated residuals and time-varying covariate effects in Cox regression

Summary. The Cox proportional hazards model, which is widely used for the analysis of treatment and prognostic effects with censored survival data, makes the assumption that the hazard ratio is constant over time. Nonparametric estimators have been developed for an extended model in which the hazard ratio is allowed to change over time. Estimators based on residuals are appealing as they are easy to use and relate in a simple way to the more restricted Cox model estimator. After fitting a Cox model and calculating the residuals, one can obtain a crude estimate of the time-varying coefficients by adding a smooth of the residuals to the initial (constant) estimate. Treating the crude estimate as the fit, one can re-estimate the residuals. Iteration leads to consistent estimation of the nonparametric time-varying coefficients. This approach leads to clear guidelines for residual analysis in applications. The results are illustrated by an analysis of the Medical Research Council's myeloma trials, and by simulation.     

A set of independent sequential residuals for the multivariate regression model

In this paper a set of residuals for the multivariate linear regression model is introduced. These residuals are shown to be independent with known distributions which do not depend on the parameters of the model. Transformations of the mentioned residuals may be used to construct exact α goodness-of-fit tests for the multivariate regression model.     

A geometric characterization of optimal designs for regression models with correlated observations

We consider the problem of optimal design of experiments for random effects models, especially population models, where a small number of correlated observations can be taken on each individual, while the observations corresponding to different individuals are assumed to be uncorrelated. We focus on c-optimal design problems and show that the classical equivalence theorem and the famous geometric characterization of Elfving (1952) from the case of uncorrelated data can be adapted to the problem of selecting optimal sets of observations for the n individual patients. The theory is demonstrated by finding optimal designs for a linear model with correlated observations and a nonlinear random effects population model, which is commonly used in pharmacokinetics.     

Recursive residuals in generalised linear models

Recursive estimation and recursive residuals are introduced for generalised linear models (GLIM). Their definitions parallel those of normal theory regression models and relate to one of the outlier model definitions of GLIM residuals. An example illustrates their use.