Robust multivariate diagnostics for PLSR and application on high dimensional spectrally overlapped drug systems

ABSTRACT

Statistical methods are effectively used in the evaluation of pharmaceutical formulations instead of laborious liquid chromatography. However, signal overlapping, nonlinearity, multicollinearity and presence of outliers deteriorate the performance of statistical methods. The Partial Least Squares Regression (PLSR) is a very popular method in the quantification of high dimensional spectrally overlapped drug formulations. The SIMPLS is the mostly used PLSR algorithm, but it is highly sensitive to outliers that also effect the diagnostics. In this paper, we propose new robust multivariate diagnostics to identify outliers, influential observations and points causing non-normality for a PLSR model. We study performances of the proposed diagnostics on two everyday use highly overlapping drug systems: Paracetamol–Caffeine and Doxylamine Succinate–Pyridoxine Hydrochloride.     

Study of partial least squares and ridge regression methods

This article considers both Partial Least Squares (PLS) and Ridge Regression (RR) methods to combat multicollinearity problem. A simulation study has been conducted to compare their performances with respect to Ordinary Least Squares (OLS). With varying degrees of multicollinearity, it is found that both, PLS and RR, estimators produce significant reductions in the Mean Square Error (MSE) and Prediction Mean Square Error (PMSE) over OLS. However, from the simulation study it is evident that the RR performs better when the error variance is large and the PLS estimator achieves its best results when the model includes more variables. However, the advantage of the ridge regression method over PLS is that it can provide the 95% confidence interval for the regression coefficients while PLS cannot.     

Modified Wynn's Sequential Algorithm for Constructing D-Optimal Designs: Adding Two Points at a Time

Partial least squares (PLS) is a class of methods for modeling relations between sets of observed variables by using the latent components where the predictors are highly collinear. SIMPLS is a commonly used PLS algorithm that calculates the latent components directly as linear combinations of the original variables. However, SIMPLS is known to be very sensible to outliers since it is based on the empirical cross-covariance matrix. RoPLS is a recently proposed iterative method for robust SIMPLS. In this article, the influence function for the RoPLS coefficient estimator is derived. It is demonstrated that under certain conditions, the RoPLS estimator has infinitesimal robustness.     

A Novel Regression Approach: Least Squares Ratio

Regression Analysis (RA) is one of the frequently used tool for forecasting. The Ordinary Least Squares (OLS) Technique is the basic instrument of RA and there are many regression techniques based on OLS. This paper includes a new regression approach, called Least Squares Ratio (LSR), and comparison of OLS and LSR according to mean square errors of estimation of theoretical regression parameters (mse ß) and dependent value (mse y).     

Shrinkage Structure of Partial Least Squares

Partial least squares regression (PLS) is one method to estimate parameters in a linear model when predictor variables are nearly collinear. One way to characterize PLS is in terms of the scaling (shrinkage or expansion) along each eigenvector of the predictor correlation matrix. This characterization is useful in providing a link between PLS and other shrinkage estimators, such as principal components regression (PCR) and ridge regression (RR), thus facilitating a direct comparison of PLS with these methods. This paper gives a detailed analysis of the shrinkage structure of PLS, and several new results are presented regarding the nature and extent of shrinkage.     

A comparison of various methods for multivariate regression with highly collinear variables

Regression tends to give very unstable and unreliable regression weights when predictors are highly collinear. Several methods have been proposed to counter this problem. A subset of these do so by finding components that summarize the information in the predictors and the criterion variables. The present paper compares six such methods (two of which are almost completely new) to ordinary regression: Partial least Squares (PLS), Principal Component regression (PCR), Principle covariates regression, reduced rank regression, and two variants of what is called power regression. The comparison is mainly done by means of a series of simulation studies, in which data are constructed in various ways, with different degrees of collinearity and noise, and the methods are compared in terms of their capability of recovering the population regression weights, as well as their prediction quality for the complete population. It turns out that recovery of regression weights in situations with collinearity is often very poor by all methods, unless the regression weights lie in the subspace spanning the first few principal components of the predictor variables. In those cases, typically PLS and PCR give the best recoveries of regression weights. The picture is inconclusive, however, because, especially in the study with more real life like simulated data, PLS and PCR gave the poorest recoveries of regression weights in conditions with relatively low noise and collinearity. It seems that PLS and PCR are particularly indicated in cases with much collinearity, whereas in other cases it is better to use ordinary regression. As far as prediction is concerned: Prediction suffers far less from collinearity than recovery of the regression weights.     

Local linear regression with adaptive orthogonal fitting for the wind power application

Short-term forecasting of wind generation requires a model of the function for the conversion of meteorological variables (mainly wind speed) to power production. Such a power curve is nonlinear and bounded, in addition to being nonstationary. Local linear regression is an appealing nonparametric approach for power curve estimation, for which the model coefficients can be tracked with recursive Least Squares (LS) methods. This may lead to an inaccurate estimate of the true power curve, owing to the assumption that a noise component is present on the response variable axis only. Therefore, this assumption is relaxed here, by describing a local linear regression with orthogonal fit. Local linear coefficients are defined as those which minimize a weighted Total Least Squares (TLS) criterion. An adaptive estimation method is introduced in order to accommodate nonstationarity. This has the additional benefit of lowering the computational costs of updating local coefficients every time new observations become available. The estimation method is based on tracking the left-most eigenvector of the augmented covariance matrix. A robustification of the estimation method is also proposed. Simulations on semi-artificial datasets (for which the true power curve is available) underline the properties of the proposed regression and related estimation methods. An important result is the significantly higher ability of local polynomial regression with orthogonal fit to accurately approximate the target regression, even though it may hardly be visible when calculating error criteria against corrupted data.     

A NOTE ON ESTIMATING LINEAR TREND IN A REGRESSION MODEL WITH SERIALLY CORRELATED ERROR COMPONENTS

ABSTRACT

We consider a linear trend regression model when the disturbances follow a serially correlated one-way error component model. In this model, we investigate the performance of the Ordinary Least Squares Esitmator (OLSE), First Difference Estimator (FDE), Generalized Least Squares Estimator (GLSE) and the Cochrane-Orcutt-Transformation Estimator (COTE) of slope coefficient in terms of efficiency. The main findings are as follows: (1) when the autocorrelation is close to unity, then the FDE is approximately the GLSE; (2) the OLSE is better than the COTE; and (3) when the value of the autocorrelation is kept constant and T → ∞, the OLSE, COTE and GLSE are asymptotically equivalent whereas the FDE is worse than the other estimators in terms of efficiency.     

Bias in the ordinary least squares estimator in the dynamic linear regression model with autocorrelated disturbances

We consider the bias in the Ordinary Least Squares estimator in the linear regression model with a lagged dependent variable as regressor. Results are obtained with independent and auto-correlated disturbances. Asymptotic results are obtained analytically, and finite sample results based on a Monte Carlo study. The substantial biases found suggest the need for an alternative estimator to Ordinary Least Squares and powerful tests for autocorrelated disturbances in the dynamic model.     

On the asymptotic bias of OLS in dynamic regression models with autocorrelated errors

It is well-known that Ordinary Least Squares (OLS) yields inconsistent estimates if applied to a regression equation with lagged dependent variables and correlated errors. Bias expressions which appear in the literature usually assume the exogenous variables to be non-stochastic. Due to this assumption the numerical sizes of these expressions cannot be determined. Further, the analysis is mostly restricted to very simple models. In this paper the problem of calculating the asymptotic bias of OLS is generalized to stationary dynamic regression models, where the errors follow a stationary ARMA process. A general bias expression is derived and a method is introduced by which its actual size can be computed numerically.     

中国货币需求波动研究——广义可塑最小二乘法及其推广

中国货币需求波动研究——广义可塑最小二乘法及其推广刘勤顾岚ＡＢＳＴＲＡＣＴＡｎｅｗｍｅｔｈｏｄｏｆｐａｒａｍｅｔｅｒｓｅｓｔｉｍａｔｉｏｎ－ＧｅｎｅｒａｌｉｚｅｄＦｌｅｘｉｂｌｅＬｅａｓｔＳｑｕａｒｅｓ（ＧＦＬＳ）ｗａｓｉｎｔｒｏｄｕｃｅｄ,ｔｈｅｎ...     

Implementing partial least squares

Partial least squares (PLS) regression has been proposed as an alternative regression technique to more traditional approaches such as principal components regression and ridge regression. A number of algorithms have appeared in the literature which have been shown to be equivalent. Someone wishing to implement PLS regression in a programming language or within a statistical package must choose which algorithm to use. We investigate the implementation of univariate PLS algorithms within FORTRAN and the Matlab (1993) and Splus (1992) environments, comparing theoretical measures of execution speed based on flop counts with their observed execution times. We also comment on the ease with which the algorithms may be implemented in the different environments. Finally, we investigate the merits of using the orthogonal invariance of PLS regression to improve the algorithms.     

Efficiency analysis of ten estimation procedures for quantitative linear models with autocorrelated errors

Many estimation procedures for quantitative linear models with autocorrelated errors have been proposed in the literature. A number of these procedures have been compared in various ways for different sample sizes and autocorrelation parameters values and for structured or random explanatory vaiables. In this paper, we revisit three situations that were considered to some extent in previous studies, by comparing ten estimation procedures: Ordinary Least Squares (OLS), Generalized Least Squares (GLS), estimated Generalized Least Squares (six procedures), Maximum Likelihood (ML), and First Differences (FD). The six estimated GLS procedures and the ML procedure differ in the way the error autocovariance matrix is estimated. The three situations can be defined as follows: Case 1, the explanatory variable x in the simple linear regression is fixed; Case 2,x is purely random; and Case 3x is first-order autoregressive. Following a theoretical presentation, the ten estimation procedures are compared in a Monte Carlo study conducted in the time domain, where the errors are first-order autoregressive in Cases 1-3. The measure of comparison for the estimation procedures is their efficiency relative to OLS. It is evaluated as a function of the time series length and the magnitude and sign of the error autocorrelation parameter. Overall, knowledge of the model of the time series process generating the errors enhances efficiency in estimated GLS. Differences in the efficiency of estimation procedures between Case 1 and Cases 2 and 3 as well as differences in efficiency among procedures in a given situation are observed and discussed.     

A MORE GENERAL CRITERION FOR SUBSET SELECTION IN MULTIPLE LINEAR REGRESSION

ABSTRACT

In this article, we propose a more general criterion called S_p -criterion, for subset selection in the multiple linear regression Model. Many subset selection methods are based on the Least Squares (LS) estimator of β, but whenever the data contain an influential observation or the distribution of the error variable deviates from normality, the LS estimator performs ‘poorly’ and hence a method based on this estimator (for example, Mallows’ C_p -criterion) tends to select a ‘wrong’ subset. The proposed method overcomes this drawback and its main feature is that it can be used with any type of estimator (either the LS estimator or any robust estimator) of β without any need for modification of the proposed criterion. Moreover, this technique is operationally simple to implement as compared to other existing criteria. The method is illustrated with examples.     

门槛面板单位根模型及中国区域经济增长的非线性

基于面板门槛单位根模型,利用可行广义最小二乘法（KH5）,运用bootstrap模拟计算临界值的方法,从新的视角研究我国区域经济发展趋势特征。研究结论表明：中国区域经济增长呈现出显著非线性转换特征,经济增长率在收敛的方式上存在着显著的门槛效应。第一组的部分区域在区制I中（小于门槛值1．81）表现绝对收敛的转移动态,反之,则表现为条件收敛;第二组以及第三组中的部分区域在其所对应的区制I均表现为绝对收敛,而在区制Ⅱ中则为发散的。     

A non stochastic ridge regression estimator and comparison with the James-Stein estimator

Abstract

This article presents a non-stochastic version of the Generalized Ridge Regression estimator that arises from a discussion of the properties of a Generalized Ridge Regression estimator whose shrinkage parameters are found to be close to their upper bounds. The resulting estimator takes the form of a shrinkage estimator that is superior to both the Ordinary Least Squares estimator and the James-Stein estimator under certain conditions. A numerical study is provided to investigate the range of signal to noise ratio under which the new estimator dominates the James-Stein estimator with respect to the prediction mean square error.     

Spurious Instrumental Variables

Spurious regression phenomenon has been recognized for a wide range of Data Generating Processes: driftless unit roots, unit roots with drift, long memory, trend and broken-trend stationarity, etc. The usual framework is Ordinary Least Squares. We show that the spurious phenomenon also occurs in Instrumental Variables estimation when using non stationary variables, whether the non stationarity component is stochastic or deterministic. Finite sample evidence supports the asymptotic results.     

The construction of a partial least-squares biplot

Biplots are useful tools to explore the relationship among variables. In this paper, the specific regression relationship between a set of predictors X and set of response variables Y by means of partial least-squares (PLS) regression is represented. The PLS biplot provides a single graphical representation of the samples together with the predictor and response variables, as well as their interrelationships in terms of the matrix of regression coefficients.     

Generalized mean squared error properties of regression estimators

Theobald (1974) compares Ordinary Least Squares and Ridge Regression estimators of regression parameters using a generalized mean squared error criterion. This paper presents the generalized mean squared error of a Principal Components Regression estimator and comparisons are made with each of the above estimators. In general the choice of which estimator to use depends on the magnitude and the orientation of the unknown parameter vector.     

Two stage least squares estimation in structural cointegration models

This paper considers single-equation estimation of simultaneous equation models with integrated processes. The aim of the paper is to investigate asymptotic and small sample properties of some estimators in this framework. We deal with two groups of estimators: such that originally were designated for reduced form estimation and such for simultaneous equation models. In the first group we deal with Least Squares and Fully Modified Least Squares. The second group comprises Two Stage Least Squares and two modifications of it. The asymptotic analysis in section 2 shows that it is true that all estimators are super-consistent in this context but in principle, only the methods of the second group enable valid inference. Section 3 presents the results of a simulation study which is designed for specific problems of simultaneous equation models. This paper was presented at the European Meeting of the Econometric Society in Istanbul, 1996. The author thanks an anonymous referee for helpful suggestions.