A superiority problem in misspecified restricted linear models

Abstract

In this article we study the relationship between principal component analysis and a multivariate dependency measure. It is shown, via simulated examples and real data, that the information provided by principal components is compatible with that obtained via the dependency measure δ. Furthermore, we show that in some instances in which principal component analysis fails to give reasonable results due to nonlinearity among the random variables, the dependency statistic δ still provides good results. Finally, we give some ideas about using the statistic δ in order to reduce the dimensionality of a given data set.     

Interpretable dimension reduction

The analysis of high-dimensional data often begins with the identification of lower dimensional subspaces. Principal component analysis is a dimension reduction technique that identifies linear combinations of variables along which most variation occurs or which best “reconstruct” the original variables. For example, many temperature readings may be taken in a production process when in fact there are just a few underlying variables driving the process. A problem with principal components is that the linear combinations can seem quite arbitrary. To make them more interpretable, we introduce two classes of constraints. In the first, coefficients are constrained to equal a small number of values (homogeneity constraint). The second constraint attempts to set as many coefficients to zero as possible (sparsity constraint). The resultant interpretable directions are either calculated to be close to the original principal component directions, or calculated in a stepwise manner that may make the components more orthogonal. A small dataset on characteristics of cars is used to introduce the techniques. A more substantial data mining application is also given, illustrating the ability of the procedure to scale to a very large number of variables.     

Principal components analysis of nonstationary time series data

The effect of nonstationarity in time series columns of input data in principal components analysis is examined. Nonstationarity are very common among economic indicators collected over time. They are subsequently summarized into fewer indices for purposes of monitoring. Due to the simultaneous drifting of the nonstationary time series usually caused by the trend, the first component averages all the variables without necessarily reducing dimensionality. Sparse principal components analysis can be used, but attainment of sparsity among the loadings (hence, dimension-reduction is achieved) is influenced by the choice of parameter(s) (λ _1,i ). Simulated data with more variables than the number of observations and with different patterns of cross-correlations and autocorrelations were used to illustrate the advantages of sparse principal components analysis over ordinary principal components analysis. Sparse component loadings for nonstationary time series data can be achieved provided that appropriate values of λ _1,j are used. We provide the range of values of λ _1,j that will ensure convergence of the sparse principal components algorithm and consequently achieve sparsity of component loadings.     

Optimal Whitening and Decorrelation

Whitening, or sphering, is a common preprocessing step in statistical analysis to transform random variables to orthogonality. However, due to rotational freedom there are infinitely many possible whitening procedures. Consequently, there is a diverse range of sphering methods in use, for example, based on principal component analysis (PCA), Cholesky matrix decomposition, and zero-phase component analysis (ZCA), among others. Here, we provide an overview of the underlying theory and discuss five natural whitening procedures. Subsequently, we demonstrate that investigating the cross-covariance and the cross-correlation matrix between sphered and original variables allows to break the rotational invariance and to identify optimal whitening transformations. As a result we recommend two particular approaches: ZCA-cor whitening to produce sphered variables that are maximally similar to the original variables, and PCA-cor whitening to obtain sphered variables that maximally compress the original variables.     

CONDITIONING AND CONCENTRATION OF PRINCIPAL COMPONENTS

This paper uses various gauges to construct principal variables that satisfy criteria of maximal scatter. The solutions coincide with Hotelling's (1933) principal components in structured ensembles and mixtures, including heavy-tailed distributions not having moments. Thus, normal-theory tests are exact in level and power under nonstandard models allowing for correlated vector observations and for certain mixtures having neither moments nor unimodal marginals.     

Continuity and Analysis of Sequences of Principal Components

Double arrays of n rows and p columns can be regarded as n drawings from some p-dimensional population. A sequence of such arrays is considered. Principal component analysis for each array forms sequences of sample principal components and eigenvalues. The continuity of these sequences, in the sense of convergence with probability one and convergence in probability, is investigated, that appears to be informative for pattern study and prediction of principal components. Various features of paths of sequences of population principal components are highlighted through an example.     

A Procedure for Identification of Principal Variables by Least Generalized Dependence

Principal components are often used for reducing dimensions in multivariate data, but they frequently fail to provide useful results and their interpretation is rather difficult. In this article, the use of entropy optimization principles for dimensional reduction in multivariate data is proposed. Under the assumptions of multivariate normality, a four-step procedure is developed for selecting principal variables and hence discarding redundant variables. For comparative performance of the information theoretic procedure, we use simulated data with known dimensionality. Principal variables of cluster bean (Guar) are identified by applying this procedure to a real data set generated in a plant breeding experiment.     

A Note on Unwanted Variance in Exploratory Factor Models

One strategy of exploratory factor analysis is to decide on the number of factors to extract by means of the eigenvalues of an initial principal component analysis. The present article proves that there is a non zero covariance of the factors with the components rejected when the number of factors to extract is determined by means of principal components analysis. Thus, some of the variance declared as irrelevant or unwanted in an initial principal component analysis is again part of the final factor model.     

析主成分分析和因子分析——以Frobenius范数下矩阵近似的新视角

本文通过引入数据阵在Frobenius范数下的最优近似等概念来重新探讨主成分和因子分析。我发现，主成分分析中主成分和因子分析中因子得分（通过主成分解因子载荷，然后用最小二乘解因子得分）的估计为数据阵的最优近似（在Frobenius范数下）在不同正交坐标方向矩阵下的坐标。两种方法分别采用了不同的约束条件分解的最优近似（在Frobenius范数下），因为该分解并不唯一。     

Clustering of Variables Around Latent Components

Abstract

Clustering of variables around latent components is investigated as a means to organize multivariate data into meaningful structures. The coverage includes (i) the case where it is desirable to lump together correlated variables no matter whether the correlation coefficient is positive or negative; (ii) the case where negative correlation shows high disagreement among variables; (iii) an extension of the clustering techniques which makes it possible to explain the clustering of variables taking account of external data. The strategy basically consists in performing a hierarchical cluster analysis, followed by a partitioning algorithm. Both algorithms aim at maximizing the same criterion which reflects the extent to which variables in each cluster are related to the latent variable associated with this cluster. Illustrations are outlined using real data sets from sensory studies.     

Choosing principal components for multivariate statistical process control

Principal components are useful for multivariate process control. Typically, the principal component variables are often selected to summarize the variation in the process data. We provide an analysis to select the principal component variables to be included in a multivariate control chart that incorporates the unique aspects of the process control problem (rather than using traditional principal component guidelines).     

主成分与因子分析中指标同趋势化方法探讨

样本主成分和样本因子分析法已成为一种最主要的综合评价方法之一,指标变量的同趋势化是运用该方法的重要步骤。文章总结了主成分与因子分析中指标同趋势化的具体方法,论述了这些方法对综合评价的影响,并指出了这些方法的适用条件。     

Interpreting an Inequality in Multiple Regression

This article provides a method of interpreting a surprising inequality in multiple linear regression: the squared multiple correlation can be greater than the sum of the simple squared correlations between the response variable and each of the predictor variables. The interpretation is obtained via principal component analysis by studying the influence of some components with small variance on the response variable. One example is used as an illustration and some conclusions are derived.     

A Monte Carlo examination of the broken-stick distribution to identify components to retain in principal component analysis

ABSTRACT

The broken-stick (BS) is a popular stopping rule in ecology to determine the number of meaningful components of principal component analysis. However, its properties have not been systematically investigated. The purpose of the current study is to evaluate its ability to detect the correct dimensionality in a data set and whether it tends to over- or underestimate it. A Monte Carlo protocol was carried out. Two main correlation matrices deemed usual in practice were used with three levels of correlation (0, 0.10 and 0.30) between components (generating oblique structure) and with different sample sizes. Analyses of the population correlation matrices indicated that, for extremely large sample sizes, the BS method could be correct for only one of the six simulated structure. It actually failed to identify the correct dimensionality half the time with orthogonal structures and did even worse with some oblique ones. In harder conditions, results show that the power of the BS decreases as sample size increases: weakening its usefulness in practice. Since the BS method seems unlikely to identify the underlying dimensionality of the data, and given that better stopping rules exist it appears as a poor choice when carrying principal component analysis.     

Generalized Sobol sensitivity indices for dependent variables: numerical methods

The hierarchically orthogonal functional decomposition of any measurable function η of a random vector X=(X₁,?…?, X_p) consists in decomposing η(X) into a sum of increasing dimension functions depending only on a subvector of X. Even when X₁,?…?, X_p are assumed to be dependent, this decomposition is unique if the components are hierarchically orthogonal. That is, two of the components are orthogonal whenever all the variables involved in one of the summands are a subset of the variables involved in the other. Setting Y=η(X), this decomposition leads to the definition of generalized sensitivity indices able to quantify the uncertainty of Y due to each dependent input in X [Chastaing G, Gamboa F, Prieur C. Generalized Hoeffding–Sobol decomposition for dependent variables – application to sensitivity analysis. Electron J Statist. 2012;6:2420–2448]. In this paper, a numerical method is developed to identify the component functions of the decomposition using the hierarchical orthogonality property. Furthermore, the asymptotic properties of the components estimation is studied, as well as the numerical estimation of the generalized sensitivity indices of a toy model. Lastly, the method is applied to a model arising from a real-world problem.     

A powerful affine invariant test for multivariate normality based on interpoint distances of principal components

In this paper, a probability plots class of tests for multivariate normality is introduced. Based on independent standardized principal components of a d-variate normal data set, we obtained the sum of squared differences between corresponding observations of an ordered set of each principal component observations and the set of the population pth quantiles of the standard normal distribution. We proposed the sum of these d-sums of squared differences as an appropriate statistic for testing multivariate normality. We evaluated empirical critical values of the statistic and compared its power with those of some highly regarded techniques with a wonderful result.     

Diffusion Indexes With Sparse Loadings

The use of large-dimensional factor models in forecasting has received much attention in the literature with the consensus being that improvements on forecasts can be achieved when comparing with standard models. However, recent contributions in the literature have demonstrated that care needs to be taken when choosing which variables to include in the model. A number of different approaches to determining these variables have been put forward. These are, however, often based on ad hoc procedures or abandon the underlying theoretical factor model. In this article, we will take a different approach to the problem by using the least absolute shrinkage and selection operator (LASSO) as a variable selection method to choose between the possible variables and thus obtain sparse loadings from which factors or diffusion indexes can be formed. This allows us to build a more parsimonious factor model that is better suited for forecasting compared to the traditional principal components (PC) approach. We provide an asymptotic analysis of the estimator and illustrate its merits empirically in a forecasting experiment based on U.S. macroeconomic data. Overall we find that compared to PC we obtain improvements in forecasting accuracy and thus find it to be an important alternative to PC. Supplementary materials for this article are available online.     

A Tale of Two Matrix Factorizations

In statistical practice, rectangular tables of numeric data are commonplace, and are often analyzed using dimension-reduction methods like the singular value decomposition and its close cousin, principal component analysis (PCA). This analysis produces score and loading matrices representing the rows and the columns of the original table and these matrices may be used for both prediction purposes and to gain structural understanding of the data. In some tables, the data entries are necessarily nonnegative (apart, perhaps, from some small random noise), and so the matrix factors meant to represent them should arguably also contain only nonnegative elements. This thinking, and the desire for parsimony, underlies such techniques as rotating factors in a search for “simple structure.” These attempts to transform score or loading matrices of mixed sign into nonnegative, parsimonious forms are, however, indirect and at best imperfect. The recent development of nonnegative matrix factorization, or NMF, is an attractive alternative. Rather than attempt to transform a loading or score matrix of mixed signs into one with only nonnegative elements, it directly seeks matrix factors containing only nonnegative elements. The resulting factorization often leads to substantial improvements in interpretability of the factors. We illustrate this potential by synthetic examples and a real dataset. The question of exactly when NMF is effective is not fully resolved, but some indicators of its domain of success are given. It is pointed out that the NMF factors can be used in much the same way as those coming from PCA for such tasks as ordination, clustering, and prediction. Supplementary materials for this article are available online.     

An approach to non-linear principal components analysis using radially symmetric kernel functions

An approach to non-linear principal components using radially symmetric kernel basis functions is described. The procedure consists of two steps: a projection of the data set to a reduced dimension using a non-linear transformation whose parameters are determined by the solution of a generalized symmetric eigenvector equation. This is achieved by demanding a maximum variance transformation subject to a normalization condition (Hotelling's approach) and can be related to the homogeneity analysis approach of Gifi through the minimization of a loss function. The transformed variables are the principal components whose values define contours, or more generally hypersurfaces, in the data space. The second stage of the procedure defines the fitting surface, the principal surface, in the data space (again as a weighted sum of kernel basis functions) using the definition of self-consistency of Hastie and Stuetzle. The parameters of this principal surface are determined by a singular value decomposition and crossvalidation is used to obtain the kernel bandwidths. The approach is assessed on four data sets.