Interval estimation in discriminant analysis for large dimension

This paper is concerned with the interval estimation for the log odds of the posterior probability that the observation vector belongs to one of two homoscedastic multivariate normal distributions (Π₁ and Π₂). We give the limiting distribution of the unbiased estimator for the log odds as the sample sizes and the dimension jointly tend to infinity, and approximate the confidence interval based on the asymptotic distribution. Small-scale simulations are performed to check the precision of the approximation.     

Error-rate estimation in two-group discriminant analysis using the linear discriminant function

Fisher's linear discriminant function, adapted by Anderson for allocating new observations into one of two existing groups, is considered in this paper. Methods of estimating the misclassification error rates are reviewed and evaluated by Monte Carlo simulations. The investigation is carried out under both ideal (Multivariate Normal data) and non-ideal (Multivariate Binary data) conditions. The assessment is based on the usual mean square error (MSE) criterion and also on a new criterion of optimism. The results show that although there is a common cluster of good estimators for both ideal and non-ideal conditions, the single best estimators vary with respect to the different criteria     

Modified linear discriminant analysis using block covariance matrix in high-dimensional data

Among many classification methods, linear discriminant analysis (LDA) is a favored tool due to its simplicity, robustness, and predictive accuracy but when the number of genes is larger than the number of observations, it cannot be applied directly because the within-class covariance matrix is singular. Also, diagonal LDA (DLDA) is a simpler model compared to LDA and has better performance in some cases. However, in reality, DLDA requires a strong assumption based on mutual independence. In this article, we propose the modified LDA (MLDA). MLDA is based on independence, but uses the information that has an effect on classification performance with the dependence structure. We suggest two approaches. One is the case of using gene rank. The other involves no use of gene rank. We found that MLDA has better performance than LDA, DLDA, or K-nearest neighborhood and is comparable with support vector machines in real data analysis and the simulation study.     

A hierarchical eigenmodel for pooled covariance estimation

Summary.  Although the covariance matrices corresponding to different populations are unlikely to be exactly equal they can still exhibit a high degree of similarity. For example, some pairs of variables may be positively correlated across most groups, whereas the correlation between other pairs may be consistently negative. In such cases much of the similarity across covariance matrices can be described by similarities in their principal axes, which are the axes that are defined by the eigenvectors of the covariance matrices. Estimating the degree of across-population eigenvector heterogeneity can be helpful for a variety of estimation tasks. For example, eigenvector matrices can be pooled to form a central set of principal axes and, to the extent that the axes are similar, covariance estimates for populations having small sample sizes can be stabilized by shrinking their principal axes towards the across-population centre. To this end, the paper develops a hierarchical model and estimation procedure for pooling principal axes across several populations. The model for the across-group heterogeneity is based on a matrix-valued antipodally symmetric Bingham distribution that can flexibly describe notions of 'centre' and 'spread' for a population of orthogonal matrices.     

Rank covariance matrix estimation of a partially known covariance matrix

Classical multivariate methods are often based on the sample covariance matrix, which is very sensitive to outlying observations. One alternative to the covariance matrix is the affine equivariant rank covariance matrix (RCM) that has been studied in Visuri et al. [2003. Affine equivariant multivariate rank methods. J. Statist. Plann. Inference 114, 161–185]. In this article we assume that the covariance matrix is partially known and study how to estimate the corresponding RCM. We use the properties that the RCM is affine equivariant and that the RCM is proportional to the inverse of the regular covariance matrix, and hence reduce the problem of estimating the original RCM to estimating marginal rank covariance matrices. This is a great computational advantage when the dimension of the original data vector is large.     

On a distributionfree method in discriminant analysis

Linear discriminant rules for two symmetrical distributions, which only need the first and second moments of these distributions, are presented. The rules are based on Zhezhel's idea using the most unfavourable probabilities of misclassification as an optimality criterion. Also a rule is considered which deals with distributions differing in a location and scale parameter.     

Estimation of the population spectral distribution from a large dimensional sample covariance matrix

This paper introduces a new method to estimate the spectral distribution of a population covariance matrix from high-dimensional data. The method is founded on a meaningful generalization of the seminal Mar?enko–Pastur equation, originally defined in the complex plane, to the real line. Beyond its easy implementation and the established asymptotic consistency, the new estimator outperforms two existing estimators from the literature in almost all the situations tested in a simulation experiment. An application to the analysis of the correlation matrix of S&P 500 daily stock returns is also given.     

Empirical bayes estimation of the mean in a multivariate normal distribution

We consider the problem of estimating the mean vector of a multivariate normal distribution under a variety of assumed structures among the parameters of the sampling and prior distributions. We adopt a pragmatic approach. We adopt distributional familites, assess hyperparmeters, and adopt patterned mean and coveariance structures when it is relatively simple to do so; alternatively, we use the sample data to estimate hyperparameters of prior distributions when assessment is a formidable task; such as the task of assessing parameters of multidimensional problems. James-Stein-like estimators are found to result. In some cases, we've been abl to show that the estimators proposed uniformly dominate the MLE's when measured with respect to quadratic loss functions.     

On the computation of noncentral chi-squared distributions

A computational formula for computing the cumulative distribution function of noncentral chi-squared distributions with odd degrees of freedom is given.     

A study on discriminant analysis techniques applied to multivariate lognormal data

The purpose of this paper is to examine the multiple group (>2) discrimination problem in which the group sizes are unequal and the variables used in the classification are correlated with skewed distributions. Using statistical simulation based on data from a clinical study, we compare the performances, in terms of misclassification rates, of nine statistical discrimination methods. These methods are linear and quadratic discriminant analysis applied to untransformed data, rank transformed data, and inverse normal scores data, as well as fixed kernel discriminant analysis, variable kernel discriminant analysis, and variable kernel discriminant analysis applied to inverse normal scores data. It is found that the parametric methods with transformed data generally outperform the other methods, and the parametric methods applied to inverse normal scores usually outperform the parametric methods applied to rank transformed data. Although the kernel methods often have very biased estimates, the variable kernel method applied to inverse normal scores data provides considerable improvement in terms of total nonerror rate.     

Asymptotic optimality of a two-stage procedure in Bayes sequential estimation

The problem of sequential estimation of the mean with quadratic loss and fixed cost per observation is considered within the Bayesian framework. Instead of fully sequential sampling, a two-stage sampling technique is introduced to solve the problem. The proposed two-stage procedure is robust in the sense that it does not depend on the distribution of outcome variables and the prior. It is shown to be asymptotically not worse than the optimal fixed-sample-size procedures for the arbitrary distributions, and to be asymptotically Bayes for the distributions of one-parameter exponential family.     

Minimax linear empirical bayes estimation of the binomial parameter

The minimax linear Empirical Bayes estimators for a binomial parameter are obtained, assuming some information about the moments of the prior. The form of these estimates is used to propose a criterion which may be helpful in determining whether Empirical Bayes estimation is Indicated for a given problem.     

Regularized covariance matrix estimation under the common principal components model

The common principal components (CPC) model provides a way to model the population covariance matrices of several groups by assuming a common eigenvector structure. When appropriate, this model can provide covariance matrix estimators of which the elements have smaller standard errors than when using either the pooled covariance matrix or the per group unbiased sample covariance matrix estimators. In this article, a regularized CPC estimator under the assumption of a common (or partially common) eigenvector structure in the populations is proposed. After estimation of the common eigenvectors using the Flury–Gautschi (or other) algorithm, the off-diagonal elements of the nearly diagonalized covariance matrices are shrunk towards zero and multiplied with the orthogonal common eigenvector matrix to obtain the regularized CPC covariance matrix estimates. The optimal shrinkage intensity per group can be estimated using cross-validation. The efficiency of these estimators compared to the pooled and unbiased estimators is investigated in a Monte Carlo simulation study, and the regularized CPC estimator is applied to a real dataset to demonstrate the utility of the method.     

The application of bias to discriminant analysis

When classification rules are constructed using sample estimatest it is known that the probability of misclassification is not minimized. This article introduces a biased minimum X² rule to classify items from a multivariate normal population. Using the principle of variance reduction, the probability of misclassification is reduced when the biased procedure is employed. Results of sampling experiments over a broad range of conditions are provided to demonstrate this improvement.     

Further applications of bias to discriminant analysis

This article extends the biased minimum x² rule to the unequal covariance matrix case and to the case of several populations, The biased procedure is shown to improve the performance of the commonly used classification procedures. Results of sampling experiments over a broad range of conditions are provided to demonstrate this improvement.     

Sparse discriminant analysis based on estimation of posterior probabilities

ABSTRACT

Fisher's linear discriminant analysis (FLDA) is known as a method to find a discriminative feature space for multi-class classification. As a theory of extending FLDA to an ultimate nonlinear form, optimal nonlinear discriminant analysis (ONDA) has been proposed. ONDA indicates that the best theoretical nonlinear map for maximizing the Fisher's discriminant criterion is formulated by using the Bayesian a posterior probabilities. In addition, the theory proves that FLDA is equivalent to ONDA when the Bayesian a posterior probabilities are approximated by linear regression (LR). Due to some limitations of the linear model, there is room to modify FLDA by using stronger approximation/estimation methods. For the purpose of probability estimation, multi-nominal logistic regression (MLR) is more suitable than LR. Along this line, in this paper, we develop a nonlinear discriminant analysis (NDA) in which the posterior probabilities in ONDA are estimated by MLR. In addition, in this paper, we develop a way to introduce sparseness into discriminant analysis. By applying L1 or L2 regularization to LR or MLR, we can incorporate sparseness in FLDA and our NDA to increase generalization performance. The performance of these methods is evaluated by benchmark experiments using last_exam17 standard datasets and a face classification experiment.     

Another approach to inverting a covariance matrix when data are unbalanced

Searle and Rudan (1973) derive the inverse of a covariance matrix for unbalanced data in ANOVA. Their expression is highly complicated. This paper presents an alternative derivation and shows how unbalancedness enters in.