Estimation of covariance matrix via the sparse Cholesky factor with lasso

In this paper, we discuss a parsimonious approach to estimation of high-dimensional covariance matrices via the modified Cholesky decomposition with lasso. Two different methods are proposed. They are the equi-angular and equi-sparse methods. We use simulation to compare the performance of the proposed methods with others available in the literature, including the sample covariance matrix, the banding method, and the L₁-penalized normal loglikelihood method. We then apply the proposed methods to a portfolio selection problem using 80 series of daily stock returns. To facilitate the use of lasso in high-dimensional time series analysis, we develop the dynamic weighted lasso (DWL) algorithm that extends the LARS-lasso algorithm. In particular, the proposed algorithm can efficiently update the lasso solution as new data become available. It can also add or remove explanatory variables. The entire solution path of the L₁-penalized normal loglikelihood method is also constructed.     

Robust Coordinate Descent Algorithm Robust Solution Path for High-dimensional Sparse Regression Modeling

The L₁-type regularization provides a useful tool for variable selection in high-dimensional regression modeling. Various algorithms have been proposed to solve optimization problems for L₁-type regularization. Especially the coordinate descent algorithm has been shown to be effective in sparse regression modeling. Although the algorithm shows a remarkable performance to solve optimization problems for L₁-type regularization, it suffers from outliers, since the procedure is based on the inner product of predictor variables and partial residuals obtained from a non-robust manner. To overcome this drawback, we propose a robust coordinate descent algorithm, especially focusing on the high-dimensional regression modeling based on the principal components space. We show that the proposed robust algorithm converges to the minimum value of its objective function. Monte Carlo experiments and real data analysis are conducted to examine the efficiency of the proposed robust algorithm. We observe that our robust coordinate descent algorithm effectively performs for the high-dimensional regression modeling even in the presence of outliers.     

Model-averaged ℓ1 regularization using Markov chain Monte Carlo model composition

Bayesian model averaging (BMA) is an effective technique for addressing model uncertainty in variable selection problems. However, current BMA approaches have computational difficulty dealing with data in which there are many more measurements (variables) than samples. This paper presents a method for combining ?₁ regularization and Markov chain Monte Carlo model composition techniques for BMA. By treating the ?₁ regularization path as a model space, we propose a method to resolve the model uncertainty issues arising in model averaging from solution path point selection. We show that this method is computationally and empirically effective for regression and classification in high-dimensional data sets. We apply our technique in simulations, as well as to some applications that arise in genomics.     

A MORE GENERAL CRITERION FOR SUBSET SELECTION IN MULTIPLE LINEAR REGRESSION

ABSTRACT

In this article, we propose a more general criterion called S_p -criterion, for subset selection in the multiple linear regression Model. Many subset selection methods are based on the Least Squares (LS) estimator of β, but whenever the data contain an influential observation or the distribution of the error variable deviates from normality, the LS estimator performs ‘poorly’ and hence a method based on this estimator (for example, Mallows’ C_p -criterion) tends to select a ‘wrong’ subset. The proposed method overcomes this drawback and its main feature is that it can be used with any type of estimator (either the LS estimator or any robust estimator) of β without any need for modification of the proposed criterion. Moreover, this technique is operationally simple to implement as compared to other existing criteria. The method is illustrated with examples.     

OPTIMAL DESIGNS FOR FITTING A PROPORTIONAL HAZARDS REGRESSION MODEL TO DATA SUBJECT TO CENSORING

Suppose the probability model for failure time data, subject to censoring, is specified by the hazard function λ(t)exp(β_T x), where x is a vector of covariates. Analytical difficulties involved in finding the optimal design are avoided by assuming that λ is completely specified and by using D-optimality based on the information matrix for β Optimal designs are found to depend on β, but some results of practical consequence are obtained. It is found that censoring does not affect the choice of design appreciably when β^T x ≥ 0 for all points of the feasible region, but may have an appreciable effect when β_ix_i 0, for all i and all points in the feasible experimental region. The nature of the effect is discussed in detail for the cases of one and two parameters. It is argued that in practical biomedical situations the optimal design is almost always the same as for uncensored data.     

Performance characteristics of the adjusted r2 algorithm for determining the start of the terminal disposition phase and comparison with a simple r2 algorithm and a visual inspection method

The adjusted r² algorithm is a popular automated method for selecting the start time of the terminal disposition phase (t_z) when conducting a noncompartmental pharmacokinetic data analysis. Using simulated data, the performance of the algorithm was assessed in relation to the ratio of the slopes of the preterminal and terminal disposition phases, the point of intercept of the terminal disposition phase with the preterminal disposition phase, the length of the terminal disposition phase captured in the concentration‐time profile, the number of data points present in the terminal disposition phase, and the level of variability in concentration measurement. The adjusted r² algorithm was unable to identify t_z accurately when there were more than three data points present in a profile's terminal disposition phase. The terminal disposition phase rate constant (λ_z) calculated based on the value of t_z selected by the algorithm had a positive bias in all simulation data conditions. Tolerable levels of bias (median bias less than 5%) were achieved under conditions of low measurement variability. When measurement variability was high, tolerable levels of bias were attained only when the terminal phase time span was 4 multiples of t_1/2 or longer. A comparison of the performance of the adjusted r² algorithm, a simple r² algorithm, and t_z selection by visual inspection was conducted using a subset of the simulation data. In the comparison, the simple r² algorithm performed as well as the adjusted r² algorithm and the visual inspection method outperformed both algorithms. Recommendations concerning the use of the various t_z selection methods are presented.     

A modified Cp statistic in a system-of-equations model

A new statistic, SΓ(p), is developed for variable selection in a system-of-equations model. The standardized total mean square error in the SΓ(p)statistic is weighted by the covariance matrix of dependent variables instead of the error covariance matrix of the true model as in the original definition. The new statistic can be also used for model selection in the non-nested models. The estimate of SΓ(p), SC(p), is derived and shown to become SC_ε(p) in the similar form of C_p in a single-equation model when the covariance matrix of sampled dependent variables is replaced by the error covariance matrix under the full model.     

Location adjustment for the minimum volume ellipsoid estimator

Estimating multivariate location and scatter with both affine equivariance and positive breakdown has always been difficult. A well-known estimator which satisfies both properties is the Minimum Volume Ellipsoid Estimator (MVE). Computing the exact MVE is often not feasible, so one usually resorts to an approximate algorithm. In the regression setup, algorithms for positive-breakdown estimators like Least Median of Squares typically recompute the intercept at each step, to improve the result. This approach is called intercept adjustment. In this paper we show that a similar technique, called location adjustment, can be applied to the MVE. For this purpose we use the Minimum Volume Ball (MVB), in order to lower the MVE objective function. An exact algorithm for calculating the MVB is presented. As an alternative to MVB location adjustment we propose L ₁ location adjustment, which does not necessarily lower the MVE objective function but yields more efficient estimates for the location part. Simulations compare the two types of location adjustment. We also obtain the maxbias curves of L ₁ and the MVB in the multivariate setting, revealing the superiority of L ₁.     

Algorithms for restricted least absolute value estimation

We present a concise summary of recent progress in developing algorithms for restricted least absolute value (LAV) estimation (i. e. ?₁ approximation subject to linear constraints). The emphasis is on our own new algorithm, and we provide some numerical results obtained with it.     

Conditions for D A -Maximin Marginal Designs for Generalized Linear Mixed Models to be Uniform

Optimal design theory deals with the assessment of the optimal joint distribution of all independent variables prior to data collection. In many practical situations, however, covariates are involved for which the distribution is not previously determined. The optimal design problem may then be reformulated in terms of finding the optimal marginal distribution for a specific set of variables. In general, the optimal solution may depend on the unknown (conditional) distribution of the covariates. This article discusses the D _A -maximin procedure to account for the uncertain distribution of the covariates. Sufficient conditions will be given under which the uniform design of a subset of independent discrete variables is D _A -maximin. The sufficient conditions are formulated for Generalized Linear Mixed Models with an arbitrary number of quantitative and qualitative independent variables and random effects.     

On least absolute values estimation

The resistance of least absolute values (L₁) estimators to outliers and their robustness to heavy-tailed distributions make these estimators useful alternatives to the usual least squares estimators. The recent development of efficient algorithms for L₁ estimation in linear models has permitted their use in practical data analysis. Although in general the L₁ estimators are not unique, there are a number of properties they all share. The set of all L₁ estimators for a given model and data set can be characterized as the convex hull of some extreme estimators. Properties of the extreme estimators and of the L₁-estimate set are considered.     

An iterative plug-in algorithm for decomposing seasonal time series using the Berlin Method

We propose a fast data-driven procedure for decomposing seasonal time series using the Berlin Method, the procedure used, e.g. by the German Federal Statistical Office in this context. The formula of the asymptotic optimal bandwidth h _A is obtained. Methods for estimating the unknowns in h _A are proposed. The algorithm is developed by adapting the well-known iterative plug-in idea to time series decomposition. Asymptotic behaviour of the proposal is investigated. Some computational aspects are discussed in detail. Data examples show that the proposal works very well in practice and that data-driven bandwidth selection offers new possibilities to improve the Berlin Method. Deep insights into the iterative plug-in rule are also provided.     

Attribute Charts for Zero-Inflated Processes

The classical Shewhart c-chart and p-chart which are constructed based on the Poisson and binomial distributions are inappropriate in monitoring zero-inflated counts. They tend to underestimate the dispersion of zero-inflated counts and subsequently lead to higher false alarm rate in detecting out-of-control signals. Another drawback of these charts is that their 3-sigma control limits, evaluated based on the asymptotic normality assumption of the attribute counts, have a systematic negative bias in their coverage probability. We recommend that the zero-inflated models which account for the excess number of zeros should first be fitted to the zero-inflated Poisson and binomial counts. The Poisson parameter λ estimated from a zero-inflated Poisson model is then used to construct a one-sided c-chart with its upper control limit constructed based on the Jeffreys prior interval that provides good coverage probability for λ. Similarly, the binomial parameter p estimated from a zero-inflated binomial model is used to construct a one-sided np-chart with its upper control limit constructed based on the Jeffreys prior interval or Blyth–Still interval of the binomial proportion p. A simple two-of-two control rule is also recommended to improve further on the performance of these two proposed charts.     

Clustering gene expression time course data using mixtures of multivariate t-distributions

Clustering gene expression time course data is an important problem in bioinformatics because understanding which genes behave similarly can lead to the discovery of important biological information. Statistically, the problem of clustering time course data is a special case of the more general problem of clustering longitudinal data. In this paper, a very general and flexible model-based technique is used to cluster longitudinal data. Mixtures of multivariate t-distributions are utilized, with a linear model for the mean and a modified Cholesky-decomposed covariance structure. Constraints are placed upon the covariance structure, leading to a novel family of mixture models, including parsimonious models. In addition to model-based clustering, these models are also used for model-based classification, i.e., semi-supervised clustering. Parameters, including the component degrees of freedom, are estimated using an expectation-maximization algorithm and two different approaches to model selection are considered. The models are applied to simulated data to illustrate their efficacy; this includes a comparison with their Gaussian analogues—the use of these Gaussian analogues with a linear model for the mean is novel in itself. Our family of multivariate t mixture models is then applied to two real gene expression time course data sets and the results are discussed. We conclude with a summary, suggestions for future work, and a discussion about constraining the degrees of freedom parameter.     

L1 for the simple linear regression model

By modifying the direct method to solve the overdetermined linear system we are able to present an algorithm for L₁ estimation which appears to be superior computationally to any other known algorithm for the simple linear regression problem.     

On perturbations of Stein operator

In this article, we obtain a Stein operator for the sum of n independent random variables (rvs) which is shown as the perturbation of the negative binomial (NB) operator. Comparing the operator with NB operator, we derive the error bounds for total variation distance by matching parameters. Also, three-parameter approximation for such a sum is considered and is shown to improve the existing bounds in the literature. Finally, an application of our results to a function of waiting time for (k₁, k₂)-events is given.     

D s -optimal designs for Kozak's tree taper model

In this work, we study D _s -optimal design for Kozak's tree taper model. The approximate D _s -optimal designs are found invariant to tree size and hence create a ground to construct a general replication-free D _s -optimal design. Even though the designs are found not to be dependent on the parameter value p of the Kozak's model, they are sensitive to the s×1 subset parameter vector values of the model. The 12 points replication-free design (with 91% efficiency) suggested in this study is believed to reduce cost and time for data collection and more importantly to precisely estimate the subset parameters of interest.     

Stochastic Comparisons and Dependence of Spacings from Two Samples of Exponential Random Variables

Let X ₁,X ₂,…,X _n be independent exponential random variables such that X _i has hazard rate λ for i = 1,…,p and X _j has hazard rate λ* for j = p + 1,…,n, where 1 ≤ p < n. Denote by D _i:n (λ, λ*) = X _i:n  ? X _i?1:n the ith spacing of the order statistics X _1:n  ≤ X _2:n  ≤ ··· ≤ X _n:n , i = 1,…,n, where X _0:n ≡ 0. It is shown that the spacings (D _1,n ,D _2,n ,…,D _n:n ) are MTP₂, strengthening one result of Khaledi and Kochar (2000), and that (D _1:n (λ₂, λ*),…,D _n:n (λ₂, λ*)) ≤ _lr (D _1:n (λ₁, λ*),…,D _n:n (λ₁, λ*)) for λ₁ ≤ λ* ≤ λ₂, where ≤ _lr denotes the multivariate likelihood ratio order. A counterexample is also given to show that this comparison result is in general not true for λ* < λ₁ < λ₂.     

Numerical algorithms for solving nonlinear L р-norm estimation problems: part II - a mixture method for large residual and illo-conditioned problems

The nonlinear least squares algorithm of Gill and Murray (1978) is extended and modified to solve nonlinear L _р-norm estimation problems efficiently. The new algorithm uses a mixture of 1st-order derivative (Guass-Newton) and 2nd-order derivative (Newton) search directions. A new rule for selecting the “grade” r of the p-jacobiab matrix J_p was also incorporated. This brought about rapid convergence of the algorithm on previously reported test examples.     

Remarks on the L1 distance in statistical data analysis

We propose the L₁ distance between the distribution of a binned data sample and a probability distribution from which it is hypothetically drawn as a statistic for testing agreement between the data and a model. We study the distribution of this distance for N-element samples drawn from k bins of equal probability and derive asymptotic formulae for the mean and dispersion of L₁ in the large-N limit. We argue that the L₁ distance is asymptotically normally distributed, with the mean and dispersion being accurately reproduced by asymptotic formulae even for moderately large values of N and k.